bjarvis: (Default)
bjarvis ([personal profile] bjarvis) wrote2005-08-29 09:28 am
Entry tags:

SysAdmin Rite of Passage

Greg, one of my colleagues in an adjacent cube, is having a very hard day today and it's only 9:30 AM.

About 30 minutes ago, he meant to type the following (with root privileges):
crontab -l | grep reboot

What he actually typed was:
crontab -l | reboot

For the non-geek, the first command would have searched through the list of automatically-scheduled commands for the time & date of the automated reboot. The second one listed the commands and initiated the reboot immediately. It's a small typo with big consequences: the unscheduled reboot set off all manner of alarms & automated pages for the system administrator staff, our managers, the business users, the developers, the customer support staff and our senior corporate management. There's a conference call in progress at the moment to examine the causes & consequences of the reboot.

Since Greg is something of a perfectionist, he's dying of embarrassment as I type this, and would like nothing better than have the earth swallow him up.

This significant faux pas is, for our staff at least, something of a rite of passage: every senior admin in our data centers has made this kind of error at least once in his/her career. I've done it myself at least twice in the past decade or so, which isn't a bad average since I support a division of approximately 400 servers. In a twisted kind of way, Greg has finally arrived: he's now truly One Of Us. :-)

It's not much of a comfort at the moment, but we'll have a good laugh about it some day.

[identity profile] otterpop58.livejournal.com 2005-08-29 05:54 pm (UTC)(link)
ouch!

back in 1983 (on a Sun 1 !) typed "rm *" when I thought was I in some deep subdirectory and I was actually in "/" as root. bootloader gone, vmunix gone, etc.

needless to say this was unrecoverable and we had to reload the OS.

[identity profile] bjarvis.livejournal.com 2005-08-29 08:53 pm (UTC)(link)
Ah, the Sun 1 models... I remember them well. I had to write a machine language assembler for them once as a school project. Quite the step up from my Commodore 64. :-)

Yeah, I've nuked my root partition accidentally more than once in my younger, dumber days. And dropped a stack of punched cards just before setting them in the reader. And accidentally formatted the wrong floppy (back when they actually were floppy). It's the joy/pain of the trade.