Oh, my...

[bjarvis]

Test your backups! And for god's sake, don't format data hard drives on production servers!

http://news.yahoo.com/s/ap/20070320/ap_on_hi_te/lost_data_3

Comments

[g8rcub.livejournal.com]

This one is currenlty being thrown around my office...thanks for the link...all we can say is "what an idiot" The techs working for me KNOW better than to format a disc that hasn't been backed up. No matter what the users says....back it up, and keep it for a while.

[manley1.livejournal.com]

Ugh, I feel for this guy. I once wiped out our data warehouse by overwriting five of the production SAN volumes with backup copies from a test database. I had been, ironically, been working to do the opposite. If you've ever worked with EMC's Symmetrix devices and BCV map files, you know the horrible, horrible hoops you need to jump to map devices under Solaris. I had written the data down one way, then one the next page (after receiving a couple of annoying phone calls) wrote it down another. Luckily, we didn't lose any data, but it was very embarrassing to bring down the main production data warehouse for the day. No one was mad because I fessed up too, which made me feel better. Still, I think we all have our fat-fingering stories...

[bjarvis.livejournal.com]

I'm a big believer in multiple backups, reinforced by my recent USB flash drive failure. Tracking the version of the backups, that's a bigger problem but one I'd much rather have than finding unreadable/nonexistant backups.

[bjarvis.livejournal.com]

I've done my share of bone-headed moves too. Most commonly, they've been cut-and-paste errors between the wrong windows, accidentally triggering actions on the wrong server. I've accidentally rebooted a server or two in the past as well. I haven't wiped out whole SAN volumes, but I have been present when it happened and it isn't a good feeling, even vicariously.

[manley1.livejournal.com]

Oh wow, I love that LJ icon!

My bonehead event.

[furrbear.livejournal.com]

Deleting the account of a former sysadmin who instead of using 'su', set all of his logins to UID 0. Didn't catch the UID 0 part and let the script delete his files - kernel panic trying to delete /bin/sh. Ooooooops.

[allanh.livejournal.com]

I have NO sympathy whatsoever. I long ago learned to back up anything I'm going to work on before I touch it. "Long ago" as in ... high school, during my first programming class.

I'm also unsympathetic because a substantial portion of my job involves creating and implementing backup/restore systems for customers ... most of whom fight me on the need for such a system every step of the way.

("Can't we just copy everything to another hard drive?" "NO!")

There's a laissez-faire attitude about backups in recent years that's just been driving me up the wall. People assume that what works for a home machine (doing an image backup to a CD, DVD, or USB external disk drive) will work just fine for a corporate system. Unfortunately, those methods don't scale up adequately for business needs.

Don't think I won't use this article in my remaining sessions here at BrainShare. I'm co-teaching a session on "Preventing Server Crashes" tomorrow morning, and I will be CERTAIN to quote this story under "User Error" as a cause of crashes and data loss.

Thank you!

Re: My bonehead event.

[bjarvis.livejournal.com]

I'd have made the same error... it wouldn't have occurred to me to check as any admin who sets his default uid to 0 deserves to be flogged (IMHO, of course).

[bjarvis.livejournal.com]

Thank the good people of Alaska! Bless their hearts!