Oh, my...

Mar. 20th, 2007 11:57 am
bjarvis: (Plankton)
[personal profile] bjarvis
Test your backups! And for god's sake, don't format data hard drives on production servers!

http://news.yahoo.com/s/ap/20070320/ap_on_hi_te/lost_data_3

Date: 2007-03-20 05:06 pm (UTC)
From: [identity profile] manley1.livejournal.com
Ugh, I feel for this guy. I once wiped out our data warehouse by overwriting five of the production SAN volumes with backup copies from a test database. I had been, ironically, been working to do the opposite. If you've ever worked with EMC's Symmetrix devices and BCV map files, you know the horrible, horrible hoops you need to jump to map devices under Solaris. I had written the data down one way, then one the next page (after receiving a couple of annoying phone calls) wrote it down another. Luckily, we didn't lose any data, but it was very embarrassing to bring down the main production data warehouse for the day. No one was mad because I fessed up too, which made me feel better. Still, I think we all have our fat-fingering stories...

Date: 2007-03-20 07:18 pm (UTC)
From: [identity profile] bjarvis.livejournal.com
I've done my share of bone-headed moves too. Most commonly, they've been cut-and-paste errors between the wrong windows, accidentally triggering actions on the wrong server. I've accidentally rebooted a server or two in the past as well. I haven't wiped out whole SAN volumes, but I have been present when it happened and it isn't a good feeling, even vicariously.

Date: 2007-03-20 07:19 pm (UTC)
From: [identity profile] manley1.livejournal.com
Oh wow, I love that LJ icon!

My bonehead event.

Date: 2007-03-20 10:34 pm (UTC)
From: [identity profile] furrbear.livejournal.com
Deleting the account of a former sysadmin who instead of using 'su', set all of his logins to UID 0. Didn't catch the UID 0 part and let the script delete his files - kernel panic trying to delete /bin/sh. Ooooooops.

Re: My bonehead event.

Date: 2007-03-21 12:41 pm (UTC)
From: [identity profile] bjarvis.livejournal.com
I'd have made the same error... it wouldn't have occurred to me to check as any admin who sets his default uid to 0 deserves to be flogged (IMHO, of course).

January 2021

S M T W T F S
     1 2
3456789
10111213141516
17181920212223
24252627282930
31      

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Jan. 9th, 2026 05:29 am
Powered by Dreamwidth Studios