Not Enough Hours in a Day
Nov. 8th, 2011 11:48 amLast Thursday, I did an all-nighter at the data center. It was scheduled maintenance, a complete site outage in which we would update our production software, upgrade the RAM in some servers, update the firmware in our storage array and shuffle some existing servers between various racks.
It didn't go exactly as planned.
While the RAM upgrades and software updates went well, there were some challenges. Two of the re-racked servers didn't reboot as they should. One had developed slight amnesia and couldn't locate its boot disk: that was quickly fixed. The other server's motherboard went bad but it was part of a cluster so we were able to run on reduced capacity until a replacement was installed Monday.
The storage array firmware didn't happen either. The upgrade process requires a fully functional array but as we were preparing to begin, we found there was a bad disk. Checking the logs, we found the array did try to tell us but because our service agreement had lapsed briefly while our finance dept quibbled about whether or not to renew it, the reporting mechanism which would automatically issue the work ticket was disabled. And while we did get a replacement disk before the night's maintenance window was out, we couldn't do the firmware upgrade while the disk RAID group was being reconstructed, a process which would take another 12 hours or so. sigh
I got to bed around 4:30am.
I nominally had Friday off work to recover. Later that morning, the guys placed me more or less vertically in the minivan and we headed to the trailer for the last camping season of the weekend.
The campground was very quiet: we had perhaps a dozen people attending. I had hoped the trees might have more leaves so I could photograph some fall foliage but they were nearly all barren, the bulk of the leaves taken down by the weight of snow the weekend prior. That snow had since melted away and despite that unexpected cold snap, our weekend was bright, warm and sunny. And I spent nearly all of it catching up on sleep.
I'm happy to report my Droid Bionic mobile phone works well at the campground in rural West Virginia. Having a working mobile phone and wireless Internet allows me to work from the trailer if I want a long weekend or need to deal with a work-related emergency. The campground is remote enough that most AT&T and Spring users have spotty voice access at best; while Verizon has better coverage there, not every model of phone can pull in sufficient signal to be reliable. My old Blackberry Storm worked well enough even if my old wireless broadband modem only worked if I was outside the trailer. It's a huge relief that my new phone and its mobile hotspot functionality seem to work solidly.
Winterizing and wrapping up the trailer was all very routine. It's always sad when another camping season comes to an end but we're planning to open early, possibly mid-March next year so the interregnum will be shorter this time.
We were home by early evening and the real fun began.
It didn't go exactly as planned.
While the RAM upgrades and software updates went well, there were some challenges. Two of the re-racked servers didn't reboot as they should. One had developed slight amnesia and couldn't locate its boot disk: that was quickly fixed. The other server's motherboard went bad but it was part of a cluster so we were able to run on reduced capacity until a replacement was installed Monday.
The storage array firmware didn't happen either. The upgrade process requires a fully functional array but as we were preparing to begin, we found there was a bad disk. Checking the logs, we found the array did try to tell us but because our service agreement had lapsed briefly while our finance dept quibbled about whether or not to renew it, the reporting mechanism which would automatically issue the work ticket was disabled. And while we did get a replacement disk before the night's maintenance window was out, we couldn't do the firmware upgrade while the disk RAID group was being reconstructed, a process which would take another 12 hours or so. sigh
I got to bed around 4:30am.
I nominally had Friday off work to recover. Later that morning, the guys placed me more or less vertically in the minivan and we headed to the trailer for the last camping season of the weekend.
The campground was very quiet: we had perhaps a dozen people attending. I had hoped the trees might have more leaves so I could photograph some fall foliage but they were nearly all barren, the bulk of the leaves taken down by the weight of snow the weekend prior. That snow had since melted away and despite that unexpected cold snap, our weekend was bright, warm and sunny. And I spent nearly all of it catching up on sleep.
I'm happy to report my Droid Bionic mobile phone works well at the campground in rural West Virginia. Having a working mobile phone and wireless Internet allows me to work from the trailer if I want a long weekend or need to deal with a work-related emergency. The campground is remote enough that most AT&T and Spring users have spotty voice access at best; while Verizon has better coverage there, not every model of phone can pull in sufficient signal to be reliable. My old Blackberry Storm worked well enough even if my old wireless broadband modem only worked if I was outside the trailer. It's a huge relief that my new phone and its mobile hotspot functionality seem to work solidly.
Winterizing and wrapping up the trailer was all very routine. It's always sad when another camping season comes to an end but we're planning to open early, possibly mid-March next year so the interregnum will be shorter this time.
We were home by early evening and the real fun began.