The Day from Heck

It wasn't quite the day from hell but it was pretty close.

The morning began with the location services apps on my employer's production systems spontaneously dying. Without the location services, nearly all of the core customer applications fail so this was a big deal. We have three servers running separate instances, load balancing between the lot but it didn't help: each one was collapsing in sequence so I spent nearly two hours playing whack-a-mole to restart the services while the applications teams tried to debug the issue.

No sooner had that little issue settled down when the next crisis happened along: my Blackberry Storm suffered a stroke.

It was working flawlessly all morning, but it crashed, then hit some form of error within 15 seconds of starting a reboot so the reboot would start over again. At no point was an error actually displayed: the screen hadn't been activated before the next crash would overtake it. I tried pulling the battery to do a cold power cycle but it did no good.

Why is this bad? I'm supposed to be on-call this week for work, it's a holiday tomorrow so the shops won't be open, traffic was already hell and we were supposed to leave for southern Virginia tonight. Wonderful timing.

I signed off from work and began driving to the local Verizon store. I got half-way there when the Blackberry decided it would work after all so I turned around and headed back home again, hoping to get it plugged into my workstation for a full backup before it died.

The machine crashed again when I got within a half-mile of home.

I set out for Verizon once again, this time taking my laptop with me so I could perform the backup immediately should the machine resurrect itself. Sure enough, as soon as I got into the parking lot of the Verizon store, it booted flawlessly. After performing a full backup, I went to talk to the store weasels but found there was such a line it would be two hours before I could get anything done. Figures.

I couldn't spare the time so I'm crossing my fingers that the beast will behave itself for another day or two. So far, it's been peachy but I can't say I trust it.

The balance of the day has been spent working on imaging a couple of blades into new servers over the Internet in our California office. It felt like a slow-motion root canal without anaesthetic. The infrastructure in the San Francisco really needs an update as soon as possible... it should never take this kind of pain to kickstart a new box.

BTW, if Cthulhu pops by to claim a soul, send him in my direction. I did get the machines imaged but the estimated cost was a little higher than budgeted.

Even now, the blades aren't done yet: they aren't talking to the correct network subnets. I'm too frustrated and annoyed to deal with this currently. I suspect the folks who deal with these machines daily have some ideas of what's gone weird but everyone has left early for Thanksgiving so it will have to wait.

Now to go pack for our trip tonight...

The Day from Heck

It wasn't quite the day from hell but it was pretty close.

The morning began with the location services apps on my employer's production systems spontaneously dying. Without the location services, nearly all of the core customer applications fail so this was a big deal. We have three servers running separate instances, load balancing between the lot but it didn't help: each one was collapsing in sequence so I spent nearly two hours playing whack-a-mole to restart the services while the applications teams tried to debug the issue.

No sooner had that little issue settled down when the next crisis happened along: my Blackberry Storm suffered a stroke.

It was working flawlessly all morning, but it crashed, then hit some form of error within 15 seconds of starting a reboot so the reboot would start over again. At no point was an error actually displayed: the screen hadn't been activated before the next crash would overtake it. I tried pulling the battery to do a cold power cycle but it did no good.

Why is this bad? I'm supposed to be on-call this week for work, it's a holiday tomorrow so the shops won't be open, traffic was already hell and we were supposed to leave for southern Virginia tonight. Wonderful timing.

I signed off from work and began driving to the local Verizon store. I got half-way there when the Blackberry decided it would work after all so I turned around and headed back home again, hoping to get it plugged into my workstation for a full backup before it died.

The machine crashed again when I got within a half-mile of home.

I set out for Verizon once again, this time taking my laptop with me so I could perform the backup immediately should the machine resurrect itself. Sure enough, as soon as I got into the parking lot of the Verizon store, it booted flawlessly. After performing a full backup, I went to talk to the store weasels but found there was such a line it would be two hours before I could get anything done. Figures.

I couldn't spare the time so I'm crossing my fingers that the beast will behave itself for another day or two. So far, it's been peachy but I can't say I trust it.

The balance of the day has been spent working on imaging a couple of blades into new servers over the Internet in our California office. It felt like a slow-motion root canal without anaesthetic. The infrastructure in the San Francisco really needs an update as soon as possible... it should never take this kind of pain to kickstart a new box.

BTW, if Cthulhu pops by to claim a soul, send him in my direction. I did get the machines imaged but the estimated cost was a little higher than budgeted.

Even now, the blades aren't done yet: they aren't talking to the correct network subnets. I'm too frustrated and annoyed to deal with this currently. I suspect the folks who deal with these machines daily have some ideas of what's gone weird but everyone has left early for Thanksgiving so it will have to wait.

Now to go pack for our trip tonight...