LJ returns, not that it should have gone anywhere in the first place...

[bjarvis]

LJ isn't such a huge portion of my life that I was severely inconvenienced by last night's outage. God knows I had enough other things to work on.

Still, how is it that a single power outage can take down the entire system? Do they not have backup power, servers distributed between more than one data center to provide redundancy, etc?

Six Apart, you're looking like a bunch of high school amateurs. Time to step up to the first tier of professionalism.

Comments

Those who do not remember the past ...

[allanh.livejournal.com]

The current news is that 365 Main's data center (located on swamp landfill in South of Market San Francisco) uses a flywheel power transfer device to avoid allowing PG&E (utility) power to directly touch any building systems.

One sustained power outage would have triggered the diesel backup generator into action, which would then have kept the flywheel spinning.

Unfortunately, nobody at 365 Main ever stopped to think about SIX short power outages, spaced closely together.

Power went off and came on too quickly the first time for the built-in timer to kick in the generator ... so the flywheel kept operating on utility power, but was spinning a little bit slower.

Power went off and came on too quickly the second time ... and the generator didn't kick in, and the flywheel kept on spinning down.

Repeat above step four more times, and the flywheel wasn't spinning fast enough to transfer power into the data center ... and there's no provision for a slowly spinning flywheel triggering the generator.

And none of the (totally stupid) employees at 365 Main thought to slap the manual cutover switch IMMEDIATELY upon being signaled of the first power outage.

That, or even more horrifyingly, they have no power outage signals in the control center.

Stupid.

What's worse ... this isn't the first time that downtown SF has had multiple small power failures leading to a big power failure. Such as happened during the Loma Prieta Quake of '89.

Really stupid.

Re: Those who do not remember the past ...

[bjarvis.livejournal.com]

Yeah, stupid. The part that I find painful though is that there doesn't appear to be an alternate data center.

My current employer runs two data centers, although I'm not keen on them being quite so close together (about 50 miles). My prior employer had one in the greater DC area, the alternate in Montana.

It strikes me as spectacularly foolish that any online company would place all of their family jewels in a single data center, esp. one located on a former swamp inside a an earthquake zone.

Dumb, dumb, dumb.