bjarvis: (Default)
[personal profile] bjarvis
This morning, a fruit basket & candy package arrived, sent by my COO, CTO & head of HR, addressed to the "Jarvis Family." The note attached thanked my clan for supporting me as I worked extra hours on the latest set of office projects. Nice touch --although I'd rather have a small bonus rather than $50 gift package. Thoughtful, though.

I've been working extra hours (extra extra hours?) this week as we run down the clock to our big data center migration. Today, two people who now report to me (one is my former boss, on contract!) are flying from San Francisco to Sterling, VA. Tomorrow morning, I also head to Virginia to meet with them, ensure their badges & keys work at the data center, and generally show them the cages, servers & tools we have on-site for this migration. I also hope we can discuss in person the sequence of steps we're taking once the site goes offline Friday night, filling in any details I may have overlooked.

I'm staying at a hotel in Virginia Friday through Sunday so I can be as close as possible to the data center. I'm also expecting that after many extended hours of battle, I'd be in no shape for a 40 minute drive home and return the following day.

Friday night, about 10pm Easter time, our site will go down and the fun begins. In our preliminary testing, we were getting speed improvements of 5x or so, but I think that's just a happy dream of what we'll be able to do in another two months: even after the apps move from the old cage to the new, the databases will largely be reaching back to the storage arrays in the old cage over a 4Gbps fibre link until we can migrate the data. I do intend to start migrating the data after we go live this weekend, but it will take weeks of effort to finish that (I'm hoping to have the bulk of it done by July 1).

I cannot say how relieved I am to get the new cage online. It's not just that we've been working on it constantly the past several months, but we've been letting maintenance of the old cage slide a bit, and it had inherent issues we couldn't easily fix anyway. Killing the old cage removes a lot of legacy equipment & unfortunate architecture decisions: the slate gets swept clean.

And even when we shut down the old cage, there is still much to do. We could only make this deadline by physically moving the Sun T4-1 servers with their Oracle databases to the new cage. We were originally planning to let that equipment be retired, but that aspect is a huge project in itself. In the next three month work sprint, we're going to: identify the data we need to retain, convert the data from SPARC data word format into Intel data word format, and restructure the database & LUN layout. All on live systems. The database team has 80% of this workload, but I'm still in the mix, doing the storage allocations and helping where I can with optimizing the process.

At this moment, here and now, I'm feeling rather serene. I've just ticked off the last of my pre-migration tasks, and finished scripting a lot of Veritas cluster stuff I need to do the moment the site is offline Friday night. In all, I'm now in wait mode. There is nothing left to do but wait for the dawn.

September 2017

S M T W T F S
     12
3 45678 9
10111213141516
17181920212223
24252627282930

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Sep. 26th, 2017 02:06 am
Powered by Dreamwidth Studios