May. 30th, 2017

bjarvis: (Default)
The past few months, I've been working on a major project for work: building out a new cage in our data center near Sterling, VA. Our current two cages are working reasonably well, but the equipment has aged, enough that much of it is no longer viable under Paycard Industry 3.2 standards (PCI). Even on the hardware we could keep using, we desperately need operating system upgrades and extra capacity.

The new cage has 9 racks, compared to the 27 racks in the old cage. Nearly everything is virtualized and clustered, all of it has the latest patches of whatever OS they're running, and we have RAM, storage & CPU cycles to spare.

And this past weekend, we went live.

It was a bit rocky in parts. One of the first tasks I had in the migration plan was to fix some issues in our Sun Microsystems/Oracle database servers, and our Veritas Cluster System. It took more hours than I was hoping/expecting, but I did get through it all. I think I spent more time delving into the depths of VCS that one night than I did the previous ten years.

By midday Saturday, everything had been migrated and we were starting running test traffic through it. We found some issues in routing, permissions, ownerships and such, but not many. Most effort was focused on getting the F5 traffic managers fully tuned for our requirements.

Today was our first regular business day since the cutover, and although we've had some problems getting our new IP ranges white-listed with a couple of our larger customers and had some performance problems with our hotel search databases, the day has been a success. We're getting great comments about the vastly improved speed & performance of our systems as well.

The road ahead is still a long one. While the travel portion of our systems have migrated, our Purchase and Car Service divisions have not yet. The Sun servers moved to the new cage, but their data still resides on a storage array in the old cage. At this moment, I'm still waiting for the license codes to built out a new monitoring system.

Once all of that has finished, there's a tonne of dismantling & disposal to do with the old cages & equipment. Some will be redeployed in our non-production environments, but 80% will be trashed completely. The most intensive part of the project is over, but I have work for the rest of the summer.

July 2017

S M T W T F S
      1
2345678
9101112131415
16171819202122
23 242526272829
3031     

Most Popular Tags

Page Summary

Style Credit

Expand Cut Tags

No cut tags
Page generated Jul. 28th, 2017 02:41 am
Powered by Dreamwidth Studios