bjarvis | Entries tagged with work

Well, 2020 couldn't end soon enough for my liking, but despite that, I'm a bit surprised that it's 2021 now. More truthfully, I can accept that it is 2021, but I'm having difficulty wrapping my head around the fact that it is now January. It feels like it should be some other month. Odd.

So far, all of us have avoided covid-19. The county went into lockdown again in December, so there's no dining-in at our restaurants and schools have gone back to remote exclusively. I'm a bit surprised that gyms are still open, but they are now capped at 25% occupancy. Vaccines are being distributed but it's still just groups 1a and 1b: front-line medical workers and the residents of long-term care facilities. I'm not expecting a shot at a shot until late spring at best.

I was awarded my black belt in karate in mid-December. Like other belts I've earned since the dojo closed in March, I consider it to be provisional since we haven't been able to meet in person to do sparring, teaching of juniors, etc.. I plan to re-test when we finally get to practice together in one space once again.

Our ten-day Amsterdam to Bern river cruise last August was scrapped because of covid-19. However, Viking River Cruises offered us a 20% bonus credit if we rolled over to a 14-day Amsterdam-to-Budapest cruise in 2021, so we took them up on it. If we can't go because Europe is still closed to American tourists, or if we haven't been vaccinated by then, we can still get our money back, but I'm choosing to be optimistic.

Travel to Canada is still blocked. At this moment, I could enter Canada as a citizen, but I'd need a negative covid test within 72 hours to be able to board the plane, and submit an extensive quarantine plan electronically before arriving. Once there, I'd be quarantined for two weeks. I have plans ready in case I need to make urgent trips to southern Ontario or northern Ontario, but I'm hoping we get clear of this damn plague before I need to test either one.

Work continues. Our company is cleaning up all of the various bonus programs and performance metrics into a single cohesive plan, and it's not a bad plan at all. My next pay (Jan 8) will pay the last of the three year signing bonuses from our purchase by Enterprise Holdings Inc.; under the new plan, I'll be eligible for up to 20% of my base salary in annual bonuses. In the next month, all staff should receive a comprehensive benefits statement, including any pay adjustments due to title changes, adjustments to responsibilities, or to keep pace with industry averages. I have no idea what if any raise I might get, but the bonus is nothing to sneeze at, even if it's only half of what my last installment from the old plan will be.

Speaking of work, I've been on vacation these past two weeks. With Christmas and New Year's on Fridays, and with the regular company holidays, it only cost me seven vacation days. There is currently one more full day off before work resumes on Monday, and I'm savouring every hour.

I wish I had more to report about the plague year of 2020, but covid made it pretty boring. No square dancing, no travels, no skating, no visits, no bar nights, no day trips, no weekends away (except to the trailer where we'd self-isolate anyway), etc.. I have no idea if square dancing will bounce back, or in what form. I just want to be able to return to Canada again to see family.

Actually, things are pretty good!

Things still suck on so many levels: unemployment is hard to even measure, the economy is tanking in ways I've never seen before, and US deaths have topped 50,000 with no obvious end in sight. But as ugly as it all is, my attitude is vastly better than it was couple of weeks ago.

My job is still secure, as is Kent's. Michael's travel business is non-existent but that's not for lack of effort on his part. We've fallen into some stable & workable routines, and while I'd like better lighting in my basement bunker, it's all working pretty well.

We're all healthy, as is our extended family: we've heard of friends of friends down with covid-19, but no first or second degree connections.

Some supply shortages have been corrected, others haven't but don't affect us too much. At least the bare shelves aren't getting more bare: the shortages haven't spread into other products or stores. And the seniors' shopping hours are kinda fun if you can bear getting out of bed at such an early hour.

We miss going out to eat, but we've figured out which of our regular restaurants are open, and can handle the new curb-side pick-up procedures, among other variances. Tipping huge, needless to say, trying to help out the tipped wait staff as much as possible.

Zoom-based karate sessions haven't been too bad. It's nothing like being in an actual dojo with a proper non-cement floor and real people, but I'm able to keep a bottle of water nearby and make recordings so I can review new sequences in slow motion later. Michael's zoom-based yoga seems to be satisfactory as well. It's better than nothing.

The closed library system is a pain, but I've had a steady stream of books on my Kindle. I'd be happier if many of the books I wanted weren't already lent out, but I have several on hold and should be able to get them soon-ish.

I'm missing some of the square dance stuff, but not too badly. I like having many of my evenings back, and the lack of extra event coordination & overhead is nice.

I miss the gym. A Lot.

Mostly though, I miss travel beyond our immediate neighbourhood, especially the trailer at Roseland. Knowing that I can't fly to San Francisco for work, or to Canada for family is a fairly severe limitation.

But despite it all, we've largely acclimatized to this lower standard of living. Nationally, I think we've largely hit the floor after a long & seemingly endless plummet, so the bulk of the uncertainty has passed. Now we need to continue the distancing, keep large gathering places closed, and get a vaccine. None of these will happen quickly, but we can tough this out.

I got home from Canada on March 9, just as closures & lockdowns in the US were starting to take full effect. Michael arrived home from Ireland a week later, on March 16; he was three days early on his return, but international flight restrictions were being instituted and we wanted him home before every airline was grounded.

And since then? Well, nothing much to report.

I've been working from home as part of my general practice of the past 11 years. The San Francisco main office has been evacuated so everyone is now working from home, whether they want to or not. Business travel, our core business, is 20% of what it was this time last year so while our parent company Enterprise has told us to continue with our original product line & releases, hiring has been frozen. I did get a $10k raise just this past week, and my position is secure. Even if they did lay me off, they'd still be contractually obligated to give me a pile of severance, unpaid vacation time, and a large bonus already scheduled for next February.

Kent's office has gone to work-from-home as a matter of policy, replacing their grudgingly-permitting-it-only-once-per-week. They have done some staff reductions, but Kent's position was spared.

Michael's travel business has all but evaporated at this point: his Milan trip for the international LGBT travel agent convention is cancelled. We are still planning to go to Amsterdam in August, pandemic permitting.

Nearly all of our square dance gigs --dancing, calling & running events-- are cancelled indefinitely. The only thing left in the next several months is the IAGSDC convention in Denver in early July. Hopefully, the worst of this will be over by then.

Karate has also closed, but we started doing online sessions last night. It's been a huge comfort to see the gang together again, even if just virtually.

Weightlifting at the gym is my preferred method of de-stressing, but that hasn't been an option for two weeks now. Last week, I was checking websites to see if there was a gym ban in neighbouring states, but yup, there was.

I had hoped we could go to the trailer this coming weekend at Roseland, West Virginia. We haven't de-winterized yet: everything is still stored in our basement. Alas, yesterday, the shelter-at-home order was given for West Virginia, essentially closing the campground and restricting movement, so our trailer plans are dead.

Virginia and DC school systems have both announced their academic years are toast: the schools will not re-open before June. I expect our county at least in Maryland to do the same shortly. Most of our square dance clubs follow the school boards' open/closed policies, so we've lost our venues for a while longer. Ditto my karate dojo.

Back in Canada, things aren't much better: nearly every province has a shelter-at-home order, or nearly so. None of my family are sick: even Mom and Grandma are doing fine, just frustrated by being housebound. My brother and his wife were about to move from Nova Scotia to Ontario; while the sale & purchase of homes has closed, I don't know if they've found a moving company still available to actually do the move yet.

So far though, this has just been a major annoyance, not a crisis for us. The local grocery stores are still open and while there are shortages on some shelves, we're doing OK with things we can buy and the supplies we already have in stock. We had no pending medical appointments to cancel, and our prescriptions were already stocked. While our dentist's office is closed for elective stuff, they were open to finish outstanding dental projects, such as the final filling cap on my root canal of Feb 5, and the replacement crown installation for Kent this morning.

So I'm still working from the basement computer bunker, stess-eating a lot, going for four mile walks when weather permits, and reading more than I have in years. We'll get through this mess somehow.

I'm writing this on the plane from San Francisco back to DC after a week onsite at the SF office and the Santa Clara data center. And in all, it went well.

My old boss, Bret McGinnis, left the company at the end of December 2019. It had been a long time coming, as he and our current CTO didn't see eye-to-eye on a lot of high priority issues. Our CTO is classic Silicon Valley start-up: break things, fail forward, be disruptive, avoid committee, beg forgiveness rather than ask permission, etc.. Bret worshiped at the alter of 100% Uptime, discussed joint issues with other teams as needed, and kept a very close eye not just on security issues, but audit compliance topics. Our CTO gives lip service to the issue of our PCI and SOC audits, but flatly disregards the actual rules if they are inconvenient and expects somehow that Bret & the rest of us will find some way of bridging the impossible gaps.

Anyway, Bret is gone and we have a new interim boss, Josh Woodward, working remotely from Armonk, NY. He seems nice and listens well, but he's a contractor brought on a few months ago by our CTO to consult on software development project planning. The hope is that he'll agree to go fulltime with us, but I'm unsure if that will actually happen.

So while there is a sudden gap in our team, there is some effort to fill it, and I had some good one-on-one meetings with both the CTO and Josh this past week. I can work with them in various degrees of enthusiasm, depending on the topic.

On the good side, on Friday, I received the first retention bonus following our acquisition by Enterprise this time last year. In 2019, I was offered a pile of cash for my stock options and a retention bonus of 40% of my base salary in January 2020 and again in January 2021. After taxes and 401k, it's still north of $40k, so I'm happy to have it, especially if we decide to go ahead with the house renovations this year.

I hired a new guy for my systems engineering group recently. I originally wanted him to start Jan 6 so I would be in San Francisco to onboard him myself, but he really wanted to start asap --I think there were some financial pressures-- so he started Dec 23 instead. I finally met him last week and he has been working out pretty well. Interestingly, I hired him based on the content of his resume and our telephone interviews, as well as the onsite interviews by other people on the Operations team; his references checked out well too. When I finally met him, I learned he's 10 years older than me, about 63 or so. Silicon Valley looks askance at anyone older than 30, so I suddenly have an appreciation for how hard it was for him to land a new position, and the value of working from the data rather than from appearances. If I were slightly more shallow than I already am, I might have overlooked him too in the interview process.

Our search for a new network engineer was stalled, but the CTO has named me the hiring manager for that spot so I'm now working to pick up where Bret had left off. The sooner we get an extra person in that position, the better.

Our San Francisco office at 642 Harrison Street is going month-to-month on the lease as we seek cheaper digs elsewhere. There is a small office being built out in Campbell, CA (San Jose area), primarily for the executives who dwell in the south bay area. I visited that site Friday to see how the office space is being renovated and it's looking pretty good. Naturally, all of the networking equipment and videoconferencing systems are top-of-the-line and brand new: the execs insisted on that for themselves; the new Oakland office for the plebes will not be so lucky. Interestingly, while the executive space has side meeting rooms, they've still gone with mostly an open office environment, including a standing desk in the main room for the CEO.

The renovations are only just starting on the new Oakland space, two floors at 1300 Broadway, just above the 12th Street BART station. I looked around the area Saturday, just to see the location and the amenities in the neighbourhood, but had no access to the building itself. I don't know how comfortable the offices will be in the end, but no one will starve and the public transit accessibility is good.

Having testing the BART accessibility and timing to the airport, I've largely decided that future trips will still likely be via SFO rather than Oakland's airport. Yes, the Oakland airport will be vastly closer, but it's hard to find a non-stop flight from the east coast to Oakland at decent times on an non-sucking airline. The Alaska Air non-stops between Dulles and SFO have been largely good to me, and I can handle an hour on BART to Oakland: it's only four more stations than the Powell one I've been using to my San Francisco hotel. Of course, I'll need to find a decent hotel near the office that's within policy...

We're about to start a round of SOC2 audits & evidence collection once again. In fact, I'm supposed to be onsite in San Francisco for the week of February 24 to face the auditors directly. I should book that air fare & hotel soon.

There's a boatload of small projects still in flight, but I'm feeling pretty confident about my working environment, relations with colleagues, and my future in general. The departure of Bret and a couple more people has made some space for me to grow into other managerial functions as a director, and as long as I ask for more responsibility from Josh and Neil, I'm relatively confident I'll get it. As long as I don't screw up, I'll be making their lives easier.

The biggest challenge for me at the moment is a kind of time management. Working three hours ahead of California has permitted me to do some personal stuff in the mornings: gym, medical appointments, commuting to the data center, etc.. My day picks up at noon eastern time though as meetings start in San Francisco, and they tend to run into the early evening because, obviously, that's still the business day for SF. I have to be firm about not letting much slide past my 6pm so I can still have dinner with my family, attend karate, go square dancing, etc.. I need to be more efficient in the 12-6pm hours, and to make sure my colleagues on the other side are as efficient about that too.

I also suspect that as I take on more management, I'm going to be spending more time working with Bangalore, which means more late night hours. But I'm typically up until midnight anyway, so that's not a huge burden.

So many pieces in progress, but it's working out well.

My flight home from Canada a week ago was uneventful. Per my usual luck, I had to join a work-related defcon situation the moment I walked in the door at home: no one else in the company has experience working with Solaris, Veritas volume manager, or Veritas Cluster Service. Indeed, I'd rather not myself but until we phase out that stuff in a few months, I'm still the go-to guy.

It's been 9 days since Dad died. Oddly, I don't really feel it yet. I do have moments of sadness that I'll never see or talk to him again, but nothing I would call grief or intense emotion. I suspect a huge part is that once he passed away, my primary task was to keep Mom grounded and to help her get through the essential paperwork. In essence, Dad became a project rather than a death in the family.

The Christmas holidays and the sudden trip to northern Ontario have upset my usual routine considerably. It's not a bad or resentful upset though: I was glad to spend time with the kids, and always love going back home to see the farm although the circumstances of this trip were horrible. But it was nice to get back to karate last Friday: I've missed that. Heavy snows this weekend and a dental appointment this morning have kept me from the gym, but I'll probably be back tomorrow, the first time in nearly three weeks.

I missed little work over the past while. The load was light during the holidays, even though I was on-call. Almost nothing came up while I was away, although I did log in from time to time, just to read emails and to make a few tweaks here & there to help keep the production systems stable. My task list is only just now gearing back up to a normal load, and there are good things in progress.

So, it's all a return to normality & routine now. It just feels so anti-climactic.

I'm still at the same firm I've been at for ten years, Deem Inc.. Indeed, this past week was my 10th anniversary, if you count from when I first started doing work for them, Jan 5 is my 10th anniversary if you count from my official start date.

On January 25, I became the manager of our systems engineering team. I was already the senior member and was doing the equivalent of team manager since the prior fall, but this was when it was made official. Unfortunately, it did not come with extra money or stock options: management pleaded poverty. I accepted it however as it definitively shifted my job title from mere systems guy into a management roll; this is something which would look good on a resume should I move elsewhere, especially if the company was pleading poverty.

It has been a good year. Besides the day-to-day activities of keeping our systems running, we cleaned out & closed down the one cage in DC3 which I had originally been hired to support & manage ten years ago. We had switched over 90% of all activity to the new cage in May 2017, so we just had to be aggressive about moving that last 10%, then securely disposing of the old equipment.

The other big project was getting our PCI compliance certificate in the fall. Because our systems store credit card information of our customers, we are required to be certified by the paycard industry as compliant with their security guidelines. Some rules had changed since our last certification, and some of our systems had aged out of compliance on their own: it took a significant amount of effort just to collect the proof of compliance for our good systems, not to mention upgrading & modifying our non-compliant ones. In all, we were successful by the deadline but it was a huge effort.

There are still quarterly activities which must be taken as part of the compliance. I just received word yesterday that we have cleared the hurdle for this quarter. Good times will be here again around March 20.

On August 8, I was promoted from Manager, Systems Engineering, to Director, Operations. This promotion did come with extra money, and a pile of additional stock options. Now several teams report to me (and me to my senior VP, Bret). I'm spending more time in coordinating activities and in managerial meetings, but I still have my finger in the systems administration side of the house. I'm still the only person who lives even approximately near our production data center near Dulles Airport.

I had two trips to San Francisco this year, two more than last year. At the moment, there is a strong possibility of another trip for the first week of February, but I'm awaiting word to confirm that.

And while I can't write more about it here (or anywhere, really) because of confidentiality issues, there is something big and wonderful happening right now with work, and I'm very excited about it. I hope to write something a lot more informative by the end of January.

I started working for Deem, Inc. (formerly Rearden Commerce) in December, 2008. At that time, I had passed all the interviews, left my job at Fannie Mae and was taking a break of a few weeks before my official start date of Jan 5, 2009, but there were four guys moving equipment from a Savvis data center near Boston to one near Sterling, VA. My job would be too look after this hardware once they finished and returned to the San Francisco Bay area where they normally lived & worked. It would be advantageous to work with the new systems as they were being installed, and to meet the guys I'd be working with for the next while.

It is now nine years later, and I'm still at Deem, even when all of those guys have left for other jobs. And Tuesday, April 10, 2018, I completed the decommission & demolish the cage of equipment which we had collectively installed back in late 2008.

The new cage we had assembled in May, 2017, as our new production system occupied nine racks; the old systems occupied 21. The new cage uses about 1/5 the electricity the old one did. I'm certain our air conditioning requirements are also substantially smaller.

Decommissioning is both harder and easier than one might suspect. On the good side, one no longer need be gentle with equipment, especially hard drives. I'm so accustomed to holding equipment gingerly and ensuring it receives no shocks, physical, static or otherwise. When decommissioning though, it's rather nice to toss a hard drive or drop an entire stack into a bin instead of placing them carefully into padded trays. Taking the old equipment to the electronics recycling dumpster was a lot of fun, kinda like throwing a discus but with servers.

We brought in an outside vendor to shred the hard drives so we would have a certificate of destruction for our auditors, and naturally to protect the customer data which was on them (encrypted, naturally). In all, we shredded 608 drives.

Trashing an entire cage meant as well erasing a number of embarrassments. Yes, I did my best to keep cabling tidy & colour-coded, but sometimes we needed to cut corners because of urgency or a lack of parts. And over nine years, some systems are decommissioned, some new ones added: even when starting clean, it's harder to keep things neat as systems change organically. Trashing the cage removed all of the eyesores and little compromises.

Getting rid of old equipment also means fewer future trips to the data center. Fewer hard drives and newer hard drives means the failure rate overall has plummetted. Towards the end, I was making trips to the cage every 2-3 days, but now it's once every 2-3 weeks. I have to admit though some of the drives have been working 24/7 for at least nine years --some were still the originals from the Boston data center.

There was an obvious evolution of racking kits over the past nine years. The oldest equipment had rack nuts & bolts to hold trays in place on which the equipment would sit. Then they became rails held in place with rack nuts & bolts. Then we got simple rails which locked themselves in place without requiring pre-installed nuts & bolts. Most of the equipment required the nuts & bolts, so the electric screwdriver I had became my best friend ever: without, I'd still be removing equipment today.

The hardest part, besides lifting so much equipment, was disconnecting all of the ethernet and fibre cabling. It's a simple pinch to unlock the cable from the network port, then a slight pull to remove it. But do this several thousand times and your fingers get very worn and bruised. Booted/snagless cables are the worst: I have learned a new hatred for them. Add to this the occasional scrape, scratch or cut. Merely washing my hands was agony each time. It took a week before I could hold a pen comfortably, and nearly a month for my fingers to return to normal.

It was a long & glorious ride. The original equipment worked longer & harder than we had any right to ask of it. We had some scares & nightmares, but on the whole, it all worked well. Rest in peace.

For reasons unknown, someone somewhere decided that the web certificate admin account and my account at work should both be the subject of a major spam attack. These two email addresses were injected into a vast number of online forms, including email list subscription forms. Every one of these targets then generated at least one response, sometimes many, back to my accounts.

After waking up to 400 unexpected emails, I began setting up my own filters to trim out the excess while our Corporate IT folks worked with Exchange to see if they could do a better job of blocking these. Since that time, I've received another 8,500 messages, nearly all blocked or shunted into a spam folder.

This experience has revealed how much crappy web design is out there. Why would you run a sales ordering web site that accepts orders for hardware with no physical address entered? Why are there still mailing lists that allow anyone to subscribe someone else? And why do so many lists not yet send a click-to-confirm-you-actually-requested-this message? Why does any list still exist which doesn't tell you how to unsubscribe? And why does Outlook.com (our email platform) have systems to block senders, but not the senders' domain?

The biggest question for me though is: who profits from this? What gain could there possibly be for adding my name to thousands of mailing lists, nearly all in country I've never visited in languages I can't read? The list owners don't gain, I don't gain... who would benefit that they would undertake this kind of thing?

I think the worst is over... instead of 20+ messages per minute, the rate has received to a few per hour. My bigger worry now is checking that my filters weren't overly aggressive and accidentally swept up legitimate work emails.

Today, I was formally named Manager, Systems Engineering for Deem, Inc., my employer of nine years.

I've had five managers in syseng since I started, and now after all those years of working for The Man, I am now The Man --and working for a Different Man since I still have a VP to report to, naturally.

In a longer view, this is my return to management. I was the manager of Physics Computing Services when I worked for the University of Toronto. Of course, that was 22 years ago in another country, so that aged off my resume some time ago. If nothing else, I'm grateful for the management title as it legitimizes my less formal management experience when/if I go job hunting again in future.

It's not really much of a change overall since I was already informally the team manager since mid-November or so. My team is pretty small too: me, Chris in California, Leeno in Bangalore, and Allan (a prior holder of my current title) who is a contractor with us in California. And my projects & priorities haven't changed a great deal: I still will be looking after the DC3 data center, and have a hand in the SC4 data center. The only adjustment will be doing performance reviews and approving vacations & expenses, and representing our team at management meetings. 90% of my day-to-day work will be exactly the same.

A half-dozen people besides me were promoted today, but not one of us got a raise or extra stock options. Finance simply refuses to release the funds for that. We are informed that we're shortlisted collectively for any raises or bonuses if/when Finance ever relaxes the purse strings, but that may not happen for a while. I did receive a lovely $5 Starbucks gift card. I don't drink coffee, but

cuyhogarvr can use it.

My plan was to upgrade the RAM in three servers at the data center. They're at 64GB each right now, but we want to migrate some Windows Server virtual machines onto these boxes so bringing them up to 128GB would be a huge help. It seemed so easy a plan.

Alas, after driving to the data center, I learned the magnetic lock system on our cage isn't working. The
badge reader is reading, but it is unable to reach the database to confirm my authorization. The staff here learned it was offline earlier today but didn't have a fix yet.

So they issued me the override key which I could use to offline the magnetic lock. But for that to work, the override mechanism has to face the outside of the cage so it can be reached by a person standing outside. Our lock was installed facing inwards so it is unusable.

Their only option left is to physically power off the mag lock entirely so I can get into the cage. Or lift some floor tiles so I can crawl underneath (not recommended).

I'm giving up on it today as the RAM upgrades can wait. They promise a fix for the mag lock shortly, and will submit a work order to get the override lock facing the correct direction.

Yes, it's a Monday... why do you ask?

The past few months, I've been working on a major project for work: building out a new cage in our data center near Sterling, VA. Our current two cages are working reasonably well, but the equipment has aged, enough that much of it is no longer viable under Paycard Industry 3.2 standards (PCI). Even on the hardware we could keep using, we desperately need operating system upgrades and extra capacity.

The new cage has 9 racks, compared to the 27 racks in the old cage. Nearly everything is virtualized and clustered, all of it has the latest patches of whatever OS they're running, and we have RAM, storage & CPU cycles to spare.

And this past weekend, we went live.

It was a bit rocky in parts. One of the first tasks I had in the migration plan was to fix some issues in our Sun Microsystems/Oracle database servers, and our Veritas Cluster System. It took more hours than I was hoping/expecting, but I did get through it all. I think I spent more time delving into the depths of VCS that one night than I did the previous ten years.

By midday Saturday, everything had been migrated and we were starting running test traffic through it. We found some issues in routing, permissions, ownerships and such, but not many. Most effort was focused on getting the F5 traffic managers fully tuned for our requirements.

Today was our first regular business day since the cutover, and although we've had some problems getting our new IP ranges white-listed with a couple of our larger customers and had some performance problems with our hotel search databases, the day has been a success. We're getting great comments about the vastly improved speed & performance of our systems as well.

The road ahead is still a long one. While the travel portion of our systems have migrated, our Purchase and Car Service divisions have not yet. The Sun servers moved to the new cage, but their data still resides on a storage array in the old cage. At this moment, I'm still waiting for the license codes to built out a new monitoring system.

Once all of that has finished, there's a tonne of dismantling & disposal to do with the old cages & equipment. Some will be redeployed in our non-production environments, but 80% will be trashed completely. The most intensive part of the project is over, but I have work for the rest of the summer.

This morning, a fruit basket & candy package arrived, sent by my COO, CTO & head of HR, addressed to the "Jarvis Family." The note attached thanked my clan for supporting me as I worked extra hours on the latest set of office projects. Nice touch --although I'd rather have a small bonus rather than $50 gift package. Thoughtful, though.

I've been working extra hours (extra extra hours?) this week as we run down the clock to our big data center migration. Today, two people who now report to me (one is my former boss, on contract!) are flying from San Francisco to Sterling, VA. Tomorrow morning, I also head to Virginia to meet with them, ensure their badges & keys work at the data center, and generally show them the cages, servers & tools we have on-site for this migration. I also hope we can discuss in person the sequence of steps we're taking once the site goes offline Friday night, filling in any details I may have overlooked.

I'm staying at a hotel in Virginia Friday through Sunday so I can be as close as possible to the data center. I'm also expecting that after many extended hours of battle, I'd be in no shape for a 40 minute drive home and return the following day.

Friday night, about 10pm Easter time, our site will go down and the fun begins. In our preliminary testing, we were getting speed improvements of 5x or so, but I think that's just a happy dream of what we'll be able to do in another two months: even after the apps move from the old cage to the new, the databases will largely be reaching back to the storage arrays in the old cage over a 4Gbps fibre link until we can migrate the data. I do intend to start migrating the data after we go live this weekend, but it will take weeks of effort to finish that (I'm hoping to have the bulk of it done by July 1).

I cannot say how relieved I am to get the new cage online. It's not just that we've been working on it constantly the past several months, but we've been letting maintenance of the old cage slide a bit, and it had inherent issues we couldn't easily fix anyway. Killing the old cage removes a lot of legacy equipment & unfortunate architecture decisions: the slate gets swept clean.

And even when we shut down the old cage, there is still much to do. We could only make this deadline by physically moving the Sun T4-1 servers with their Oracle databases to the new cage. We were originally planning to let that equipment be retired, but that aspect is a huge project in itself. In the next three month work sprint, we're going to: identify the data we need to retain, convert the data from SPARC data word format into Intel data word format, and restructure the database & LUN layout. All on live systems. The database team has 80% of this workload, but I'm still in the mix, doing the storage allocations and helping where I can with optimizing the process.

At this moment, here and now, I'm feeling rather serene. I've just ticked off the last of my pre-migration tasks, and finished scripting a lot of Veritas cluster stuff I need to do the moment the site is offline Friday night. In all, I'm now in wait mode. There is nothing left to do but wait for the dawn.

Folks are storing their data in the cloud, things are happening in the cloud, our businesses are becoming more cloud-based, etc..

Great buzzwords. Still a lot of hype.

There is no cloud, just other peoples' computers. There are still servers and hard drives out there, stuffed into data centers: you just get to rent a little part of it to store your files. The "cloud" nomenclature was created to demonstrate directly that you, the customer, have absolutely no idea where those servers, drives, and your data actually are. The sales pitch is that you don't need to know, the reality is that you can't know.

As one who has worked in major data centers for decades, trust me: I know where the cloud actually is. I've had a direct hand in building small portions of it. Indeed, since each machine has a number of sharp corners & edges, I've had more than a few injuries getting the equipment assembled & racked, and I'm of course not the only one.

It's an interesting thought: the 'cloud' contains not just your data, but an awful lot of very real blood, my own included.

We survived the weekend in Rehoboth Beach, DE. Travel to Rehoboth is always a dodgy thing and it's not a place I enjoy, so I'm relieved we're done for another year.

My boss flew into town late last night to work in the data center. We'll be doing some long hours Tuesday, Wednesday & Thursday; he flies back to California Friday morning. I'm going to help on some of the network stuff, but my major tasks are:
- moving four Sun servers from the old cage to the new one;
- configure said servers;
- install 20 new servers (due to arrive any day now);
- install 4.8TB of RAM in the form of 16GB DIMMs. Half are replacements for 8GB DIMMs in 35 machines, the rest are upgrades to existing servers, taking them to 128GB of RAM each. It's overkill for what we need today but will give us serious room to grow for the next couple of years.

Come Thursday, Bill Eyler will be coming to stay chez nous as we shuttle him around to various square dance gigs in the greater DC area. The recent minivan engine trouble was worrisome but we're back on schedule now that the PrincessMobile is fully repaired.

Speaking of square dancing, I've submitted my availability for the Zig Zagger's calling schedule for 2017-18. I've also sent my preferences for slots at the upcoming IAGSDC event in Palm Springs, CA, although I don't expect to hear back on that for another month or so.

Yesterday at work, we received an email from a former employee who left the firm about three months ago. Apparently, he was getting deluged with automated alert txt messages from our systems and wanted them to stop.

Digging into it, we were at first mystified: he's not in any of the recipient lists in our monitoring packages or even email distribution lists. He's not in the corporate directories or any other sources of record. Then we also realized that the messages were for systems we had decommissioned and removed from our monitoring tools. So where were these being generated?

After some considerable effort, we discovered that the bozo had:
1. created a monitoring script for his production environment which he didn't document;
2. the monitoring script wasn't folded into our suite of monitoring tools so it wasn't using our alert management & scheduling systems;
3. he was running this monitoring script for the prod env from a dev workstation, not the proper prod services;
4. he hard-coded his personal contact info into the script so that he alone would get the alerts.

In short, he created the very mess he was now complaining to us to have fixed for him, and he did it in the most incredibly unprofessional means possible.

While I'm sympathetic that his mobile is getting flooded with txt messages and costing him a bundle if he doesn't have unlimited messaging, my sympathy ends there. He built this mess for himself and I was sorely tempted to let him wallow in it a while longer as a lesson in how not to do things.

It's been a busy few days but I'm slowly catching up.

We've been slowly decommissioning one old storage array at the data center, requiring us to move the data to a newer array. We discovered some performance issues this week so we had to migrate two particular volumes to a different RAID pool but these activities required essentially two all-nighters this week.

That wouldn't be catastrophic but I also had my regular build work for the new data center cage, and since I'm the only employee on the east coast, it's not going to get done so long as I'm being sucked into these other spontaneous demands. That was unavoidable in this instance but I've made it clear to the dev teams that I'm only available to them for major issues, not trivial ones for the rest of this month.

And as life would have it, I had a few square dance calling gigs this week too: our Wednesday C2 group, a special C1 night for the DC Lambda Squares, a Friday evening holiday party called by John Marshall which I really wanted to attend, and co-calling a six hour C2 event Saturday morning & afternoon with Kent.

I'm happy to report that all of the tasks for the week were accomplished successfully, although at the expense of my gym workout schedule. Still, that's a small price to pay for the pleasure of knowing the other items are under control.

Today's migration isn't about data, but relocating my computer bunker from the basement to the first floor sewing room & middle bedroom. The basement bunker is convenient and optimized for my work, but it gets cold down there during the winter. A small electric heater helps but it has to run nearly constantly to keep the room comfortable. It's easier (and cheaper) just to work from the main floor bedroom until spring.

In all, life should be a bit more stable & normal for a couple of weeks. I hope.

I read rumours this morning that Oracle was going to be shutting down all further Solaris development.
Solaris being canned, at least 50% of teams to be RIF'd in short term. All hands meetings being cancelled on orders from legal to prevent news from spreading. Hardware teams being told to cease development. There will be no Solaris 12, final release will be 11.4. Orders coming straight from Larry.

Even if development is stopped, there is still promised support for existing versions for a couple more years, but once the last version runs its course, the game is over.

I have mixed feelings about this, if it is true. I've been with Sun Microsystems since the Sun 3 line and SunOS 3.5, back in the 1980s when the Motorola 68000 CPU was hot stuff. Hell, in those heady days, the OS included a compiler! The machines were sturdy, the screens were huge (cathode ray tubes, naturally) and while they were expensive, they sold like hot cakes. I worked for a Sun VAR in Toronto in the early 1990s, then for the University of Toronto caring for a Sun 3/280 server.

The transition to SPARC and the Sun 4 line was joyful and traumatic. I loved the faster & more powerful CPUs, and the upgrade of our machine was as simple as swapping out a VME board. I did not love Solaris, however. Yeah, SunOS 4.1.5 at that time needed a complete refresh to handle newer communications technologies, extra cores, multi-CPU architectures and such, but it was a solid OS and worked well. Slowlaris was a painfully poor performer and a resource pig by comparison. And it didn't come on quarter-inch tapes: one had to lay down serious money for a CD drive since that was the only distribution method available. And adding insult to injury, it didn't have a development environment by default: it was an extra.

Over the years, my Sun 4/280 gained extra memory and SCSI drives. It was running better than ever, albeit two versions of Solaris later.

After some extra years, a couple of extra jobs and a move to the US, I landed at Fannie Mae for ten years. We were told Fannie Mae was the second largest Sun customer on the east coast (after NASA): I was part of the team which built and maintained their MornetPlus system, mostly built on Sun 250 and Sun 450 machines for data processing and a large pair of Sun 6800 machines for their core cluster. The 6800 machines were standalone, but the 450 models would fit two to a rack --and they weighed a tonne. We were mostly running Solaris 2.6 when I arrived, transitioned to Solaris 8 during my tenure, and began migrating to Solaris 10 as I left (now eight years ago). I loved having a single operating system for our entire enterprise: it made support so much easier, and Solaris 8 was again pretty solid.

While I used Solaris 10 at Fannie Mae and again at Talaris/Rearden Commerce/Deem where I work currently, I've never loved it. Solaris 10 and I tolerated each other. It felt snobbish and repressed. It ran solidly and had some interesting new features (introducing zones), but other kids on the block (eg Linux) seemed to be moving faster and offered more flexibility. And most of all, the new kids were vastly cheaper.

Fannie Mae paid an enormous amount to Sun Microsystems every year for support. Millions of dollars. Oracle bought up Sun Microsystems and continued to support Solaris and release new models of the Sun hardware, but they added their own special Oracle DNA, that is, their desperate desire to drain customers of every penny they had. Support costs soared and purchase prices spiked, although discounts sometimes be had if you bundled together other Oracle products, especially their software.

Even now, I'm typing this while monitoring a storage issue on a Sun T4-1 machine running Solaris 10. It's fine, nothing much to write about. But we're also building a new data center cage, refreshing our entire hardware base and allowing us to retire & scrap our old systems by spring of 2017. Sun will not be part of the new cage: the Solaris stops here.

As I said, Solaris 10 and I never loved each other, but after Larry Ellison got his mitts on it all, I knew it was time for me to start dating other operating systems. Our on-again-off-again affair had run its full course.

So reading that Oracle is tossing in the metaphoric towel on Solaris (and presumably the hardware line too) is like seeing an obituary notice in the newspaper for an old boyfriend. It's a sad thing and I'll remember the good times, but I let go along time ago.

I'd like to say I had a really great long weekend. Really, I'd love nothing better than to say that. Truly, I would.

But lest one thinks otherwise, I should immediately say that it was a perfectly fine long weekend. It just wasn't the weekend I was looking forward to.

Our office was supposed to be shut down for the entirety of US Thanksgiving week to force all employees to burn up their vacation days as we transition to a would-be unlimited vacation policy. And while that worked for most of the company, I was hauled back from vacation mode on Monday and Tuesday to deal with various technical issues. One of the major problems was that while the US operation was on vacation, the Bangalore portion was working a normal schedule and demanded that any service they needed from the US side be made instantly available as they had their own deadlines to meet.

I'm fine with dealing with the occasional unforeseen hardware issue --random things happen. I'm a bit pissed that expectations between the US and India teams were so poorly communicated that my division, supposedly on vacation, were essentially just made to work from home. I'm having a few words with management tomorrow: I want my Monday and Tuesday back, either not being counted as vacation or given two lieu days to make up for this farce.

We saw the movie "Arrival" on Thursday. Quick review: I liked it. It was more cerebral than a lot of other films on offer, and my inner linguist was fascinated by the challenges of learning a language from an entity with whom one has literally nothing in common. There was a major item of scifi silliness which irks me a bit, but it was a construct essential to the storytelling so I'm willing myself to let it slide --watch the movie and you'll understand what I mean.

I wish I could say what I did on Friday but I can't for the life of me think what it was. Which probably means napping, reading and relaxing.

Maurita & Lucas and their daughter Elodie (now 17 months old) came for Thanksgiving dinner on Saturday. Elodie was a delight: she's so much fun. We had the traditional turkey and a stack of low-carb, gluten-free sides which the Pilgrims would never recognize. Tasted good though, so what else matters? In an unusual burst of restraint, I didn't stuff myself to the point of explosion. And there will be turkey leftovers for days.

I even got the gym regularly during this past week: every day except Tuesday morning and today. I typically take a rest day after chest day as the chest, shoulders and arms take quite a pounding a need a brief recovery space before the next workout. If I'm clever, I try to book my data center visits or outside appointments on those recovery days to help smooth out my calendar.

This day has been more relaxing and tinkering, and I'm glad of it as the following week is going to be heavy. I've cleaned up a lot of files on my laptop, freeing up another 16GB of disk space. I've finally sorted through a stack of digital photos, cleaning 4GB of them off my mobile phone and filing them away on an external archive. At this moment, I have some cleanups in progress on my Google Drive too.

After being inactive for the past 2-3 weeks, I'm back into square dance calling this week in a big way. We have our regular C2 group meeting on Wednesday, I'm calling a C1/C2 night for the DC Lambda Squares Thursday, and Kent & I are co-calling an all-day C2 session for our crowd & friends Saturday. John Marshall is calling a holiday party Friday evening so we'll likely attend that as well. In all, my reduced calling schedule has helped my professional schedule immensely, and I hope to keep this new balance as long as I can.

Sadly, tomorrow is Monday, with all the excitement that entails. More news as it develops.

Our Car Service division is a mess of disorganization. It has been run as a separate division of our firm since we acquired them about 4-5 years ago and as such, I've had little interaction with them.

Three senior people from Car Service left the firm in the past couple of months so there's been movement to merge the support of their Windows servers & infrastructure into the Operations group of the rest of the firm. This makes sense and I'm happy to be part of it.

Except that now I get to see how the sausage is being made. Or rather, I now see that the sausage isn't always sausage, it isn't necessarily being made or made on time, or isn't made out of meat or even something organic, or isn't even made on the equipment we thought it was.

What has been hidden from view is that the Windows Servers are creaking under their own weight and have frequent undetected/unreported hardware errors. We have also discovered the apps need to be touched or helped by a human being every few hours or the app will simply die. Nothing is automated, monitoring is sparse at best, there are no performance monitors at all, and nearly nothing was documented. This is a classic case study in how not to do business.

But all of this isn't news: we've been repairing & cleaning up this mess for much of October and have made great progress.

Today was another surprise. A domain I've never heard of (transponet.com) apparently expired last night. Huh? A look into the Whois records doesn't help: the registration info is just a privacy proxy firm in Florida, shielding the actual registration holder & contact information. The domain isn't listed in our portfolio of domain names and we never received a renewal notice.

In the massive email thread which ensued, we have a strong suspicion that the domain name was registered by one of the software engineers in their personal account, but was never transferred to the corporate account as they should have. They probably did this as a convenience many years ago and it slipped through the cracks like so many other things. Unfortunately, the engineer we suspect did this died two years ago. Because of that, he's not available to renew the domain or even transfer it to our corporate account for renewal. *sigh*

We're now working legal channels to see about getting this transferred as needed. The people who have access to various mailboxes & files of departed employees aren't in the office yet so I can't check for any renewal notices or even confirmation of our suspicions on how the domain was registered. At this moment, I'm at a dead stop.

Things like this aren't supposed to happen. It's a profound embarrassment & shame upon the clowns who created this situation, and the rest of us for not catching it in time. Needless to say, the balance of my day is going to be spent scanning all of the source code in that division to look for any other domain surprises and ensuring all of them are in our corporate portfolio where they can be properly managed.

Update: We've renewed the problem domain but are still trying to transfer it and several others to our regular portfolio. The snag we're finding in many of these domains is that the registrant contact information is incorrect: there's a typo in the email address! We're now in the process of getting around this by registering the typoed domain name, then creating MX records to point it to our corporate email server. *sigh*

We use Akamai as a content distribution network. For the non-geeky crowd, this means that some of the items in our firm's web applications are distributed to the planet by Akamai: when a user in, say, the UK, uses our application, the graphics, images & logos in our app are retrieved from a local server in the UK close to the user rather than reaching across the planet to get a copy from our server in Virginia. This gives the user a faster web experience, and it eases the load on our web servers in Virginia.

Naturally, we use web certificates so users can be certain (a) they are getting data from an authorized source instead of Someone Evil, and (b) it is encrypted for privacy, (c) it is secured for consistency to ensure the data wasn't manipulated in transit.

We have several web certificates for many domains in Akamai. Three of the certs will expire this week so Akamai contacted us for approval to renew them. I've been trying to tell them that we don't need the certs and to let them go, but they just don't believe me.

Akamai: There's three three certs which are gonna expire. Let's coordinate a time to talk to the cert vendor to renew them!
Me: Not needed. We don't use those domains so we can let the certs expire and save money.
A: But there's customer traffic!
Me: Not from us. We don't use those domains.
A: But there's customer traffic!
Me: If any, it's probably web bots and spiders. Don't care.
A: But there's customer traffic! Look, one domain name is CNAMEd traffic to our caches!
Me: Yes, the data is in the zone files, but we don't use those domains.
A: But there's customer traffic! You need to renew immediately!
Me: No, that's not us. We don't care. *Logs into the Akamai control panel, removes the certs. Updates DNS to remove any reference to Akamai in those zones* Look, I've wiped out the references to Akamai. We don't use the domains, we don't care about the certs, we want to them to expire.
A: But there's customer traffic! It will all fail now!
Me: What can I do to persuade you we aren't using these services and can let them expire?
A: *silence*

Profile

bjarvis

Brian's web page

January 2021

S	M	T	W	T	F	S
					1	2
3	4	5	6	7	8	9
10	11	12	13	14	15	16
17	18	19	20	21	22	23
24	25	26	27	28	29	30
31

Syndicate

Page Summary

Style Credit

Style: by timeasmymeasure

Expand Cut Tags

No cut tags

Page generated Dec. 28th, 2025 07:16 am

Are We Having Fun Yet?

Entries tagged with work

2021? Already?!

COVID-19 Lockdown Day #47

Surviving the Lockdown

Work Status, January 2020

In Today's News...

2018 In Summary: Work

Demolition & Data Centers

Mega-Spam

Promotion!

Good Security Gone Bad

It is Done.

Office Madness

There is No Cloud

Better Moods

File Under "Self-Inflicted Wounds"

Migrations in Process

Solaris EOL

Post-Thanksgiving

Avoidable Office Crisis of the Day

OMFG, Akamai!