bjarvis: (Default)
The past few months, I've been working on a major project for work: building out a new cage in our data center near Sterling, VA. Our current two cages are working reasonably well, but the equipment has aged, enough that much of it is no longer viable under Paycard Industry 3.2 standards (PCI). Even on the hardware we could keep using, we desperately need operating system upgrades and extra capacity.

The new cage has 9 racks, compared to the 27 racks in the old cage. Nearly everything is virtualized and clustered, all of it has the latest patches of whatever OS they're running, and we have RAM, storage & CPU cycles to spare.

And this past weekend, we went live.

It was a bit rocky in parts. One of the first tasks I had in the migration plan was to fix some issues in our Sun Microsystems/Oracle database servers, and our Veritas Cluster System. It took more hours than I was hoping/expecting, but I did get through it all. I think I spent more time delving into the depths of VCS that one night than I did the previous ten years.

By midday Saturday, everything had been migrated and we were starting running test traffic through it. We found some issues in routing, permissions, ownerships and such, but not many. Most effort was focused on getting the F5 traffic managers fully tuned for our requirements.

Today was our first regular business day since the cutover, and although we've had some problems getting our new IP ranges white-listed with a couple of our larger customers and had some performance problems with our hotel search databases, the day has been a success. We're getting great comments about the vastly improved speed & performance of our systems as well.

The road ahead is still a long one. While the travel portion of our systems have migrated, our Purchase and Car Service divisions have not yet. The Sun servers moved to the new cage, but their data still resides on a storage array in the old cage. At this moment, I'm still waiting for the license codes to built out a new monitoring system.

Once all of that has finished, there's a tonne of dismantling & disposal to do with the old cages & equipment. Some will be redeployed in our non-production environments, but 80% will be trashed completely. The most intensive part of the project is over, but I have work for the rest of the summer.
bjarvis: (Default)
This morning, a fruit basket & candy package arrived, sent by my COO, CTO & head of HR, addressed to the "Jarvis Family." The note attached thanked my clan for supporting me as I worked extra hours on the latest set of office projects. Nice touch --although I'd rather have a small bonus rather than $50 gift package. Thoughtful, though.

I've been working extra hours (extra extra hours?) this week as we run down the clock to our big data center migration. Today, two people who now report to me (one is my former boss, on contract!) are flying from San Francisco to Sterling, VA. Tomorrow morning, I also head to Virginia to meet with them, ensure their badges & keys work at the data center, and generally show them the cages, servers & tools we have on-site for this migration. I also hope we can discuss in person the sequence of steps we're taking once the site goes offline Friday night, filling in any details I may have overlooked.

I'm staying at a hotel in Virginia Friday through Sunday so I can be as close as possible to the data center. I'm also expecting that after many extended hours of battle, I'd be in no shape for a 40 minute drive home and return the following day.

Friday night, about 10pm Easter time, our site will go down and the fun begins. In our preliminary testing, we were getting speed improvements of 5x or so, but I think that's just a happy dream of what we'll be able to do in another two months: even after the apps move from the old cage to the new, the databases will largely be reaching back to the storage arrays in the old cage over a 4Gbps fibre link until we can migrate the data. I do intend to start migrating the data after we go live this weekend, but it will take weeks of effort to finish that (I'm hoping to have the bulk of it done by July 1).

I cannot say how relieved I am to get the new cage online. It's not just that we've been working on it constantly the past several months, but we've been letting maintenance of the old cage slide a bit, and it had inherent issues we couldn't easily fix anyway. Killing the old cage removes a lot of legacy equipment & unfortunate architecture decisions: the slate gets swept clean.

And even when we shut down the old cage, there is still much to do. We could only make this deadline by physically moving the Sun T4-1 servers with their Oracle databases to the new cage. We were originally planning to let that equipment be retired, but that aspect is a huge project in itself. In the next three month work sprint, we're going to: identify the data we need to retain, convert the data from SPARC data word format into Intel data word format, and restructure the database & LUN layout. All on live systems. The database team has 80% of this workload, but I'm still in the mix, doing the storage allocations and helping where I can with optimizing the process.

At this moment, here and now, I'm feeling rather serene. I've just ticked off the last of my pre-migration tasks, and finished scripting a lot of Veritas cluster stuff I need to do the moment the site is offline Friday night. In all, I'm now in wait mode. There is nothing left to do but wait for the dawn.
bjarvis: (Default)
Folks are storing their data in the cloud, things are happening in the cloud, our businesses are becoming more cloud-based, etc..

Great buzzwords. Still a lot of hype.

There is no cloud, just other peoples' computers. There are still servers and hard drives out there, stuffed into data centers: you just get to rent a little part of it to store your files. The "cloud" nomenclature was created to demonstrate directly that you, the customer, have absolutely no idea where those servers, drives, and your data actually are. The sales pitch is that you don't need to know, the reality is that you can't know.

As one who has worked in major data centers for decades, trust me: I know where the cloud actually is. I've had a direct hand in building small portions of it. Indeed, since each machine has a number of sharp corners & edges, I've had more than a few injuries getting the equipment assembled & racked, and I'm of course not the only one.

It's an interesting thought: the 'cloud' contains not just your data, but an awful lot of very real blood, my own included.
bjarvis: (Default)
We survived the weekend in Rehoboth Beach, DE. Travel to Rehoboth is always a dodgy thing and it's not a place I enjoy, so I'm relieved we're done for another year.

My boss flew into town late last night to work in the data center. We'll be doing some long hours Tuesday, Wednesday & Thursday; he flies back to California Friday morning. I'm going to help on some of the network stuff, but my major tasks are:
- moving four Sun servers from the old cage to the new one;
- configure said servers;
- install 20 new servers (due to arrive any day now);
- install 4.8TB of RAM in the form of 16GB DIMMs. Half are replacements for 8GB DIMMs in 35 machines, the rest are upgrades to existing servers, taking them to 128GB of RAM each. It's overkill for what we need today but will give us serious room to grow for the next couple of years.

Come Thursday, Bill Eyler will be coming to stay chez nous as we shuttle him around to various square dance gigs in the greater DC area. The recent minivan engine trouble was worrisome but we're back on schedule now that the PrincessMobile is fully repaired.

Speaking of square dancing, I've submitted my availability for the Zig Zagger's calling schedule for 2017-18. I've also sent my preferences for slots at the upcoming IAGSDC event in Palm Springs, CA, although I don't expect to hear back on that for another month or so.
bjarvis: (Default)
Yesterday at work, we received an email from a former employee who left the firm about three months ago. Apparently, he was getting deluged with automated alert txt messages from our systems and wanted them to stop.

Digging into it, we were at first mystified: he's not in any of the recipient lists in our monitoring packages or even email distribution lists. He's not in the corporate directories or any other sources of record. Then we also realized that the messages were for systems we had decommissioned and removed from our monitoring tools. So where were these being generated?

After some considerable effort, we discovered that the bozo had:
1. created a monitoring script for his production environment which he didn't document;
2. the monitoring script wasn't folded into our suite of monitoring tools so it wasn't using our alert management & scheduling systems;
3. he was running this monitoring script for the prod env from a dev workstation, not the proper prod services;
4. he hard-coded his personal contact info into the script so that he alone would get the alerts.

In short, he created the very mess he was now complaining to us to have fixed for him, and he did it in the most incredibly unprofessional means possible.

While I'm sympathetic that his mobile is getting flooded with txt messages and costing him a bundle if he doesn't have unlimited messaging, my sympathy ends there. He built this mess for himself and I was sorely tempted to let him wallow in it a while longer as a lesson in how not to do things.
bjarvis: (Default)
It's been a busy few days but I'm slowly catching up.

We've been slowly decommissioning one old storage array at the data center, requiring us to move the data to a newer array. We discovered some performance issues this week so we had to migrate two particular volumes to a different RAID pool but these activities required essentially two all-nighters this week.

That wouldn't be catastrophic but I also had my regular build work for the new data center cage, and since I'm the only employee on the east coast, it's not going to get done so long as I'm being sucked into these other spontaneous demands. That was unavoidable in this instance but I've made it clear to the dev teams that I'm only available to them for major issues, not trivial ones for the rest of this month.

And as life would have it, I had a few square dance calling gigs this week too: our Wednesday C2 group, a special C1 night for the DC Lambda Squares, a Friday evening holiday party called by John Marshall which I really wanted to attend, and co-calling a six hour C2 event Saturday morning & afternoon with Kent.

I'm happy to report that all of the tasks for the week were accomplished successfully, although at the expense of my gym workout schedule. Still, that's a small price to pay for the pleasure of knowing the other items are under control.

Today's migration isn't about data, but relocating my computer bunker from the basement to the first floor sewing room & middle bedroom. The basement bunker is convenient and optimized for my work, but it gets cold down there during the winter. A small electric heater helps but it has to run nearly constantly to keep the room comfortable. It's easier (and cheaper) just to work from the main floor bedroom until spring.

In all, life should be a bit more stable & normal for a couple of weeks. I hope.

Solaris EOL

Dec. 2nd, 2016 04:46 am
bjarvis: (Default)
I read rumours this morning that Oracle was going to be shutting down all further Solaris development.
Solaris being canned, at least 50% of teams to be RIF'd in short term. All hands meetings being cancelled on orders from legal to prevent news from spreading. Hardware teams being told to cease development. There will be no Solaris 12, final release will be 11.4. Orders coming straight from Larry.

Even if development is stopped, there is still promised support for existing versions for a couple more years, but once the last version runs its course, the game is over.

I have mixed feelings about this, if it is true. I've been with Sun Microsystems since the Sun 3 line and SunOS 3.5, back in the 1980s when the Motorola 68000 CPU was hot stuff. Hell, in those heady days, the OS included a compiler! The machines were sturdy, the screens were huge (cathode ray tubes, naturally) and while they were expensive, they sold like hot cakes. I worked for a Sun VAR in Toronto in the early 1990s, then for the University of Toronto caring for a Sun 3/280 server.

The transition to SPARC and the Sun 4 line was joyful and traumatic. I loved the faster & more powerful CPUs, and the upgrade of our machine was as simple as swapping out a VME board. I did not love Solaris, however. Yeah, SunOS 4.1.5 at that time needed a complete refresh to handle newer communications technologies, extra cores, multi-CPU architectures and such, but it was a solid OS and worked well. Slowlaris was a painfully poor performer and a resource pig by comparison. And it didn't come on quarter-inch tapes: one had to lay down serious money for a CD drive since that was the only distribution method available. And adding insult to injury, it didn't have a development environment by default: it was an extra.

Over the years, my Sun 4/280 gained extra memory and SCSI drives. It was running better than ever, albeit two versions of Solaris later.

After some extra years, a couple of extra jobs and a move to the US, I landed at Fannie Mae for ten years. We were told Fannie Mae was the second largest Sun customer on the east coast (after NASA): I was part of the team which built and maintained their MornetPlus system, mostly built on Sun 250 and Sun 450 machines for data processing and a large pair of Sun 6800 machines for their core cluster. The 6800 machines were standalone, but the 450 models would fit two to a rack --and they weighed a tonne. We were mostly running Solaris 2.6 when I arrived, transitioned to Solaris 8 during my tenure, and began migrating to Solaris 10 as I left (now eight years ago). I loved having a single operating system for our entire enterprise: it made support so much easier, and Solaris 8 was again pretty solid.

While I used Solaris 10 at Fannie Mae and again at Talaris/Rearden Commerce/Deem where I work currently, I've never loved it. Solaris 10 and I tolerated each other. It felt snobbish and repressed. It ran solidly and had some interesting new features (introducing zones), but other kids on the block (eg Linux) seemed to be moving faster and offered more flexibility. And most of all, the new kids were vastly cheaper.

Fannie Mae paid an enormous amount to Sun Microsystems every year for support. Millions of dollars. Oracle bought up Sun Microsystems and continued to support Solaris and release new models of the Sun hardware, but they added their own special Oracle DNA, that is, their desperate desire to drain customers of every penny they had. Support costs soared and purchase prices spiked, although discounts sometimes be had if you bundled together other Oracle products, especially their software.

Even now, I'm typing this while monitoring a storage issue on a Sun T4-1 machine running Solaris 10. It's fine, nothing much to write about. But we're also building a new data center cage, refreshing our entire hardware base and allowing us to retire & scrap our old systems by spring of 2017. Sun will not be part of the new cage: the Solaris stops here.

As I said, Solaris 10 and I never loved each other, but after Larry Ellison got his mitts on it all, I knew it was time for me to start dating other operating systems. Our on-again-off-again affair had run its full course.

So reading that Oracle is tossing in the metaphoric towel on Solaris (and presumably the hardware line too) is like seeing an obituary notice in the newspaper for an old boyfriend. It's a sad thing and I'll remember the good times, but I let go along time ago.
bjarvis: (Default)
I'd like to say I had a really great long weekend. Really, I'd love nothing better than to say that. Truly, I would.

But lest one thinks otherwise, I should immediately say that it was a perfectly fine long weekend. It just wasn't the weekend I was looking forward to.

Our office was supposed to be shut down for the entirety of US Thanksgiving week to force all employees to burn up their vacation days as we transition to a would-be unlimited vacation policy. And while that worked for most of the company, I was hauled back from vacation mode on Monday and Tuesday to deal with various technical issues. One of the major problems was that while the US operation was on vacation, the Bangalore portion was working a normal schedule and demanded that any service they needed from the US side be made instantly available as they had their own deadlines to meet.

I'm fine with dealing with the occasional unforeseen hardware issue --random things happen. I'm a bit pissed that expectations between the US and India teams were so poorly communicated that my division, supposedly on vacation, were essentially just made to work from home. I'm having a few words with management tomorrow: I want my Monday and Tuesday back, either not being counted as vacation or given two lieu days to make up for this farce.

We saw the movie "Arrival" on Thursday. Quick review: I liked it. It was more cerebral than a lot of other films on offer, and my inner linguist was fascinated by the challenges of learning a language from an entity with whom one has literally nothing in common. There was a major item of scifi silliness which irks me a bit, but it was a construct essential to the storytelling so I'm willing myself to let it slide --watch the movie and you'll understand what I mean.

I wish I could say what I did on Friday but I can't for the life of me think what it was. Which probably means napping, reading and relaxing.

Maurita & Lucas and their daughter Elodie (now 17 months old) came for Thanksgiving dinner on Saturday. Elodie was a delight: she's so much fun. We had the traditional turkey and a stack of low-carb, gluten-free sides which the Pilgrims would never recognize. Tasted good though, so what else matters? In an unusual burst of restraint, I didn't stuff myself to the point of explosion. And there will be turkey leftovers for days.

I even got the gym regularly during this past week: every day except Tuesday morning and today. I typically take a rest day after chest day as the chest, shoulders and arms take quite a pounding a need a brief recovery space before the next workout. If I'm clever, I try to book my data center visits or outside appointments on those recovery days to help smooth out my calendar.

This day has been more relaxing and tinkering, and I'm glad of it as the following week is going to be heavy. I've cleaned up a lot of files on my laptop, freeing up another 16GB of disk space. I've finally sorted through a stack of digital photos, cleaning 4GB of them off my mobile phone and filing them away on an external archive. At this moment, I have some cleanups in progress on my Google Drive too.

After being inactive for the past 2-3 weeks, I'm back into square dance calling this week in a big way. We have our regular C2 group meeting on Wednesday, I'm calling a C1/C2 night for the DC Lambda Squares Thursday, and Kent & I are co-calling an all-day C2 session for our crowd & friends Saturday. John Marshall is calling a holiday party Friday evening so we'll likely attend that as well. In all, my reduced calling schedule has helped my professional schedule immensely, and I hope to keep this new balance as long as I can.

Sadly, tomorrow is Monday, with all the excitement that entails. More news as it develops.
bjarvis: (Default)
Our Car Service division is a mess of disorganization. It has been run as a separate division of our firm since we acquired them about 4-5 years ago and as such, I've had little interaction with them.

Three senior people from Car Service left the firm in the past couple of months so there's been movement to merge the support of their Windows servers & infrastructure into the Operations group of the rest of the firm. This makes sense and I'm happy to be part of it.

Except that now I get to see how the sausage is being made. Or rather, I now see that the sausage isn't always sausage, it isn't necessarily being made or made on time, or isn't made out of meat or even something organic, or isn't even made on the equipment we thought it was.

What has been hidden from view is that the Windows Servers are creaking under their own weight and have frequent undetected/unreported hardware errors. We have also discovered the apps need to be touched or helped by a human being every few hours or the app will simply die. Nothing is automated, monitoring is sparse at best, there are no performance monitors at all, and nearly nothing was documented. This is a classic case study in how not to do business.

But all of this isn't news: we've been repairing & cleaning up this mess for much of October and have made great progress.

Today was another surprise. A domain I've never heard of ( apparently expired last night. Huh? A look into the Whois records doesn't help: the registration info is just a privacy proxy firm in Florida, shielding the actual registration holder & contact information. The domain isn't listed in our portfolio of domain names and we never received a renewal notice.

In the massive email thread which ensued, we have a strong suspicion that the domain name was registered by one of the software engineers in their personal account, but was never transferred to the corporate account as they should have. They probably did this as a convenience many years ago and it slipped through the cracks like so many other things. Unfortunately, the engineer we suspect did this died two years ago. Because of that, he's not available to renew the domain or even transfer it to our corporate account for renewal. *sigh*

We're now working legal channels to see about getting this transferred as needed. The people who have access to various mailboxes & files of departed employees aren't in the office yet so I can't check for any renewal notices or even confirmation of our suspicions on how the domain was registered. At this moment, I'm at a dead stop.

Things like this aren't supposed to happen. It's a profound embarrassment & shame upon the clowns who created this situation, and the rest of us for not catching it in time. Needless to say, the balance of my day is going to be spent scanning all of the source code in that division to look for any other domain surprises and ensuring all of them are in our corporate portfolio where they can be properly managed.

Update: We've renewed the problem domain but are still trying to transfer it and several others to our regular portfolio. The snag we're finding in many of these domains is that the registrant contact information is incorrect: there's a typo in the email address! We're now in the process of getting around this by registering the typoed domain name, then creating MX records to point it to our corporate email server. *sigh*
bjarvis: (Default)
We use Akamai as a content distribution network. For the non-geeky crowd, this means that some of the items in our firm's web applications are distributed to the planet by Akamai: when a user in, say, the UK, uses our application, the graphics, images & logos in our app are retrieved from a local server in the UK close to the user rather than reaching across the planet to get a copy from our server in Virginia. This gives the user a faster web experience, and it eases the load on our web servers in Virginia.

Naturally, we use web certificates so users can be certain (a) they are getting data from an authorized source instead of Someone Evil, and (b) it is encrypted for privacy, (c) it is secured for consistency to ensure the data wasn't manipulated in transit.

We have several web certificates for many domains in Akamai. Three of the certs will expire this week so Akamai contacted us for approval to renew them. I've been trying to tell them that we don't need the certs and to let them go, but they just don't believe me.

Akamai: There's three three certs which are gonna expire. Let's coordinate a time to talk to the cert vendor to renew them!
Me: Not needed. We don't use those domains so we can let the certs expire and save money.
A: But there's customer traffic!
Me: Not from us. We don't use those domains.
A: But there's customer traffic!
Me: If any, it's probably web bots and spiders. Don't care.
A: But there's customer traffic! Look, one domain name is CNAMEd traffic to our caches!
Me: Yes, the data is in the zone files, but we don't use those domains.
A: But there's customer traffic! You need to renew immediately!
Me: No, that's not us. We don't care. *Logs into the Akamai control panel, removes the certs. Updates DNS to remove any reference to Akamai in those zones* Look, I've wiped out the references to Akamai. We don't use the domains, we don't care about the certs, we want to them to expire.
A: But there's customer traffic! It will all fail now!
Me: What can I do to persuade you we aren't using these services and can let them expire?
A: *silence*
bjarvis: (Default)
My dear employer purchased another company about six years ago, adding ground & car bookings to our business travel portfolio. Of all of the acquisitions we've done over the years, this was practically the only one which made sense, the only one which has been financially worthwhile and the only one still operating, but that's another story.

This particular car service division has been largely independent of the larger firm: our travel systems makes system calls into the car service systems, but we haven't tried doing a full integration of their services or their staff. Our core travel systems are all based on Linux with Solaris/Oracle handling the backend databases, while the car service machines are all Windows Server based with Microsoft products and some cloud-based services to supplement.

In the past month, two of the primary people from the car services division have left the firm, and because we have no other staffing, care & feeding of their systems have fallen to my systems engineering team. And now we're seeing the true nature of the nightmare...

These car service systems require constant care. Constant. We've learned that the core database has been receiving manual maintenance daily for the past eight years it has been in service. We've learned the logging system has been manually restarted every 48 hours or so for the past number of years. There's a stack of little things like this which have been consuming the full attention of two fulltime staff on a daily basis.

I'm horrified by the amount of work that has been required daily if not hourly to maintain uptime for these systems. I'm horrified that no one in management seems to have noticed and thought it odd. I'm horrified that no one has seen fit to fix any of these problems, especially the guys who have been doing the work. And I'm horrified that even if the guys couldn't correct the root problem, that they didn't even attempt to automate the required recovery steps. Seriously?!

My team is now trying to pick up the pieces but I have little Windows experience and the training hand-off occurred while I was on vacation so I'm missing huge chunks of knowledge about their architecture, single points of failure, and other gems one could collect from those who built & maintained these things. It doesn't take great knowledge though to know that This Isn't Right.

Remember your training, young padawan:
1. Automate everything.
2. Automate recoveries as much as possible.
3. If something breaks daily, fix it.
4. Document everything so the people coming after you have a guide.
bjarvis: (Default)
In various telephone calls today with people at our San Francisco mothership, I have learned that two more senior staff from our Car Service division in Secaucus, NJ, have left the firm. By default, my team is now caring for their servers & services, although at this moment, I don't even have login credentials for most of their machines, not to mention an inventory. This is a sad and dangerous situation.

On the good side, my boss is going through the formality of interviewing a candidate for the recently departed Car Service director. It's a guy that the boss has already worked with, and with whom I worked with on a recent visit to California. If we can make him an offer, I will be a very, very happy camper.

In a vastly bigger bit of news, the board of directors has approved a budget of $1.2 million for support & upgrades of our data center facilities. At long last, we have approval to proceed with the hardware refresh in the data center where I work, after nearly a year of false starts. The boss is making some calls to confirm numbers and then we start pushing out the purchase orders, probably before the end of this week. My October is going to be very, very busy but it will be doing the work I love.
bjarvis: (Default)
I've been away from work for two weeks, the longest single vacation I've taken in years, possibly ever.

I've been checking up on office email via my mobile from time to time, sometimes to see if there's any emergency which concerns me, but mostly to trim out the crap so I'm not overwhelmed with 1500 emails the moment I step back into work. I dislike surprises so I wanted to keep abreast of any developments. A lot of peace of mind can be bought for only ten minutes per day.

So far today, I've just been taking care of some outstanding tickets and minor issues which have been ignored in my absence. There is one server issue which has been generating an email message every two minutes for a week now... I'm unsure how that one was ignored all this time but I'll have that fixed shortly.

The HR folks have been busy. The official unlimited vacation policy is now in place, along with the steps for implementation on Sept 15, 2017. To clean up the bank of outstanding vacation days before that start date, the firm is officially shutting down for the week of Thanksgiving and the week of Christmas this year, and the week of Labour Day next year. That wipes out 15 vacation days, whether I want to or not.

I have no concerns though about the compelled vacation. Even after taking two weeks off, I have eight vacation days left in the bank. And with my seniority, I get another 20 vacation days before the policy starts. Even with the forced burn, I'll still have 13 vacation days leftover. Further bonus: the company shutdown days are exactly the date ranges I was planing for anyway.

Since my extra vacation days won't be cashed out, I'll find other uses. I still need to set aside days for my grandmother's upcoming 90th birthday in April, 2017, as well as for the IAGSDC convention in Palm Springs around July 4, 2017.
bjarvis: (Default)
My Dear Employer is going to be participating in a large travel show in Denver, the GBTA. We've been there before but with recent shake-ups in upper management, it was deemed to be important to be there just to demonstrate that we are alive & kicking. And to ensure there's something to actually show, there's been a planned series of new features & upgrades to our applications over the past two months, the last of which gets released into Production on Wednesday.

Some have asked what my employer actually does, and I have typically said we're a player in business travel. That's the core of it, but it's not a very detailed explanation. From a press release which was previewed to staff today, here's the official word:
We make business travel less complicated for travelers, less costly for employers and more profitable for service providers. Using our industry-leading software solutions, employees book travel and car service and report those expenses faster and more easily than ever before. Corporations control costs more effectively. Travel management companies deliver more engaging customer experiences. Car service operators benefit from new efficiencies and widen their customer reach. We've helped thousands of forward-thinking companies modernize their systems, improve travel management and save big money.

We also have a business expense management platform, but that's not part of this particular trade show and app upgrade package: it will, however, be the corporate focus early next year, to bring it up to par with our other offerings.

We've been given the general approval for the upgrade of our data center cages in DC3, the ones for which I am employed to maintain. The plan is to acquire a third cage in the same building, fill it with newer, higher density equipment, deploy the latest operating systems & apps to it, then direct traffic from the old systems to the new. Once fully migrated, we decommission both of the old cages.

Like I said, we have general approval, but no one has officially pulled the trigger on that project yet. There's a huge pile of spending up front, and once we acquire the third cage, the clock starts ticking: we don't want to pay for three cages more than we have to so we will need to move quickly. Now that the majority of our VC funding negotiating is done and a few major deals are signed, I hope to see the signatures on the purchase orders soon.
bjarvis: (Default)
I'm home from the annual IAGSDC convention in Toronto, and while I didn't dance much --perhaps three tips-- I spent a lot of time studying the staff callers.

In prior caller schools and in articles I've written for The Call Sheet, I've frequently said that it is OK for callers to specialize: we don't have to be all things to all people at all times. There typically are more than enough callers around to permit everyone a niche where they find maximum joy. I've walked that walk: I don't do square dance parties as I find them painful & frustrating, despite them being the most lucrative calling gig.

Of late, I've decided to pare down some of my other calling responsibilities.

I don't especially like teaching newbie dancers. I can do it --and do it well-- but it's draining to the point of physical debilitation. The last few times, I had to sit in the car in the parking lot after the gig to collect myself together sufficiently that I could drive home safely. Even when I succeed, there's the constant worry that in a moment of weakness, I might telegraph something less than joy from the stage: the mic & location amplifies everything and a newbie dancer who has less than an ideal experience their first time out may be lost to square dancing forever. For a shrinking recreational activity, the stakes and pressure to perform are high. Teaching absolute beginners is not my Happy Place.

I've also written a few times about the Dark Side of Square Dance Calling. Once you call for a club, you're forever on the hook as the expert in all things, whether you know anything or not. During the GCA grad dance in Toronto, a few people saw me on stage adjusting the volume settings and holding a microphone: for the rest of that weekend, I was suddenly their go-to guy for issues of audio in all halls, air conditioning levels, banquet table layouts, lost & found, and fun badge tours --of which I knew nothing, nor had any responsibilities. But despite my insistence that I couldn't help them and directed them to the registration table of volunteers, they kept coming back to me 'cuz they saw me with a microphone on a stage once.

As a club caller, I'm wheeled out to consult on audio issues for every festival, weekend dances, and misc events between them all. And revising & rewriting festival schedules through multiple drafts. And consulting on & contacting potential future festival staff callers. And MC-ing various events. And coordinating other resources. Yup, I spend two hours on stage for a club night, but there's so many more backstage which no one ever sees (or at least, won't notice if I've done everything correctly). It doesn't sound too bad, but multiply this by several clubs and it adds up.

All of this was unknown to me when I started calling.

The one thing I did know in advance was a minor thing back then, but has become a more onerous problem in 12+ years I've been calling: your schedule is not your own anymore.

I have so many Tuesdays, Wednesdays & Thursdays booked over this past year for various club nights. Feeling tired? Too bad: you must go. Weather sucks so bad no one in their right mind would venture out? Too bad: there hasn't been a formal cancellation so you must go. Had a spectacularly bad day with your spouse or at the office? Too bad: you must go --and fake joy about it until the gig is complete. Need to work late? Too bad: you have to do your gig, then go back to the office even later to mop up. Taking a gig is a promise and you are honour-bound to fulfill it, no matter how painful.

Over the past number of years, I've missed many professional training opportunities, business trips, professional meet-ups, business dinners and such, not to mention had to perform several all-nighters at the data center because I was booked for a dance gig. Most of my gigs are planned months in advance, but ironically, most of my best business obligations & opportunities are short notice. Being a caller has limited my professional scheduling agility. And while this isn't a huge surprise and not a catastrophic problem, the effect has been that I'm slipping professionally in my industry knowledge. It's just a little each year, but after a decade, it adds up: I need to correct this quickly before I become professionally obsolete.

So all of this is a large justification for my next steps:
1. I've formally resigned as club caller for Chesapeake Squares in Baltimore to free up my Tuesday evenings;
2. I've formally resigned as club caller for DC Lambda Squares in DC to free up my Thursday evenings.

I'm going to keep calling, primarily our Wednesday C2 group, Zig Zaggers (Advanced) and other subs as they come up. I want to do more guest spots at other festivals; I've been ignoring those for the past 2-3 years because I'm typically too exhausted from being a de facto festival organizer because of me being a club caller.

3. Mainstream and Plus are not my Happy Place. I can call them and do it well, but I like challenging take-no-prisoners, hot hash & DBD, while the market wants easy, breezy, wind-in-your-face dancing with no conscious thought required. Yup, I can do that but it's not my forte so I'll refer queries to my list of great local callers.
4. No more open houses, first-nighters and square dance parties. Yup, there's good money there, and yes, these are the things we need to make our activity grow, but I'm not the guy to run those. Again, I'll refer these to someone who does them better and actually enjoys them.
5. I do good extemporaneous calling and sight resolution. I use that exclusively at Mainstream and Plus, 50/50 at Advanced, and sparingly at Challenge-1 and Challenge-2. I want to correct that: in the next couple of years, I want to be as fluent in the A/C programs as I am in MS/Plus. This is what I was studying at the Toronto convention, and that's where I want to be.
6. To fulfill (5), I need to memorize many more get-outs, and become more confident in spotting & using great get-outs when they appear during extemporaneous calling. Every now and again, I've called something like "follow your neighbour" and while the dancers were still moving, I saw what I was 90% sure was the place to call a "right-and-left grand": I need to believe in my gut a bit more, and take a few more measured risks.
7. My music collection needs a refresh. I've been relying on the old standards too long and it's getting stale. I've bought seven new pieces this past weekend and am about to go review the CALLERLAB convention recordings of the panels discussing music to see what else I can pick up. Huge amounts of music produced annually sucks, but there are still some gems to be found if one looks hard enough.

I do understand that the changes in my path may mean I'll never stand on a stage again, as most gigs go to those who can and do offer everything. I'm fine with that: the important thing is that the best person for the role is hired so if I don't meet the requirements, I shouldn't be on any mic.

And I'm looking forward to catching up with my professional career, attending conventions, meet-ups, presentations, and such once again. I retire in 4,011 days... I can slow down then.
bjarvis: (Default)
Yes, my Dear Employer is still alive & kicking. And so am I, at Dear Employer.

The management structure is still in flux, but has nearly reached stability and a new norm. In my division, our Corporate IT and Operations groups are merging. We have had a new DevOps group for the past few months but so far, I haven't seen anything devops-y from them, just tinkering with some of our internal apps & monitors.

The COO just mentioned in an off-hand way as he was showing the new org structures in PowerPoint that they're looking at a large investment into our data centers to improve our uptime & stability. This is a relief, not only because it goes a long way to preserving my employment but also because preserving uptime is getting harder as each day goes by because of our aging infrastructure. If nothing else, I'm happy that there is now finally an announced direction.

In other fun news, there's also a proposal --and it's only a proposal-- to replace our existing vacation policy with unlimited vacation since that's what all the cool companies are doing these days. I'm not sure this would really affect me overall: at my level of seniority, I get 20 vacation days, plus national holidays, corp holidays, two flex days and a stack of sick days. Only 23 days may be accrued at any time and may only be cashed out at termination of employment; I know I haven't been using my full vacation allotment but I've been working on that this year with planned trips to Toronto and England. Anyway, the new policy would wipe out this bank, replacing it with vacation when/if my manager approves.

The snag is in the transition between policies. Management could just cash out everyone their outstanding balance so we all start from zero, but that would be a major hit to the finances at a time when management is working hard to fix our cashflows. Alternatively, they're considering a company-wide shutdown across the week of US Thanksgiving, Xmas and next year's July 4: the firm closes for a few days and everyone is compelled to use some vacation days.

I'm fine with this, largely since I have a full bank of vacation and those are the periods when I'm most likely to try taking vacation time. On the flip side, these corp shutdowns would also be the perfect windows for major infrastructure updates so vacation might be contra-indicated for my division. And I'm unsure what the relatively new hires might think since they get fewer vacation days and would now be required to dedicate half of those to the company's schedule rather than their lives.

Update: In a meeting with my new COO and new VP, it was clearly stated that the purchases of some equipment to refresh my data center should proceed immediately, and that the rebuild of the data center as a whole is corporate priority to be initiated as soon as we get refreshed quotes from vendors. Yaaaaassssss!

Update #2: My team is very<./b> opposed to the proposed unlimited vacation policy. They figure we rarely get a chance to use vacation time as is so this change doesn't benefit us at all; in fact, all benefits accrue to the company who no longer needs to carry vacation time as a P/L statement liability or cash out unused vacation time on employment termination. I'm still ambivalent on it: I don't fully exercise my current vacation allotment, and I can live with the new policy. The only impact to me is the cash-out if I left the firm but since I'm not planning to go anywhere yet...
bjarvis: (Default)
Of late, I've noticed that the work tickets submitted to my team at the office follow trends.

A month ago, the common theme was asking for impossible things: 35 servers, all equipped with 4GHz CPUs, 10TB hard drives & 128GB of RAM, to be ready in 24 hours. That kind of thing.

Last week, the trend was for the author to identify the precise problem, state exactly what team should address it --then send the ticket to my team which isn't related to any issues mentioned. I received so many networking, database and MS Exchange tickets last week; the first few I directed appropriately, but after that, I just sent them back to the author asking them why they were wasting their time sending it to my team when they already knew we couldn't help them.

This week, it's all about not even trying. I've received several tickets this week which mention casually a problem somewhere but make no effort at all to indicate where the error may lay, how to recreate it, the precise error itself, or what they'd like to do about it. Just a "Hey, something is wrong somewhere" message.

Imagine your emo teenage son walking into the house, saying "Something is wrong with the car," and then disappearing into his room. There are two cars in the driveway. Which car? What "something"? Was there a noise? A leak? An explosion? Was it related to the engine, the body, the tires? Was there really a problem or did just not like the colour of the seats?

I'm sending the vague tickets back to the author with a template for clarification. Most haven't responded back yet, probably because it's too difficult to think about such details as how to replicate the error.

And last night, I discoverd one of our senior software developers has no idea what the difference is between RAM and swap space. Not a clue.

Here's what I'd like to have happen in the next sixty days:
1. I want a major company to buy out my employer.
2. My share options get translated into real shares and I cash out.
3. A massive recession sweeps through Silicon Valley. Everyone is laid off.
4. The remaining firms regroup and hire only those with skills from the talent pool.
5. Everyone else returns to writing bad emo poetry while living in a shared flophouse above a meth lab somewhere in the midwest.

Please make it so.
bjarvis: (Default)
I'm not a big believer in new year's resolutions: if something is a good idea, it should be implemented immediately, not at the arbitrary turn of the calendar. That said, the xmas & new year's break is a great opportunity for personal introspection so new goals & targets are to be expected.

  1. I'm going to continue my gym weight training with determination. My novice phase has ended so now I will focus on development & capacity. My gym membership expires in mid-July so I have a convenient built-in deadline --and a near-certain renewal.

  2. I'm going to cut down to 15% body fat (20% currently) by May 1. Depending in how it goes, I may push onwards to 12%. More veggies & meat, more exercise, less starch, less (*sob*) sugar.

  3. I'm going to pursue additional work skills training. At the moment, I'm unsure what form that will take but I believe all education had some value so I won't turn down opportunities which may fall into my lap.

  4. I plan to see a career advisor. I like my work but I need some objective outside advice on evaluating my strengths and making possible course adjustments.

  5. More massages, fewer all-nighters & working weekends.

  6. I'm going to learn the C3A square dance program.

  7. I want to cut back on square dance teaching. It's fun but it's exhausting and a vast time commitment. I missed a number of professional goals & opportunities in 2013 because of teaching commitments: I can't afford to keep doing that.

  8. I was planning to see a chiropractor in January but the immediate physical need seems to have passed. Was it just stress or some issue now dormant? I should probably make that appointment anyway to establish a baseline.

  9. I need new glasses, probably bifocals. An appointment will be made this week for as soon as convenient.

  10. I want to cut back on old committee work. I've made my contributions and it's time to move on in several instances. I've already terminated one, reduced another and will let a third expire on its own. That said, I'm looking forward to new experiences in other board roles.

  11. We will make some household improvements. The front door is a mess, the laundry room sink leaks and we need new towels. There's more, but this is a start.

  12. Starting immediately, I'm putting $200 per paycheck into my savings account, up from $150.

  13. The demon credit cards have been slain: they will stay that way.

  14. We will update our wills & medical directives with our tax returns this spring. (Memo to self: don't die until then, much later if feasible.)

  15. I will send xmas cards in 2014 even if it kills me.

  16. Find my bliss. I haven't felt any worthy dream or goal in many years so it's time to spend some time & effort getting reacquainted with myself.
bjarvis: (Default)
...although my long absence from social media may have lead you to think otherwise.

I've been incredibly busy through November & December. I've been spending most of this week simply recharging my batteries. Is it even possible to have a negative energy level? If so, I had it. I'm back into positive territory but still have a way to go.

There hasn't been any great burden in the past 60 days, just a stack of little things. It's the proverbial nibbling to death by ducks, death by a thousand papercuts, and other similar imagery.

I don't suffer from SAD per se: it's not the length of the day which gets me down, the fewer daylight hours, the longer nights. Rather, winter implies the holiday season which further implies obligatory social events, all of which chip away huge chunks of my introverted self. It's the end of the calendar year so I have a tonne of end-of-year work commitments. The fall/winter season also has extra square dance classes and events on top of the usual cycle of club nights, all of which have their own time commitments (well, at least if one is to do them right).

New Year's can't come soon enough so I can get back to a normal cycle of routine and rest.

While I'm going to miss one of my major goals for 2013 and haven't yet found the time this week to even file the papers in my overflowing "in" tray, there have been some successes. The gym work was new as of July and has been going very, very well. My financial situation, generally healthy, has improved significantly, achieving my goals of starting a savings account and paying off all credit card debt. On the whole, life is stable. Now if I can just get a good night's sleep this next week...
bjarvis: (Default)
The past half-month has been a bit of a blur, but it's been good for the most part.

We spent Labour Day as planned at the trailer in West Virginia. All five of us were there, [profile] kent4str, [profile] cuyhogarvr and me, along with co-trailer trash Jerry & David. It makes for a crowded trailer and not a great deal of isolated rest, but it was fun nonetheless.

[profile] cuyahogarvr and I stayed at the trailer for the rest of Labour Day week. Outside of some moments of stress because of a dead car battery --it had a charge but was just shy of its 7th birthday so it didn't retain enough juice to start the car-- I caught up on some rest and reading. More precisely, I caught up on some self-study, working towards my eventual Cisco network certification later this year. Still, I could have used a few extra days to fully recharge.

Work has been active. Our team has been distracted from large outstanding projects by a surge in daily problems-of-the-moment. Not realizing the wave of minor things would run on for days, we didn't prioritize our tasks properly and it cost us some time, but this week we'll be more aggressive about it: a couple of our team will be dedicated to simply addressing the daily minor things while the rest of us attack the outstanding big projects.

That said, I did get two major projects off my plate and a third nearly completed by this past Friday. I'm confident our team overall will be back in a very good position again with in a week.

Spare time will be at a premium this coming week: the Mainstream class with DC Lambda Squares continues for another two Wednesdays, even as a new Mainstream class for Chesapeake Squares starts this Thursday. I have a club night with Chesapeake Squares this Tuesday, then a six hour calling marathon, Mainstream through Challenge-1, with the Times Squares in New York City this coming Saturday. I also have a board meeting with the Mid-Atlantic Challenge Association Sunday afternoon.

Needless to say, I'm looking forward to all of these projects but it will require some heavy lifting for the next two weeks.

September 2017

3 45678 9


RSS Atom

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Sep. 26th, 2017 02:01 am
Powered by Dreamwidth Studios