![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
Not So Restful Long Weekend
May. 29th, 2012 11:41 amIt was a long weekend. That should be restful.
We spent it at the trailer in West Virginia. That too should be restful.
Sadly, I was on-call for work. That was not restful.
Something was changed on one of our config servers that caused httpd to reproduce wildly while sucking up huge amounts of memory, causing the machine to exhaust RAM and swap space and then roll over & die. The machine died six times on Friday, including three times between midnight & 6am. Ultimately, I added a cron job to restart httpd hourly to prevent the memory exhaustion. Today, I'll check to see what my team did to this box to make it do this.
Another machine generated a lot of false alerts, setting off my pager after 3am Saturday.
A handful of vservers didn't get rebooted on schedule last month so java processes were dropping like flies Friday & Saturday, requiring manual restarts. I'd be happier if the damned software engineers would just fix their bloody code memory leaks but that plan doesn't come into effect until next month (we're adding them to the pager list so those responsible for crappy software will suffer with the rest of us).
Despite three days of "vacation," I didn't get much sleep.
On the flip side, having to spend so much time online made me really glad the campground upgraded the wifi systems to a wonderfully sweet level.
At least our trip home was relatively easy. I always dread heavy traffic but outside of a couple of minor choke points, the roads were manageable. I caught up on some sleeping & reading before calling it a night, but it still doesn't feel like I had a weekend at all.
We spent it at the trailer in West Virginia. That too should be restful.
Sadly, I was on-call for work. That was not restful.
Something was changed on one of our config servers that caused httpd to reproduce wildly while sucking up huge amounts of memory, causing the machine to exhaust RAM and swap space and then roll over & die. The machine died six times on Friday, including three times between midnight & 6am. Ultimately, I added a cron job to restart httpd hourly to prevent the memory exhaustion. Today, I'll check to see what my team did to this box to make it do this.
Another machine generated a lot of false alerts, setting off my pager after 3am Saturday.
A handful of vservers didn't get rebooted on schedule last month so java processes were dropping like flies Friday & Saturday, requiring manual restarts. I'd be happier if the damned software engineers would just fix their bloody code memory leaks but that plan doesn't come into effect until next month (we're adding them to the pager list so those responsible for crappy software will suffer with the rest of us).
Despite three days of "vacation," I didn't get much sleep.
On the flip side, having to spend so much time online made me really glad the campground upgraded the wifi systems to a wonderfully sweet level.
At least our trip home was relatively easy. I always dread heavy traffic but outside of a couple of minor choke points, the roads were manageable. I caught up on some sleeping & reading before calling it a night, but it still doesn't feel like I had a weekend at all.
