![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
The Hard Drive Gods are Angry
Feb. 22nd, 2013 02:06 pmThis work week has been all about hard drives.
The expansion units for our Fujitsu storage array arrived late last week so we scheduled time with the field engineers to install the disk enclosures and hard drives, then configure the whole lot. Unfortunately, Tuesday night, an existing disk died in the same array. Swapping it out was easy, but the Fujitsu's firmware prevents any configuration changes until everything is green, so while we could install the new hardware, we couldn't do the config updates until the RAID rebuild was completed.
The configuration was finally completed this morning. We now have an additional 13.5TB of raw disk space on that array.
While all of this was happening Wednesday, a disk died in our Network Appliance array. The support agreement on the NAS had long since expired since a renewal would have been equal to purchasing a brand new NAS. Two hard drives, a replacement for the failed disk and a spare, were ordered and arrived today. All is now good with that beast.
A 70GB disk died Tuesday night in one of our HP blade servers. Fortunately, the boot disks are mirrored by hardware RAID so there was no system interruption. I pulled the dead drive and inserted a new one --I have spares in hand-- and the array reconfigured itself automagically.
A 600GB disk died in a Silicon Mechanics server. Again, the disks are mirrored and I had spares so I swapped out the dead disk. Because of the cheap-ass disk controllers shipped with these machines, we're using Ubuntu Linux's mdadm software mirroring so I had to manually enter the extra commands to force the array rebuild, but that was easily done.
By lunchtime Friday, all arrays and configs were current once again. No data or servers were threatened in this period, but I did think it unusual that so many disks in such diverse hardware should all expire in the same week. Considering some disks in the Fujitsu have been working 24x7 for five solid years, it's a wonder we don't have more failing there than one per month.
The expansion units for our Fujitsu storage array arrived late last week so we scheduled time with the field engineers to install the disk enclosures and hard drives, then configure the whole lot. Unfortunately, Tuesday night, an existing disk died in the same array. Swapping it out was easy, but the Fujitsu's firmware prevents any configuration changes until everything is green, so while we could install the new hardware, we couldn't do the config updates until the RAID rebuild was completed.
The configuration was finally completed this morning. We now have an additional 13.5TB of raw disk space on that array.
While all of this was happening Wednesday, a disk died in our Network Appliance array. The support agreement on the NAS had long since expired since a renewal would have been equal to purchasing a brand new NAS. Two hard drives, a replacement for the failed disk and a spare, were ordered and arrived today. All is now good with that beast.
A 70GB disk died Tuesday night in one of our HP blade servers. Fortunately, the boot disks are mirrored by hardware RAID so there was no system interruption. I pulled the dead drive and inserted a new one --I have spares in hand-- and the array reconfigured itself automagically.
A 600GB disk died in a Silicon Mechanics server. Again, the disks are mirrored and I had spares so I swapped out the dead disk. Because of the cheap-ass disk controllers shipped with these machines, we're using Ubuntu Linux's mdadm software mirroring so I had to manually enter the extra commands to force the array rebuild, but that was easily done.
By lunchtime Friday, all arrays and configs were current once again. No data or servers were threatened in this period, but I did think it unusual that so many disks in such diverse hardware should all expire in the same week. Considering some disks in the Fujitsu have been working 24x7 for five solid years, it's a wonder we don't have more failing there than one per month.
