Unscheduled downtime this morning.

CS // 19 Jan 2009 // Comments Off on Unscheduled downtime this morning.

We experienced an outage this morning when one of our mail server hard drives failed, causing mail to stop.

The hard drives will need to be replaced in order to ensure data is protected.   As replacement will necessitate some downtime while spare drives are being installed, and existing ones replaced, I have scheduled this for later tonight, or early tomorrow morning.

The new drives have been ordered, and we are waiting on their arrival.

Replacement will be done out of normal office hours in order to minimize further inconvenience to clients.

We expect the server to be down for 20 minutes when we first add the new drive, then for a further 20 minutes  2-3 hours later once online backups have been copied over to the new drive.   We will update the blog with expected outage times once we have further details.

UPDATE

Seems the drive failure(s) are due to a firmware bug in the hard drives..

http://mswhs.com/2009/01/21/seagate-hard-drive-firmware-bug/

We’ve replaced drives in 2 servers so far (I replace drives yearly as a matter of principle), and still need to change another 2.
Unfortunately as one of our other servers drives decided to crap out in sympathy, the 2 drives I had for replacements had to be used immediately in different machines (thank you very much Seagate for ruining my day)

61.129.88.213 and 61.129.88.233 have both been replaced with Western Digital drives
61.129.88.121 and 61.129.49.190 both need to be replaced still.  This will get done as soon as I can arrange additional units this week (26th Jan+-).
72.51.39.20 and 61.129.65.217 are not using Seagate drives, so don’t need to be changed thankfully!

We have also replaced the fan’s in all the servers while we were doing maintenance.
Please bear with us while we complete our replacements.

Last time we buy Seagate…

Further reading –

http://seagate.custkb.com/seagate/crm/selfservice/search.jsp?DocId=207931

http://hardware.slashdot.org/article.pl?sid=09%2F01%2F17%2F0115207&from=rss