If it's making a funny noise, it's a bad sign

Two thursdays ago we were going to bed here at Patterson Acres, when we realised that the mail server that lives in the DIY server cabinet was making a funny noise.

A bearing (or something like one - the sleeve maybe) in the server's case fan had seized and the internal temperature reading said 70 degrees celsius. The server was shut down, but the next morning I started it up again to see what happened. The temperature rose high again, so I shut it down over ssh from work.

Alas, the server did not power itself off, and even though the system was totally idle it remained hot.

Suffice it to say that the machine didn't come up when I tried to boot it the next time.

The moral of the story is fourfold.

1. DIY server cabinets which don't contain servers that can power off remotely are a bad idea. There's no sysadmin to run into the room and pull the plug at home while you're at work.
2. Always keep a spare fan lying around if that single fan is all that separates your server from self-inflicted heat stroke.
3. Don't have a headless server hanging around without a monitor you can connect it to in an emergency. We only have a DVI monitor, and I'd been unable to find and buy the DVI adapter daughterboard for my EPIA PD-6000E, so I had to take the box to work to get console access! Very dumb.
4. If its fan's bust and you don't have a replacement, just leave the damn thing switched off.

We were without a mail server for a week, and mail service is now provided by a proper server in a proper server room. (Thank you, Bytemark.) Reports from friends, and mail volume, suggest that maybe the backup MX didn't store and forward like it ought, so any mail sent to me between 15 and 22 September may well have just bounced hard. Please resend if it was important.

• 26.9.2006, 20.32
• File under: DIY, IT infrastructure, server cabinet, bad idea, equipment failure