Today I ventured into the most annoying adventures a system administrator can face - reinstall. Fortunately this was a planned one, so I had just made all backups.
Problems started even before I did - all air circulation devices were broken in the company that collocates our server, so it was very hot there. I am not afraid of some heat unless it cooks my servers, so I went on with the reinstall.
While I was on the way to the location I asked the kind people there to make some kind of temporary banner page on one of their servers to say something like 'LAKA is being repaired now, come back in a few hours'. While they were doing that, it was needed to restart their corporate firewall as it stopped responding. And then ... the firewall didn't come up. Dong! Now imagine three men in a hot and tight server room struggling to untangle cables to get that firewall box out in the open to see what happened to it. Juk! After dissection the firewall claimed that it suddenly needed a video card. It's clear that the CMOS battery is dying on that thing. The way of least resistance was chosen (as people from all levels of the building were rushing in the server room every 5-7 seconds) and the Internet was returned to the people, and me as I was just going to finally start reinstalling our server.
As usually with 7+ year old hardware, the CD-ROM gave out in the worst possible case. Luckily there were some spares around. This servers case is something really interesting - it actually has a button to open the side of the case. Like a case eject :).
Installing Debian was uneventful showing the high level of this distro. In parallel I helped one guy to try to install SMB printer in Gentoo and soon I had to arrive to a conclusion that Gentoo sucks - nothing worked without a bit of tinkering. Even after installing ppds for the printer I had to unzip and install them manually. Juck!
Reinstalling needed programms was fast and seamless. The biggest problem was restoring the data. Unfortunately I made a full backup of the whole disk an that was a big problem. I mean 800 Mb is big if your downlink is less then 50 KB/sec :(. I spend a hour cutting out all the stuff I needed to revive the web and shell users. That was 150 Mb. Not wanting to waste my time waiting I wanted to put it on the download and head home to finish the thing remotely, but there was another problem lurking in the firery shadows - the firewall didn't want to give me back my IP.
I left hosters deal with that problem. When I left they were telephoning the authors of the firewall. Half an hour after I got home, the server was back online and I could start copying the reduced backup. Restoring Web and mysql services was a snap even considering migration to Apache 2 and MySQL 4 in the process. It just worked :)
After that I realized that I had screwed the user ids by copying old /etc/passwd and /etc/shadow over the new ones. Doh! I was save by some mysterious force that made backups right after the installation. I spend 20 minutes inching back into control of the system (I lost root and ssh would scream in panic because /etc/passwd had 0600 mode :P) and another 15 to carefully carry old users over to the new system.
Then I recovered a lost MySQL root password by looking at backup of my .mysql_history file. Doh!
Now I'll need most of the rest of the backup to get mail and postgresql back up. More waiting :(
Edit: Now this is getting ridiculous - the server that has the backup just shut down on his own. Ridiculous!