I've had a terrible weekend, occupied entirely with technical problems. I arrived at the the
Sydney office at 7:30am yesterday (Saturday) and worked on a server upgrade until 11:30pm -- was back here at 8am this morning, it's almost 11am, and I'm sitting waiting for the
cavalry to arrive.
Here at the agency, you see, we have a whole lot of Finux servers to handle different tasks. I took one of them down yesterday for a rebuild -- this is the server that provides fileservice to 55 or so users, mostly graphics people doing huge Photoshop jobs. I've got a sweet external 240Gb RAID disk array that connects to the server as if it were a single disk. After the backup and rebuild, I'm booting the machine off an internal 9Gb LVD disk, /dev/sda, just fine, but every time I connect the external array the machine freaks out and thinks that the external disk is sda, resulting in a VERY UNHAPPY SERVER
I've rebuilt that machine 14 times so far this weekend, in every conceivable permutation to no avail.
I've been calling around what few computer stores are open in Sydney on a Sunday fo try to find a couple of big LVD disks (say 75Gb IBMs) that I could simply connect to the internal bus, to at least get some of the critical data back on line. Typically, there isn't an LVD disk on any store's shelf. It's times like these that I curse Australia -- if I was in Silicon Valley I'd just drive to my favourite Fry's and pick up whatever tech stuff I needed...
Hopefully my high-priced consultant will sort this thing, otherwise I will have a very pissed off company tomorrow at 8am..
Dinner tonight with two local noders, alex.tan and lignocaine, at the BBQ King in Chinatown. We've been trying to get together for weeks now, and although I'm in pretty deep shit right now I'm going to make dinner, come Hell or high water.
Thanks to Rancid Pickle for keeping me company last night while I was in melt-down. E2 is better than Valium, baby.
Update mid-day, Sunday
The cavalry sorted the server out -- it seems that the bootloader was loading an image that included the RAID's SCSI adapter module before the on-board SCSI, which meant that the bootloader was loading either bogus bootstrapping or none at all. Duh.
I'm now restoring the data. Disaster averted.
Update 2pm Sunday
It seems that the woman who runs the backups here did not run a full backup on Friday like I specifically asked her to. I'm restoring maybe 10% of the data... Sheesh... Looks like I'm gonna have to run a multi-tape restore from the last incremental. This has been a horrible weekend.
Update 8:30pm Sunday
Still here, expect to be running recovery tapes until 4 or 5am. I really hate this. On a brighter note, just took a walk outside and had a funky Chinese meal with alex.tan and lignocaine, very lovely nodists both.
Update 9:30pm Monday
Just had to call a taxi to collect some offsite tapes from Jayne. Lucky she was home. Thank You Jesus! I have AIT and not DAT. Still, a robot would be cool..
Update 2:20am Monday
Still here, still swappin' tapes.... Tired eyes, E2 keeping me sane. Users arrive in 5 hours, I don't think the system will be all back for them. Oh goody, 50 users all moaning at me in a couple of hours time. I'm sure to deal with that really well...
FINAL Update 7:00am Monday
Finished the restore, recreated all the mount-points and checked permissions, tested OK, yeah! I went to init-1 to e2fsck -CV /dev/sdb and it was clean. Phew! Went back to init-3 and checked into the netatalk daemon via a Mac and HELLS BELLS THERE WAS NO DATA THERE BUT THERE SHOULD BE 230GB OF FRICKIN' DATA OH HANG ON A MINUTE I FORGOT TO REMOUNT /dev/sdb it's all okay now, man I have to get some sleep....... zzzzzzzzzzzzzzzzz......