Yesterday evening I finally got around to updating the Arch Linux installation on my main machine. I tend to put this task off – after a few recent updates I’ve been stuck with an unbootable machine and had to reach for a rescue CD. Once it was the switch to systemd, another time it was a change in how LVM started. Last night was to be no exception, only this time the disaster was for once entirely of my own creation, although it stemmed from the move of all the binaries out of /bin, /sbin and so on into /usr/bin.
The initial pacman -Syu failed because of those binaries moving. The fix was clearly explained here. Of course the first stage took ages; downloading and installing nearly 500 packages is not a quick job. So I went off to do something else. By the time this was done, it was already 1am and time to think about some sleep. By this time I’d forgotten there were two more (perhaps important!) steps in the recipe. But I thought I’d better check the machine would still boot, just so I knew what I’d be waking up to in the morning. One systemctl reboot later, and I was staring at a brick. Booting is hard with no /bin/mount, /bin/fsck and so on. It suddenly dawned on me what I had done.
So SystemRescueCD was booted, the method of choice in the past. I was hopeful of running pacman with a suitable set of switches to fix up the mess. Unfortunately, the systemrescue CD I had in the cupboard was only a 32-bit system, and the Arch system is 64-bit. Oh dear.
Next, I remembered there was an ancient Ubuntu 10.04 still present on another partition. I got that booted, fortunately remembered the passwords, and was in. By suitable use of setting LD_LIBRARY_PATH and using /mnt/tmp/usr/lib64/ld-linux*.so to run pacman, I even got pacman to run queries. But it wouldn’t quite do the rest of the installation – I think it was still looking in the wrong place for something. By this time it was very late and I gave up for the night.
Sleep works wonders for the creativity. This morning, I decided to try manually creating symlinks in the Arch filesystem for /bin/mount and friends from within Ubuntu, to see if that was enough to get Arch bootable again. Viola! Well almost. It booted, but seemed to be lacking whatever file tells systemd to start some gettys on the consoles. But luckily, sshd was running. I used connectbot on my phone to get into the machine, remove the temporary symlinks from /bin, and complete the install by running the 2nd and 3rd steps of the recipe linked to earlier.
Now there is just the usual pile of .pacnew files to deal with. Or ignore.
This has to be the closest I’ve come to needing a clean install in quite a while…