Recover Server Again

A pickup truck crashed into an electric pole just down the road yesterday. The pole fell partway over, held aloft by the power cables. The electrical mains power dropped abruptly. My Dell R530 and Qotom fanless server are powered through an uninterruptible power supply, but only the R530 has the USB connection and runs apcupsd to get notifications that the UPS is running out of energy.

Apparently, the UPS powering off took the Qotom fanless server by surprise. It did not successfully boot after the power came on. It was showing some EFI boot failure screen advocating something called a “Windows Recovery Disk”. That’s a new one on me, but really: humbug!

Fixing up the broken server

1. Made an Arch Linux Install USB thumb drive, because I couldn’t find the previous one I made.
2. Put the thumb drive in a USB slot, powered the Qotom fanless server on. When I saw the American Megatrends splash screen, I hit the Esc key to get the BIOS configuration program. It took me a couple of tries, but I got the Qotom to boot into the Arch Linux install.
3. Ran lsblk -f to find which device had what filesystem on it.

sda
├─sda1
│    vfat   FAT32             DF29-5550
├─sda2
│    swap   1                 e4323bb0-90d6-47f3-8a1e-a56033d57089
└─sda3
     ext4   1.0               72be0c8e-aee1-4006-a670-c601c64f9173

4. Since I wasn’t too sure of the root cause of the boot failure, I fixed up all the file systems.

/dev/sda3 is the giant bulk of the Linux filesystem, /, /home/, /usr/, etc etc. /dev/sda1 is the EFI stuff, mounted as /boot/ on the Linux filesystem.

$ fsck /dev/sda3
fsck from util-linux 2.42.4
e2fsck 1.47.4 (6-Mar-2025)
/dev/sda3: recovering journal
Clearing orphaned inode 88605510 (uid=0, gid=0, mode=0100644, size=7
Setting free blocks count to 320238231 (was 320239387)
/dev/sda3: clean, 6608379/122044416 files, 167932265/488170496 block

$ mkswap /dev/sda2
mkswap: /dev/sda2: warning: wiping old swap signature.

I also ran fsck /dev/sda1. I got this question:

There are differences between boot sector and its backup.
This is mostly harmless. Differences: (offset:original/backup)
  65:01/00
1) Copy original to backup
2) Copy backup to original
3) No action

Looking around the web, people say that if you choose (2), you get other questions to answer, specifically a question about clearing a filesystem “dirty bit”. No matter what I answered on these questions, writing the changes back didn’t seem to make a difference.

Label '' stored in boot sector is not valid.
1) Remove invalid label from boot sector
2) Enter new label
[12?q]? ?
1) Remove invalid label from boot sector
2) Enter new label
?) List all choices
q) Quit fsck
[12?q]? 1
Dirty bit is set. Fs was not properly unmounted and some data may be
1) Remove dirty bit
2) No action
[12?q]? 1

When I ran fsck /dev/sda1 again, the “65:01/00” difference appeared again, and the dirty bit was set. I answered with numbers to change these things and wrote the boot sector several times.

I don’t think this was the root cause. Removing the thumb drive and a power cycle got me the “Windows Recovery Disk” screen again.

5. Reinstall the Linux kernel.

  1. Boot Arch Install from USB
  2. mkdir /mnt/sda3
  3. mount /dev/sda3 /mnt/sda3
  4. mount /dev/sda1 /mnt/sda3/boot
  5. arch-chroot /mnt/sda3
  6. pacman -Sy linux

Remove thumb drive, reboot, got “Insert Windows Recovery Disk” error screen.

6. I reinstalled the grub boot loader. I did the same 5 step procedure above to get to the arch-chroot shell. It seems safer to do it that way.

This is a long, exotic command:

$ grub-install --target=x86_64-efi --efi-directory=/boot --bootloader-id=GRUB

I got the GRUB bootloader ID by looking in /boot/EFI/. Two subdirectories appear in that, Boot/ and GRUB/. The Arch Linux wiki GRUB entry sounds like you could see other strings.

I think this is the file that gets updated:

1011 % ls -l /boot/EFI/GRUB
total 164
-rwxr-xr-x 1 root root 167936 Oct  3 17:32 grubx64.efi*

There may be some hidden exotic action, like putting grubx64.efi file’s inode number or first sector number into the boot block. I don’t know.

For good measure, I reworked the GRUB config file:

$ grub-mkconfig -o /boot/grub/grub.cfg

I edited out two instances of quiet on boot commands from the new /boot/grub/grub.cfg. I like to see what’s going on, even during the esoteric boot procedure.

This fixed the issue. I quit out of the arch-chroot shell with a control-D, and exited the Arch Install with another control-D. A reboot followed, with the linux 7.2.8-arch1-2 kernel, installed in step (5) above becoming operational.

Final verdict

I think that re-installing GRUB actually fixed the problem. The EFI boot partition is a VFAT filesystem, which is notoriously easy to break.