Downgrade Arch Linux Kernel

As is my habit, I ran pacman -Syu on my Qotom fanless server to get all the fresh Arch Linux software updates. The update included a kernel, moving from “linux-hardened-7.1.3.hardened1-4” to “linux-hardened-7.1.5.hardened1-1”. Kernel upgrades are almost always no big deal. Today, I got a kernel panic!

I don’t think I’ve gotten a kernel panic on this particular machine before. It was harder to fix than it should be. I didn’t give the EFI partition enough room when I set up the Qotom’s disk, allocating only 300 Mb to it. That’s only room for one kernel. I usually put at least two kernels on any Arch Linux machine, linux-zen and linux-lts on laptops, linux-hardened and linux-lts and maybe even linux on servers. If I’d had a second kernel installed I would have had less of an ordeal.

  1. Find a USB flash drive. This should be easy and intuitive. Plug it into laptop. Ensure I know which device file represents the USB drive. It was /dev/sda, because my laptop has an NVMe storage device that it boots from, which gets mounted on /.
  2. Download Arch Linux install ISO from the nearest large server. I got archlinux-2026.07.01-x86_64.iso from asu.edu. Thank you, Sun Devils!
  3. Copy ISO to USB flash drive: dd bs=4M if=archlinux-2026.07.01-x86_64.iso of=/dev/sda conv=fsync oflag=direct status=progress
    Every other time I’ve done this, the laptop storage was /dev/sda, and the USB drive showed up as /dev/sdb. Be very careful with this one - one mistake and your laptop disk drive will have gibberish written on it.
  4. Put flash drive in Qotom Fanless Server.
  5. Stop Qotom boot at BIOS. Some hardware makes you hit the F2 key repeatedly because it doesn’t give you a good cue and only offers a small time window, but the Qotom has a big AMI splash and gives you a few seconds to hit Esc.
  6. Figure out how to get BIOS to boot the USB flash drive. This is never the same on any two machines. USB flash drive may already be the first thing. If not, it needs to be the first thing the BIOS tries. Good luck.
  7. Boot the flash drive. Usually, “save and exit” in the BIOS boot selection will trigger this.
  8. Wait for the machine to boot. Remind self to turn off silent boot later.
  9. Set password on Arch Linux install.
  10. ssh in via laptop.
  11. Mount the / partition to /mnt, something like mount /dev/sda3 /mnt.
  12. Mount the EFI partition to /mnt/boot, something like mount /dev/sda1 /mnt/boot.
  13. arch-chroot /mnt
    This is important. The package downgrade won’t work without it.
  14. cd /var/cache/pacman/pkg - this is the directory with older package files in it.
  15. Find what kernel you want to downgrade to: ls -ltr linux-hardened* worked to get a listing of linux-hardened package kernels, which is what I’m using.
  16. Downgrade kernel and kernel-headers packages:
pacman -U linux-hardened-7.1.3.hardened1-4-x86_64.pkg.tar.zst linux-hardened-headers-7.1.3.hardened1-4-x86_64.pkg.tar.zst
  1. Remove USB flash drive.
  2. Get rid of “silent boot” - I want to see what it does while booting. Edit the file /boot/grub/grub.cfg. Remove quiet from several kernel invocations.
  3. Power cycle.

I must be getting old. I kinda knew what to do at every step of the way, except I can never remember the dd command to write to a USB flash drive.

I did not mark the kernel with IgnorePkg=linux-hardened in file /etc/pacman.conf. I’m gambling that the next Linux Hardened kernel won’t have the problem. Note that I put exactly zero effort into trying to figure out the problem. Linux kernel stuff is mostly beyond me. The fact that I could back off a version and have Linux Hardened work tells me there’s some kind of regression that will cause complaints, and get fixed.