OpenWrt One Problem and Recovery
My OpenWrt One router had a problem, which looked analogous to a disk failure. It went into a reboot loop.
The initial symptom was weird ethernet behavior. The RJ-45 sockets on the OpenWrt One would light up, blink once or twice, and go dark. Repeat every 15-20 seconds.
journalctl -f on the machine on the other end
of the CAT-5 cable showed me this:
Jun 18 12:20:27 modest kernel: igc 0000:05:00.0 enp5s0: NIC Link is Down
Jun 18 12:20:36 modest kernel: igc 0000:05:00.0 enp5s0: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Jun 18 12:20:36 modest kernel: igc 0000:05:00.0 enp5s0: NIC Link is Down
Jun 18 12:20:39 modest kernel: igc 0000:05:00.0 enp5s0: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Jun 18 12:20:58 modest kernel: igc 0000:05:00.0 enp5s0: NIC Link is Down
Jun 18 12:21:04 modest kernel: igc 0000:05:00.0 enp5s0: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Sometimes a reboot loop would last long enough for the kea DHCP server to get an address out to the OpenWrt One.
I did not discover the reboot loop until I plugged in a USB cable to
get a serial console.
I finally had the idea to plug a USB cable in the OpenWrt One’s front USB-C socket.
I plugged the other end in my laptop and ran sudo minicom -D /dev/ttyACM0.
I saw an endless scroll of what looked like booting Linux kernel messages.
There were some messages about “press f to enter failsafe mode” which I was unable to capture, but this screen would also show for a few seconds:
( ( ( OpenWrt ) ) ) [SPI-NAND] U-Boot 2024.10-OpenWrt-r27876+1-3098b4bf0
1. Run default boot command.
2. Boot system via TFTP.
3. Boot production system from NAND.
4. Boot recovery system from NAND.
5. Load production system via TFTP then write to NAND.
6. Load recovery system via TFTP then write to NAND.
7. Load BL31+U-Boot FIP via TFTP then write to NAND.
8. Load BL2 preloader via TFTP then write to NAND.
9. Reboot.
a. Reset all settings to factory defaults.
0. Exit
Press UP/DOWN to move, ENTER to select, ESC to quit
It went away rapidly, so I had to watch another cycle or two of kernel messages. I was finally able to type a down-arrow during the above screen’s appearance. This caused the reboot loop to stop.
“Boot recovery mode from NAND” gets me this:
BusyBox v1.36.1 (2024-10-22 08:16:16 UTC) built-in shell (ash)
_______ ________ __
| |.-----.-----.-----.| | | |.----.| |_
| - || _ | -__| || | | || _|| _|
|_______|| __|_____|__|__||________||__| |____|
|__| W I R E L E S S F R E E D O M
-----------------------------------------------------
OpenWrt SNAPSHOT, r27876+1-3098b4bf07
-----------------------------------------------------
I installed fresh firmware a number of times, trying 24.10.0, 24.10.1. I had no luck with any of them. Once or twice I got to do a small amount of config:
root@OpenWrt:~# /etc/init.d/firewall stop
root@OpenWrt:~# /etc/init.d/firewall disable
root@OpenWrt:~# uci set wireless.radio0.country='US'
root@OpenWrt:~# uci set wireless.radio1.country='US'
root@OpenWrt:~# uci set wireless.radio0.disabled='0'
root@OpenWrt:~# uci set wireless.radio1.disabled='0'
root@OpenWrt:~# uci commit wireless
root@OpenWrt:~# wifi reload
Reboot loop followed.
I posted a question in the OpenWrt forums about this.
I was using the USB-C power supply that came with the OpenWrt One, but I had substituted a longer cable for the factory power cable. I switched to a USB-C to USB-C cable that looked to be of higher quality, and plugged it in.
The USB serial port monitor showed that OpenWrt 24.10.0 had booted
successfully.
I did a sysupgrade to 24.10.2 firmware.
The OpenWrt One has been up ever since, 30 hours at the time of this writing.
I’ve done my usual configuration,
making it bridge the WiFi and ethernet ports,
and letting my server provide DHCP, DNS and routing.
I hate having NATted WiFi clients.
All this goes to show you it’s always cables.