The Mystery of UDP Routing

While fiddling with my educational TFTP client code I noticed an oddity. Sometimes my client could talk to a TFTP server, but not vice versa. The TFTP server, when it condescended to acknowledge a connection, logged a message like recvmsg(): connection refused. I used a local TFTP server, running on my laptop, to avoid these errors, but the mystery of why the server returned no TFTP ACKs or DATA packets remained.

Here’s a diagram of the network I tried my TFTP client on. I ran TFTP servers on my laptop, on the Qotom fanless server, on the Dell R530, and on the mini PC. I could also run my TFTP client not only on my laptop, but on all three servers, because I use Linux. Linux does not make a false, marketing-and-sales-based distinction between a “server” and a “client”.

All the solid arrows represent CAT-5 cabling. The dashed lines represent WiFi connection. My laptop could wirelessly connect to the AX6000 or the Linksys Velop. When authenticated with the AX6000 access point, it got an IP address like 172.27.0.104. Connected to the velop1, it got an IP address similar to 10.0.0.35.

The Qotom Fanless server had one ethernet port with and address of 172.26.0.1/16, and another with 10.0.40.70/24. It did routing between the two address blocks/ethernet ports. It had another ethernet port with an address of 172.24.0.1. Nothing was plugged in to that port.

Both the velop1 (a Linksys Velop WHW03 V1) and ax6000 (Asus AX6000 TUF) ran in bridging mode. The Qotom fanless server provides an IP address, NTP server, routing info and DNS to clients connecting to the AX6000. The Dell R530 proves the same to clients connecting to the Velop.

The Dell R530 has many ethernet ports. One had the address 10.0.0.1, and both mini PC and velop1 are on that broadcast segment. It had two ethernet ports with nothing plugged in. Those ports had addresses of 10.0.30.1 and 10.0.20.1. Another ethernet port had the address 10.0.40.1. The Qotom fanless server was directly connected via CAT-5 cable, the only other entity on the 10.0.40.0/24 subnet. The Dell R530 did routing between 10.0.a.b subnets, and out to the CenturyLink network.

The switch in the diagram is a GS308E 8 port slightly managed switch. I put it in bridging mode, so that both laptop (when attached to velop1) and the mini-PC got DHCP, NTP and IP routing from the Dell R530.

The mystery

If I ran my server on the Qotom fanless server and my client on my laptop connected to AX6000, UDP packets arrived at the server with IP address of 172.26.0.104, my laptop’s IPv4 address. That’s just one hop.

Moving the server two hops away, I ran my server on the Dell R530, IP address 10.0.0.1, and my client on my laptop connected to AX6000. UDP packets arrived at the server with IP address of not my laptop’s address of 172.26.0.104, my laptop’s IPv4 address, but with address 10.0.40.70, the address of Qotom’s interface connected to the Dell R530

To make investigation a little clearer, I wrote a UDP client that opens a connectionless UDP socket, then sends a string representation of the local IP address it’s bound to, to a server, addressed via IP and port number. The server opens a connectionless UDP socket, reads incoming packets, and prints the address and port the packets arrive from.

Code of programs used to investigate

The client and server program’s I used to investigate are only about 30 lines each. Here they are.

package main

import (
	"fmt"
	"log"
	"net"
	"os"
	"strconv"
)

func main() {
	localAddr := os.Args[1]
	port, _ := strconv.Atoi(os.Args[3])
	serverAddr := &net.UDPAddr{IP: net.ParseIP(os.Args[2]), Port: port}

	conn, err := net.ListenUDP(
		"udp",
		&net.UDPAddr{IP: net.ParseIP(localAddr)},
	)
	if err != nil {
		log.Fatalf("net.ListenUDP: %v\n", err)
	}
	fmt.Fprintf(os.Stderr, "Local address %v\n", conn.LocalAddr())

	if n, _, err := conn.WriteMsgUDP([]byte(localAddr), nil, serverAddr); err != nil {
		fmt.Fprintf(os.Stderr, "problem writing UDP packet: %v\n", err)
	} else {
		fmt.Fprintf(os.Stderr, "wrote %d bytes to %v\n", n, serverAddr)
	}
}

The server code:

package main

import (
	"fmt"
	"log"
	"net"
	"os"
	"strconv"
)

func main() {
	port, _ := strconv.Atoi(os.Args[2])
	addr := &net.UDPAddr{IP: net.ParseIP(os.Args[1]), Port: port}
	conn, err := net.ListenUDP("udp", addr)
	if err != nil {
		log.Fatalf("net.ListenUDP: %v\n", err)
	}

	var buffer [2048]byte

	for {
		fmt.Fprintf(os.Stderr, "waiting for connection\n")
		if c, r, e := conn.ReadFromUDP(buffer[:]); e != nil {
			fmt.Fprintf(os.Stderr, "ReadFromUDP: %v\n", e)
		} else if r != nil {
			fmt.Printf("got %s connection from %s, read %d bytes\n", r.Network(), r, c)
			fmt.Printf("%q\n", string(buffer[:c]))
		}
	}
}

Puzzling Evidence

In the above table, the data values are the IP address that the server program displays when a client program runs on the machine in the “Client” column. The row with “laptop 1” as client is when the laptop uses AX6000 as its WiFi access point. The laptop has an IP address of 172.26.0.104. The row with “laptop 2” as client is when the laptop uses velop1 as its access point, the laptop has an IP address of 10.0.0.35.

The Dell R530 row has some oddities: the Dell R530 column and the laptop 1 column. I have no idea why a program running on the Dell R530 that used an unconnected UDP socket bound to 0.0.0.0 (a.k.a. all interfaces) would show up as an IP address of a network interface that has nothing plugged in it, 10.0.30.1. I can understand showing up as 10.0.40.1, that’s the interface that the Qotom is cabled to. Showing up as 172.26.0.1 to laptop 1, which is associated with the AX6000 WiFi access point, is strange. Notice that every client shows up as 172.26.0.1 to laptop 1 (except laptop 1), and ever client shows up as 10.0.0.1 (except laptop 2) to a server running on mini-PC.

This is all extremely strange but I believe it can be explained. Both the Dell R530 and the Qotom fanless server are set to be routers between several subnets, and to and from The Internet, CenturyLink’s public network via NAT and PPPoE.

Qotom Fanless Server iptables

I conducted my own little “disaster recovery” exercise in October of 2024. I failed over (by hand) from using the Dell R530 as router, NAT and PPPoE connection, to using the Qotom. I set interface enp8s0 as the PPP interface, plugging in the ethernet cable that comes out of CenturyLink’s ODNT on the side of my house.

/usr/bin/modprobe iptable_nat
/usr/bin/sysctl net.ipv4.ip_forward=1
/usr/bin/iptables -t nat -A POSTROUTING -s 172.24.0.0/16 -j MASQUERADE
/usr/bin/iptables -t nat -A POSTROUTING -s 172.26.0.0/16 -j MASQUERADE
/usr/bin/iptables -t nat -A POSTROUTING -s 10.0.0.0/16 -j MASQUERADE
/usr/bin/iptables -A FORWARD -o enp8s0 -i enp4s0 -s 172.24.0.0/16 -m conntrack --ctstate NEW -j ACCEPT
/usr/bin/iptables -A FORWARD -o enp8s0 -i enp6s0 -s 172.26.0.0/16 -m conntrack --ctstate NEW -j ACCEPT
/usr/bin/iptables -A FORWARD -o enp8s0 -i enp7s0 -s 10.0.0.0/16 -m conntrack --ctstate NEW -j ACCEPT
/usr/bin/iptables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

• Interface enp4s0 gets set to address 172.24.0.0/16
• Interface enp6s0 gets set to address 172.22.0.0/16
• Interface enp7s0 acquires an IP address of 10.0.40.70 from the Dell R530’s kea-dhcp4
• Interface enp8s0 acquires a globally routeable IP address from CenturyLink’s PPP mystery system.

These rules are overly complicated. I had no need to do NAT between two of my machines, which the laptop 1 column of the Puzzling Evidence table show was happening.

I (metaphorically) ripped out all the iptables masquerading and forwarding in favor of a simple /usr/bin/sysctl net.ipv4.ip_forward=1

After removing all the masquerading and NAT, only the mini-PC gets an apparently NAT-ed address.

Dell R530 iptables

Running my UDP client on 172.26.0.104, but having it show up at machine 10.0.0.2 with a source address of 10.0.0.1 is very revealing. It made me think that the NAT “masquerade” got applied to every interface, not just when routing packets through the PPP interface.

I modeled the Qotom’s iptables setup after that of the Dell R530. However, I’ve learned something since I set up the Dell R530’s routing and NAT. I cleaned up the Dell R530’s iptables setup:

/usr/bin/iptables -A FORWARD -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
/usr/bin/iptables -A FORWARD -s 10.0.0.0/16 -o eno3 -m conntrack --ctstate NEW -j ACCEPT
/usr/bin/iptables -A FORWARD -s 172.16.0.0/12 -o eno3 -m conntrack --ctstate NEW -j ACCEPT
/usr/bin/iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

/usr/bin/iptables -t nat -A POSTROUTING -s 10.0.0.0/16 -j MASQUERADE
/usr/bin/iptables -t nat -A POSTROUTING -s 172.16.0.0/12 -j MASQUERADE

Interface eno3 ends up getting set to do “VLAN 201” because CenturyLink is bad, and the pppd makes a virtual interface ppp0 that uses eno3 as its physical layer.

The two iptables invocations that turn on NAT don’t have a -o $INTERFACE parameters. “Masquerade” was turned on for all interfaces. That explains why the laptop 2 column of the Puzzling Evidence table contains almost all 10.0.0.1 IP addresses. The nat table had MASQUERADE turned on no matter where what the destination address of an IP packet.

I changed the MASQUERADE:

/usr/bin/iptables -t nat -A POSTROUTING -s 10.0.0.0/16   -o ppp0 -j MASQUERADE
/usr/bin/iptables -t nat -A POSTROUTING -s 172.16.0.0/12 -o ppp0 -j MASQUERADE

I believe that means any IP packet with a source address that’s in one of my LAN subnets that’s routed through ppp0 gets it’s source address set to that of ppp0.

Above, the same procedure used to get data for the Puzzling Evidence table, only after removing the Qotom’s iptables rules, cleaning up the Dell R530’s iptables rules, and limiting NAT masquerade to the ppp0 interface. No masquerading of any sort occurred. The IP routing on the Dell R530 did choose an appropriate interface and the IP address for that interface as the source address.

The Qotom does have different source IP addresses depending on the destination subnet, but that’s to be expected. I could force a source address by binding the client’s unconnected UDP port to 172.26.0.1 or 10.0.40.70 instead of 0.0.0.0, “all interfaces”.

I can now run my homemade tftp client on my laptop when connected to the AX6000, and transfer a file to a TFTP server running on my mini-PC. That’s a hop to the AX600 via WiFi, ethernet to the Qotom, Qotom routes to Dell R530, R530 routes to mini-PC via ethernet. Noticing an oddity led me to clean up my network routing, and generally improve its resilience.