Running it on real hardware

Every number up to part 8 came from Proxmox VMs, with the hypervisor bridge as the thing watching the wire. I wanted to know if any of it held up on real machines, with real NICs and no hypervisor in the way.

The setup

I used four machines on their own switch: one running xdp-bfd with FRR on top, one FRR peer doing all its sessions in software, one to generate load, and one that only captures. The first three are Intel i5-12500 boxes with Intel I210 NICs, with XDP running in native mode inside the NIC driver, on Ubuntu 26.04 with kernel 7.0.

Without a hypervisor, I needed something else to watch the wire from outside. The switch copies everything the xdp-bfd box and the peer send over to the capture box, which timestamps each packet on its NIC. I checked that against a capture on the peer itself, and the two agreed to within 0.2%.

I moved the whole 1024-session test mesh over as it was, with every kind of session from the earlier parts, plus 40 at 3x10ms. The stress ladder was the one from part 1, scaled up to 12 threads.

Stock bfdd first

I started with plain FRR bfdd doing all 1022 sessions itself. The timer load (L3) didn’t hurt it at all. The real-time load (L4) did: 18,910 flaps across 957 of the 1022 sessions, including the ones with 1 second timers. bfdd stalled for up to 2.5 seconds at a time, and it kept flapping for a while after the stress had stopped.

Then I ran xdp-bfd exactly as it had run on the VMs. It did a lot better, but it wasn’t zero: 698 flaps across 213 sessions. On the VMs the ladder had come out clean, but those runs had 64 sessions, not 1022.

Three things the VMs hadn’t shown

The first one was the dead-man gate from part 7, which stops XDP answering once userspace has been silent for longer than a bound, which was 1 second. But under real-time load, Linux lets real-time tasks use 95% of each one-second period, and normal tasks only get the last sliver of it. A normal task stuck behind real-time hogs ends up running about once a second, so the gaps in the userspace loop came out between 1.05 and 2.1 seconds. The check was taking down sessions that were fine. The default bound is 3 seconds now.

The second was in demand mode. Those sessions were timing out their own Poll even after the peer had answered it. That turned out to have three separate causes, and I fixed all of them.

The third was in the package from part 8. Its systemd unit ran the engine at real-time priority 10, below the priority-50 load, so under L4 it got no CPU at all and 957 sessions went down, once each. It runs at 60 now.

With those three fixed, the ladder came out at 0 flaps on all 1022 sessions. The 3x10ms sessions had a largest gap of 20.2ms against their 30ms budget.

Upgrading under load

Part 8 said systemctl reload and package upgrades hand over without dropping anything. On the real machines I found that neither did. Reload never worked at all, because of a check on the process name, and a package upgrade stopped the engine before installing the new one.

After fixing both, I installed a new package with dpkg -i over a running engine with 1024 sessions. The handover took 10 to 31ms, and the peer saw 0 sessions go down. The packaged engine also went through the whole ladder with 0 flaps, using about 2% of one core for 1024 sessions.

Where the replies were waiting

Next I looked at how long replies took. Inside XDP, building a reply takes about 1.5 microseconds. On the wire the replies were leaving 118 microseconds after the peer’s packet arrived, and 250 at p99.

My first guesses were interrupt moderation and CPU sleep states, but turning either of them off on its own didn’t change much. It turned out to be PCIe power saving (ASPM) on the link to the NIC, which puts the link to sleep between packets. With ASPM off it dropped to 33 microseconds. On these boxes the firmware owns ASPM, so it has to be turned off in the BIOS or with setpci. BFD timers are in milliseconds, so none of this changes detection.

Priority 60 isn’t enough either

The engine was at priority 60 now, but that only works as long as nothing runs above it. That’s the same problem I had with priority 90 back in part 2. So I tried a priority-99 hog against the engine at 60, and got 959 down events.

Linux has another scheduling class, SCHED_DEADLINE, where a task reserves a slice of CPU time in every period, and the kernel runs it ahead of every real-time priority. So the engine now reserves 1ms in every 10ms. Against the same priority-99 hog, that gave 0 flaps. Even 0.2ms was enough. If the kernel refuses the reservation, for example because the service is pinned to only some of the CPUs, the engine logs why and keeps running at priority 60.

Next is what happens when the problem isn’t a busy box but someone on the wire trying to break it: part 10.