Line-card BFD for plain Linux · part 1 of 12
Measuring the folklore
BFD is the failure detector under BGP, OSPF, and IS-IS. Two routers exchange small UDP packets at a negotiated interval, and if one side misses enough of them, typically 3 packets in 30ms at aggressive timers, it declares the link dead and the routing protocol withdraws routes.
The entire value of the protocol is timing. An implementation that sends late is worse than no BFD at all, because a false timeout tears down routes for a link that is actually fine. The failure detector becomes the failure.
The folklore
Operational folklore says software BFD cannot be trusted with aggressive timers under CPU load. It is why people configure 3x300ms instead of 3x10ms, why SONiC’s software BFD documentation caps timers at 300ms, and why hardware routers offload BFD to line cards. Juniper goes as far as dedicating an offload CPU with DPDK flow filters on the vSRX, which is a vendor telling you the forwarding plane cannot be trusted to deliver BFD packets to a busy control plane on time.
I wanted three things: to quantify the actual failure modes instead of repeating the folklore, to understand why userspace BFD fails when it fails, and to see whether plain Linux on a commodity NIC can get line-card behaviour using XDP.
This is a lab project, a few weeks of evenings on a Proxmox testbed. Every claim in it is backed by a packet capture in the repo.
The testbed, and one decision that mattered
Three Ubuntu 26.04 VMs (kernel 7.0, virtio-net with multiqueue) on an isolated Proxmox bridge: a device under test, an FRR peer, and a load generator that ended up mostly unused, because the interesting load is on the DUT itself.
The measurement instrument is tcpdump on the hypervisor bridge, not inside the guest. The host sees actual wire times and does not care what the guest’s processes believe.
That choice turned out to matter more than expected, and part 2 has a concrete example of process logs and the wire disagreeing completely.
The baseline
FRR 10.5.1’s bfdd, a single-hop session at 3x10ms (30ms detect time), and an escalating stress-ng ladder on the DUT. Gaps are DUT-to-peer inter-packet times from the host capture. Nominal spacing with RFC jitter is 7.5 to 10ms.
| Level | Load | p50 | p99 | max gap | session |
|---|---|---|---|---|---|
| L1 | 4x CPU hogs (fair sched) | 8.81 | 10.03 | ~10ms | survived |
| L2 | 8x CPU + context-switch churn | 8.73 | 10.03 | 750ms | flapped |
| L3 | CPU + timer/timerfd/hrtimer pressure | 8.82 | 10.16 | 970ms | 44 flaps in 120s |
| L4 | 4x SCHED_FIFO prio-50 hogs | 8.82 | 287 | 960ms | ~40% of packets never sent |
The number that should worry you
It is not the 970ms.
It is the p99 sitting at 10.16ms in the same run. Nominal. Perfect.
The starvation events that kill sessions are rare, a handful per hundred seconds, so they are invisible to percentile monitoring. A dashboard graphing p99 packet spacing on that box would have shown a completely healthy BFD daemon while it flapped 44 times in two minutes.
If your alerting is percentile-based, you do not find out about this class of failure from your monitoring. You find out from the routing protocol, after the withdrawal.
That pattern, p99 fine and max fatal, is invisible in any summary statistic that is not a maximum, and it shows up repeatedly in the rest of this series.
Next
Having measured the folklore, the obvious next step was to build the XDP path and prove the point. Instead I wrote the dumbest possible userspace BFD daemon first, expecting it to die the same way, and it refused to.