Why does software BFD flap?

BFD is the thing that tells BGP or OSPF a neighbour is gone. Two boxes send each other tiny UDP packets every few milliseconds, and if a few in a row go missing, the link is declared dead and routes get pulled. At 3x10ms (send every 10ms, give up after 3 misses) that happens in 30ms.

The catch is that BFD is only useful if it’s on time. If the sender is late, the other side declares a perfectly fine link dead and tears down routes for nothing. So a slow BFD daemon is worse than no BFD at all.

The common advice is that software BFD can’t be trusted with fast timers when the box is busy. That’s why people run 3x300ms, why SONiC caps software BFD at 300ms, and why real routers push BFD onto line cards. I wanted to know if that’s actually true, why, and whether a normal Linux box could get line-card behaviour out of XDP.

The setup

Three Ubuntu 26.04 VMs (kernel 7.0, virtio-net) on an isolated Proxmox bridge: the box under test, an FRR peer, and a spare. I measured everything with tcpdump on the hypervisor bridge, not inside the guest. That ended up mattering a lot, more on that below.

Baseline: FRR 10.5.1 bfdd, one session at 3x10ms, and a stress-ng ladder on the box under test. Gaps are time between packets sent to the peer, in ms. Normal spacing is 7.5 to 10ms because BFD adds jitter on purpose.

Look at L3. 44 flaps in two minutes, and the p99 is 10.16ms. That’s a perfect number. The stalls that kill the session are rare, a handful per hundred seconds, so they never show up in a percentile. If you graph p99 packet spacing on a dashboard, this box looks healthy while it’s pulling routes every few seconds. Only the max shows it.

So I wrote a dumb daemon to prove the point

Before touching the kernel I wanted a second data point. I wrote the simplest BFD daemon I could: 160 lines, one thread, a recv() with a 2ms timeout as the clock, and the RFC state machine. The plan was to watch it die like bfdd did and then go build the XDP version.

It didn’t die. Under the exact L3 load that flapped bfdd 44 times, it had zero flaps and a 13ms p99. I ran it again because I didn’t believe it.

So why did my 160 lines survive when bfdd didn’t? After digging for a bit, it came down to how Linux picks what runs next. The normal scheduler keeps track of how much CPU each task has used, and when a task wakes up, the one that’s used the least gets to go first. My daemon woke up every 2ms, did a few microseconds of work and went back to sleep, so it had barely used any CPU. Every time it woke up, it got to cut in front of the stress processes almost immediately.

bfdd is a much bigger program. It’s one event loop handling zebra, config and a lot of timers, and the timer stress I was running hit exactly the part where it wakes up. So it wasn’t that userspace can’t do BFD. bfdd just wakes up in a way that this kind of load hurts.

Then I added real-time load

Next I tried the L4 load on it: four SCHED_FIFO hogs, one per vCPU. This time my daemon broke too, with 25 flaps in 60 seconds and a 324ms max gap. Real-time tasks always run before normal ones, so with all four cores busy my daemon only got a tiny slice of CPU each second, and the quick wakeups stopped mattering.

The weird part was the daemon’s log. It showed zero timeouts for the whole run, like the peer’s packets had arrived right on time. The capture on the hypervisor said the peer had seen 300ms of silence and dropped the session.

Here’s what happened. While my daemon wasn’t running, the peer’s packets were piling up in the socket buffer. When it got CPU again it read them all at once, and from its point of view everything had arrived fine. If I’d only had the daemon’s logs I would have missed this completely, which is why every timing and flap number in this series comes from a capture taken outside the box being tested.

So the next question was whether I could get packets out on time even when userspace can’t run at all. That’s part 2.