Line-card BFD for plain Linux
Building a BFD offload whose fast path runs in XDP, from measuring the folklore about software BFD to a dual-stack engine with echo and multihop. Every claim backed by a packet capture. Starts with a primer if the terms are new.
- What all of this means
A plain-language primer on the terms the rest of this series uses. BFD, FRR, XDP, eBPF, softirq, p99, and the rest. Read this first if any of those are new.
- Measuring the folklore
Everyone says software BFD cannot hold aggressive timers under load. Before building anything, I measured it. The number that should worry you is not the 970ms gap, it is the p99 sitting at 10.16ms in the same run.
- The folklore is partly wrong
I wrote a 160-line BFD daemon expecting it to fail like bfdd did. It ran clean through the load that flapped bfdd 44 times. Userspace was never the problem as a category. The wakeup path was.
- The TX bake-off
Five transmit architectures under identical RT starvation. The SO_TXTIME approach produced the best p99 of any backend tested and flapped 48 times, worse than doing nothing special at all.
- RX-clocked TX
XDP is an ingress hook. It cannot originate a packet. The way out is to stop trying: rewrite the packet the peer just sent you and bounce it back, so your transmit clock becomes the peer's transmit clock.
- Results, and putting FRR in the loop
Zero flaps through the full stress ladder, one instructive exception that self-healed in 3.8ms, and an unmodified FRR driving the whole thing over its own dataplane protocol.
- Hardening, and a review that found real edges
Session continuity across control-plane restarts, RFC 5880 validation, and an external code review whose two most annoying suggestions turned out to be genuine bugs.
- One session to sixty-four
The maps had been sized for 64 sessions since the beginning, and exactly one had ever been tested. Going to 64 found five bugs, one of them upstream, and the design's own false-flap boundary.
- IPv6, and the bake-off argument in miniature
Dual-stack in five steps. The measurement that justifies the kernel reply runs both families on one box at the same instant, with the only variable being which side of the kernel boundary the transmit clock lives on.
- Echo mode, and instrumentation that cannot see its own failure
XDP cannot originate packets, so echo split along the line the hardware draws. Then the loss counter read zero through a 2.6 second stall, and both readings were correct.
- Multihop, and a test that validated nothing
One comparison changes from equals 255 to greater than a minimum. The TTL work passed all three cases and proved nothing, because the injector was sending to the wrong port.
- Every bug came from a capture, none from a log
Ten milestones, six upstream bug reports, and one method that survived all of it. Several of the bugs were introduced by the project's own tooling and design decisions, including ones I was confident about.