Building xdp-bfd
How I built a BFD engine that runs in XDP, from why software BFD flaps under load to running it on real hardware and under attack. Every timing and flap number comes from a packet capture.
- Why does software BFD flap?
Everyone says software BFD can't hold fast timers under load. Before building anything I measured it, and then wrote a tiny daemon that was supposed to fail and didn't.
- Getting transmit out of userspace
I tried five ways to send BFD packets under real-time starvation. The fancy one had the best p99 and flapped the most. The one that worked was to stop sending packets and bounce the peer's back instead.
- Results, FRR, and hardening
Zero flaps through the whole stress ladder, one flap that fixed itself in 3.8ms, FRR driving the engine over its own protocol, and a code review whose two most annoying comments were real bugs.
- From 1 session to 8192
Everything so far had been one session. Going to 64 found five bugs, going to 1024 needed a rewrite of how the engine waits, and 8192 ended up limited by the software peer, not the engine.
- IPv6, echo, and multihop
IPv6 gave me a clean experiment proving the kernel reply is what matters. Echo is the one feature XDP can't fully do. Multihop passed every test while testing nothing.
- Demand mode, auth and hitless restarts
Demand mode, SHA1 inside an XDP program, getting FRR to hand keys to a data plane, and restarting the engine under 1024 live sessions without a single flap.
- Checking my own work
Getting the code reviewed, an engine that kept answering after it died, fuzzing, and a test that had been passing for weeks without ever running.
- Shipping it
Packaging xdp-bfd so it runs without root, finding out which kernels and FRR versions it actually works with, and a bug that only showed up on VyOS.
- Running it on real hardware
Every number so far came from VMs. On four real machines, stock bfdd flapped 18,910 times under load, and xdp-bfd found three bugs the VMs had hidden before it got to zero.
- Under attack
What someone on the wire can do to xdp-bfd, from floods on the VMs to an hour of everything at once on real hardware, plus a neighbour that holds the key. Then where it's at, and looking back.