Results, and putting FRR in the loop

Part 4 described the RX-clocked design. This is what it measured.

The full matrix: the complete stress ladder plus a five-minute soak, running at ordinary process priority, interoperating with stock FRR on the far end.

For comparison, under that same L3 stress, bfdd flapped 44 times.

The one flap deserves telling in full

It is more informative than a column of zeros would have been.

Under the hrtimer storm, one echoed reply out of roughly 66,000 was delayed 28ms by softirq latency. The peer’s 30ms detect timer fired at the margin, correctly.

What followed, reconstructed packet by packet from the capture: the peer’s Down arrived, userspace noticed via the map within 3ms, transitioned, and answered the peer’s re-establishment Poll with Final 74 microseconds after it arrived. Down, Init, Up, Poll, Final. The complete RFC handshake, in 3.8 milliseconds.

Total session downtime across eleven minutes of hostile load: under 4ms.

So the honest characterisation is not “immune”. Kernel-path TX is immune to scheduler starvation but shares fate with softirq latency. Under extreme timer pressure it grazed the detect budget once and self-healed in less time than a single packet interval.

That is a different universe from the userspace failure mode. But it is a universe with physics in it.

Making FRR drive it

bfdd has a distributed-BFD mode where it delegates all session processing to an external dataplane over a small TCP or UNIX socket protocol called bfddp, designed so vendors can attach hardware dataplanes without patching FRR.

Implementing it took one protocol shim. bfdd creates and owns the sessions, assigns discriminators, and displays state and counters. The packets ride the XDP path. show bfd peers and show bfd peers counters report a session whose packet counts are read out of a BPF map.

The same SCHED_FIFO stress against the FRR-driven session: 0 flaps, p50 8.99, p99 11.01, max 13.01ms, uptime uninterrupted in FRR’s own CLI.

Putting FRR in the control loop cost nothing.

And the integration immediately found a bug

bfdd’s UNIX transport for that dataplane socket, --dplaneaddr unixc:, failed every connect attempt with EINVAL.

That turned into the first of six upstream bug reports, and I wrote it up separately, because the reason it had survived since the feature was introduced is more interesting than the one-line fix.

It set a pattern that held for the rest of the project: every milestone that pushed the engine somewhere new found something in bfdd that only a real dataplane at scale could reach.

Where this left things

At this point I wrote the results up and stopped, because the interesting question, “can plain Linux get line-card behaviour”, had an answer. It is worth being precise about how narrow that answer was.

What existed was lab-grade, single-hop, IPv4-only, with:

• No authentication, echo, or demand mode. Three of RFC 5880’s features simply absent.
• No session continuity. Sessions dropped and re-established across a bfdd restart. The dataplane tore them down on control-plane disconnect and bfdd re-added them on reconnect, which is correct but not hitless.
• One session, ever. The maps were sized for 64. Exactly one had been tested.
• Validation gaps. No spoofing defence worth the name, and no proof that malformed traffic could not reach userspace.
• An asynchronous-peer requirement, structural to the RX-clocked design.
• Numbers from VMs. The stress was applied inside the guest and hit every backend identically, so the comparisons are load-bearing, but the absolute figures still await bare metal.

A benchmark that only holds for one unauthenticated IPv4 session against a cooperative peer is a demo, not an implementation. Everything from here on is the unglamorous work of closing that list, and each item closed found something new, which is why the series keeps going.

Next

First the correctness debt: session continuity, RFC 5880 validation, and an external review whose two most annoying suggestions turned out to be real bugs.