Every bug came from a capture, none from a log

This closes the series. Part 1 started with folklore and a stress ladder; nine parts later the engine does dual-stack, echo, and multihop, and six bugs went upstream to FRR.

The thing worth keeping is none of that. It is the method.

Every bug came from a capture, none from a log

That is not a rhetorical flourish. Going through them:

• The Init-loop from a stale transmit schedule after timer renegotiation.
• The socket buffer convincing a starved daemon that packets were arriving on time, while the peer watched 300ms of silence.
• The etf qdisc blackholing ARP, taking the interface with it.
• The pipeline scheduling Poll answers a second into the future, producing a perfectly periodic renegotiation loop.
• The eaten Final: userspace TX suppressed at exactly the moment the Final needed sending, giving a stable 32.6ms failure loop.
• The same timestamp race written twice by the same author, on two sides of the kernel boundary, three weeks apart, producing the identical 18-quintillion millisecond log line both times.
• The multihop injector aimed at the wrong port, passing three tests while validating nothing.
• The echo returns dropped by GTSM before the echo branch was reached, because returns arrive at TTL 254 by definition.

All of them produced healthy-looking logs and a wire that told a different story. Several were introduced by the project’s own tooling and design decisions, including ones I was confident about.

The three things that survived

The first is an observer outside the system under test. The hypervisor saw what the guests could not, and when a starved daemon’s logs said packets were arriving on schedule while the peer reported 300ms of silence, only an instrument outside both could settle which one was lying. Any measurement taken by the thing being measured is suspect precisely when it matters most, which is the same lesson the echo loss counter taught in a different shape and which I had to learn twice.

The second is distributions over averages. The p99-fine, max-fatal pattern from part 1 is invisible any other way: a dashboard showing p99 packet spacing would have reported a perfectly healthy daemon straight through 44 flaps. The same idea rescued the echo milestone, where flap count turned out to vary from 0 to 20 across runs of identical code, and replacing it with a per-session distribution turned an unsupportable claim into a defensible bound.

The third is a refusal to let any claim stand untested when a tcpdump could settle it, whether the claim is the folklore’s, a reviewer’s, or my own. That is why the repo ships every capture behind every number, and why the numbers that did not survive contact with a capture are in there too.

What I would say to someone starting something similar

Measure the thing you are about to replace, before you replace it. I nearly skipped that step, and the naive daemon that refused to die reframed the entire project: the problem was never “userspace”, it was the architecture of the wakeup path.

Write down the suggestions you rejected and why. That file is now the most useful documentation in the repo.

And treat a test that passes on the first try with suspicion. Twice in this project a green result meant the harness was pointed somewhere the code was not.

The honest limitations, restated

All numbers are from VMs. The stress was applied inside the guest and hit every backend identically, so the comparisons are load-bearing, but the absolute figures await a bare-metal reproduction, which remains the top item on the roadmap.

RX-clocked TX requires an asynchronously-clocked peer. Two RX-clocked implementations facing each other would echo each other into silence.

No authentication and no demand mode.

And kernel-path TX is immune to scheduler starvation but shares fate with softirq latency. It grazed the detect budget once in eleven minutes of hostile load and self-healed in 3.8ms. That is a different universe from the userspace failure mode, but it is a universe with physics in it.

Everything is at github.com/w453y/xdp-bfd, captures included.