Abdul Wasey — Writing

Line-card BFD for plain Linux, part 11: every bug came from a capture, none from a log

awasey8905@gmail.com (Abdul Wasey) — Wed, 05 Aug 2026 00:00:00 GMT

This closes the series. Part 1 started with folklore and a stress ladder; nine parts later the engine does dual-stack, echo, and multihop, and six bugs went upstream to FRR.

The thing worth keeping is none of that. It is the method.

Every bug came from a capture, none from a log

That is not a rhetorical flourish. Going through them:

The Init-loop from a stale transmit schedule after timer renegotiation.
The socket buffer convincing a starved daemon that packets were arriving on time, while the peer watched 300ms of silence.
The etf qdisc blackholing ARP, taking the interface with it.
The pipeline scheduling Poll answers a second into the future, producing a perfectly periodic renegotiation loop.
The eaten Final: userspace TX suppressed at exactly the moment the Final needed sending, giving a stable 32.6ms failure loop.
The same timestamp race written twice by the same author, on two sides of the kernel boundary, three weeks apart, producing the identical 18-quintillion millisecond log line both times.
The multihop injector aimed at the wrong port, passing three tests while validating nothing.
The echo returns dropped by GTSM before the echo branch was reached, because returns arrive at TTL 254 by definition.

All of them produced healthy-looking logs and a wire that told a different story. Several were introduced by the project’s own tooling and design decisions, including ones I was confident about.

The three things that survived

The first is an observer outside the system under test. The hypervisor saw what the guests could not, and when a starved daemon’s logs said packets were arriving on schedule while the peer reported 300ms of silence, only an instrument outside both could settle which one was lying. Any measurement taken by the thing being measured is suspect precisely when it matters most, which is the same lesson the echo loss counter taught in a different shape and which I had to learn twice.

The second is distributions over averages. The p99-fine, max-fatal pattern from part 1 is invisible any other way: a dashboard showing p99 packet spacing would have reported a perfectly healthy daemon straight through 44 flaps. The same idea rescued the echo milestone, where flap count turned out to vary from 0 to 20 across runs of identical code, and replacing it with a per-session distribution turned an unsupportable claim into a defensible bound.

The third is a refusal to let any claim stand untested when a tcpdump could settle it, whether the claim is the folklore’s, a reviewer’s, or my own. That is why the repo ships every capture behind every number, and why the numbers that did not survive contact with a capture are in there too.

What I would say to someone starting something similar

Measure the thing you are about to replace, before you replace it. I nearly skipped that step, and the naive daemon that refused to die reframed the entire project: the problem was never “userspace”, it was the architecture of the wakeup path.

Write down the suggestions you rejected and why. That file is now the most useful documentation in the repo.

And treat a test that passes on the first try with suspicion. Twice in this project a green result meant the harness was pointed somewhere the code was not.

The honest limitations, restated

All numbers are from VMs. The stress was applied inside the guest and hit every backend identically, so the comparisons are load-bearing, but the absolute figures await a bare-metal reproduction, which remains the top item on the roadmap.

RX-clocked TX requires an asynchronously-clocked peer. Two RX-clocked implementations facing each other would echo each other into silence.

No authentication and no demand mode.

And kernel-path TX is immune to scheduler starvation but shares fate with softirq latency. It grazed the detect budget once in eleven minutes of hostile load and self-healed in 3.8ms. That is a different universe from the userspace failure mode, but it is a universe with physics in it.

Everything is at github.com/w453y/xdp-bfd, captures included.

Enabling echo mode took the session down, and the kernel picked the wrong address

awasey8905@gmail.com (Abdul Wasey) — Mon, 03 Aug 2026 00:00:00 GMT

Commit 7786df1467, “bfdd: gate IPv6 echo reflection on known sessions”, is a good change. An unguarded echo reflector will bounce a self-addressed packet back to whoever sends one, which is an amplification primitive. Requiring the packet’s source and local address to name an existing BFD session closes that.

The problem is that the sending side never picks a source address, so the guard also rejects the legitimate echoes of its own peer. Turning on echo-mode takes a working IPv6 session out of service.

Who chooses the source

ptm_bfd_echo_snd() sends IPv6 echoes through the shared per-VRF socket bvrf->bg_echov6. That socket is not bound to any address. So the kernel chooses a source from the outgoing interface, following RFC 6724 source address selection.

On the receiving end, bp_bfd_echo_in() builds its lookup key from that chosen source together with the local address, and calls bfd_key_lookup(). If the kernel picked anything other than the session’s configured local-address, the key does not match a session, the echo is dropped, and the sender’s echo detection eventually declares the path dead.

The session goes down. Not a degraded feature: an outage caused by enabling a feature.

Why most people never see it

RFC 6724 does longest-prefix match first. If your addresses live in different prefixes, the rule picks the one in the peer’s prefix, which is the right one, and everything works.

You need more than one address inside the peer’s own prefix for the selection to become ambiguous. That happens with service or anycast addresses on the link, during renumbering, or with privacy extensions turned on. Common enough to matter, rare enough that the guard shipped without anyone hitting it.

IPv4 is unaffected, and for an interesting reason. bp_udp_send_fp() builds the frame with source equal to destination, so the neighbour’s forwarding plane loops it. The received source is therefore always the peer’s own address, by construction. The v4 path never asks the kernel to choose.

The fix, and why it is not a setsockopt

The source is attached per message as an IPV6_PKTINFO control message rather than set once with setsockopt:

if (pktinfo != NULL) {
        size_t used = msg.msg_controllen ? CMSG_SPACE(sizeof(ttlval)) : 0;

        msg.msg_control = msgctl;
        msg.msg_controllen = used + CMSG_SPACE(sizeof(*pktinfo));

        cmsg = used ? CMSG_NXTHDR(&msg, CMSG_FIRSTHDR(&msg)) : CMSG_FIRSTHDR(&msg);
        cmsg->cmsg_level = IPPROTO_IPV6;
        cmsg->cmsg_type = IPV6_PKTINFO;
        cmsg->cmsg_len = CMSG_LEN(sizeof(*pktinfo));
        memcpy(CMSG_DATA(cmsg), pktinfo, sizeof(*pktinfo));
}

The used dance at the top is because the hop-limit cmsg may already be in the buffer, so this appends rather than overwrites. That is the sort of detail that is invisible in a description and unavoidable in the code.

It cannot be sticky. The same socket carries every session’s echoes and the reflection path. A source set for one session would leak into another session’s reflection, which trades one wrong-source bug for a different one. The reflection call site passes NULL and keeps the kernel’s choice, which is correct there.

The caller side is three lines and says exactly what it is defending against:

/*
 * The echo socket is shared and unbound, so the kernel would
 * otherwise pick any address on the outgoing interface. The peer
 * reflects only echoes whose source names a known session, so an
 * arbitrary source gets the echo dropped there.
 */
memcpy(&pktinfo.ipi6_addr, &bfd->key.local, sizeof(pktinfo.ipi6_addr));
if (bfd->ifp)
        pktinfo.ipi6_ifindex = bfd->ifp->ifindex;
pktinfop = &pktinfo;

Writing a test for source address selection

This was the genuinely fiddly part, and most of the diff.

The topotest needs the kernel to pick the wrong address on purpose. My first instinct was address ordering: add the decoy afterwards and let the rules sort it out. That does not work. Among equally preferred candidates, Linux keeps selecting the address that has been on the interface longest, so adding a decoy later has no effect at all.

What does work is RFC 6724 rule 3, which prefers non-deprecated addresses. Put a second global address on r1 in the session’s prefix, then mark the session’s own local-address deprecated with preferred_lft 0. Now the kernel actively prefers the decoy.

The test also asserts, as a precondition, that the kernel really does select the decoy. Without that assertion, a future change to source selection would make the test pass because the bug can no longer be triggered, rather than because the bug is fixed. A test that quietly stops testing anything is worse than no test.

Echo mode is configured on both routers, because bfdd only opens the per-VRF IPv6 echo socket when a session uses echo. With echo on the sender alone, the peer never reaches the reflection path at all.

Verified both ways: patched, everything passes. With the fix reverted, the reflection and stability assertions fail, r1 logs up -> down reason:echo-failed about every 1.3 seconds, and the source-selection precondition still passes, which is exactly the signature you want from a regression test.

Open as #22920.

What I took from it

The hardening commit was right, and it exposed a latent gap rather than creating one: the sender was always relying on the kernel to guess, and nothing had checked the guess before. Adding a correctness requirement on the receive side turned a silent assumption on the send side into an outage.

And the test taught me more than the fix. Making a kernel reliably do the wrong thing, on demand, took several attempts and a rule in RFC 6724 I had not needed before.

Line-card BFD for plain Linux, part 10: multihop, and a test that validated nothing

awasey8905@gmail.com (Abdul Wasey) — Sun, 26 Jul 2026 00:00:00 GMT

Multihop looks like the smallest milestone in the project.

Single-hop BFD proves adjacency with the TTL: a control packet must arrive at 255, so anything routed has been decremented and is rejected. Multihop gives that up by definition, since packets cross routers and arrive lower, and the receiver enforces a configured minimum instead.

One comparison changes from == 255 to >= minimum. The plumbing was already there and being thrown away: the dataplane registration carries a TTL byte and a multihop flag, bfdd fills both, and the engine parsed the byte and discarded it while the reject mask refused any session carrying the flag.

Where the comparison can live

The GTSM check sits in the parser, before any session lookup, and that placement is the point: a spoofed flood at TTL 64 dies in a handful of instructions having touched no map.

But a per-session minimum is not known until the session is found. Enforcing it there would mean every rejected packet paying for a hash lookup first, trading away the cheapest rejection path in the engine to support a mode most deployments never enable.

The resolution is a single bit in a flags map saying whether any multihop session exists at all. When it is clear, the parser behaves exactly as before, byte for byte. When it is set, the verdict for a below-255 packet defers to after the config lookup:

/* Deferred GTSM. A control packet that did not arrive at 255 is
 * acceptable only if it names a configured session whose minimum
 * admits it. An unconfigured pair must still drop: the
 * promiscuous PASS above exists for observation, not to relax
 * GTSM. Single-hop sessions carry min_ttl 255, so nothing below
 * 255 reaches them and their behaviour is unchanged. */
__u8 pttl = iph ? iph->ttl : (ip6 ? ip6->hop_limit : 0);

if (pttl != 255) {
        __u32 mt = (cfg && cfg->min_ttl) ? cfg->min_ttl : 255;

        if (!cfg || pttl < mt) {
                count(3);
                return XDP_DROP;
        }
}

The !cfg half of that condition matters more than it looks. The lookup’s miss path returns XDP_PASS for the standalone observer, so dropping only on pttl < mt would have quietly started leaking low-TTL packets for unconfigured address pairs straight to userspace, undoing one of the hardening guarantees from part 6. A relaxation added for one feature reopening a hole closed by another is the kind of interaction that no single test looks for.

Note also min_ttl defaulting to 255 when there is no config. Single-hop sessions carry 255, so the comparison is unchanged for them and the multihop code costs them nothing but a branch that is never taken.

The test that matters is not that multihop works. It is that a packet at TTL 200 aimed at a single-hop session is still dropped while multihop is active elsewhere on the same box. Enabling a relaxation for one session must not relax anything for another.

Then the milestone taught the same lesson the project keeps teaching

At this point I had a clean change, a defensible design for where the check lives, and a harness that exercised all three cases: above the minimum accepted, below it dropped, single-hop still strict. Everything passed. I was ready to call it done in an afternoon, which should have been the warning.

It validated nothing. The injector was sending to UDP 3784, and RFC 5883 multihop runs on 4784.

Real multihop traffic was reaching the parser, failing a port check that only knew about single-hop, and being passed to a stack where nothing was listening for it. The capture said so immediately: the neighbour transmitting to 4784, the engine replying to 3784, tcpdump labelling one Multihop and the other Control.

A test that agrees with you because it exercises the wrong thing is worse than no test, and the only defence is to check what the wire actually carries rather than what the harness was told to send.

The reply had a subtler version of the same problem

The kernel bounce rewrites the received frame in place and sends it back, which for single-hop is correct by accident: the packet arrives at 255 and leaves at 255.

For multihop the reply would go out already decremented, lose more crossing back, and be measured against the peer’s own minimum. The session would establish in one direction and fail in the other, and it would present as a peer that flaps rather than as a TTL problem.

RFC 5883 asks for multihop to be sent at 255 precisely so the receiver can count hops, which the userspace path had always done via a socket option. The bounce now matches it, with an incremental checksum update for v4 and a plain assignment for v6.

One IPv6 asymmetry worth recording

The single-hop socket sets IPV6_MINHOPCOUNT to 255, so the kernel discards low-hop packets before userspace ever sees them. A good defence there and a fatal one for multihop, where every packet arrives below 255 by definition.

The multihop socket deliberately omits it and lets XDP enforce the per-session minimum instead, which loses nothing, because that is where the minimum is known anyway.

Reading the validation took a trick

Both families were validated the same way: above the minimum accepted, below it dropped, a single-hop session still strict at 255 with multihop live, and an injected packet below 255 reflected back at 255.

The last of those needed a trick to read, because injected packets and their replies are indistinguishable from live session traffic by address and port. But scapy leaves the IPv6 traffic class at zero while FRR sets it, and the bounce preserves whatever arrived, so the replies to injected packets identify themselves.

That surfaced a small trap of its own: tcpdump omits the class and flowlabel fields entirely when both are zero, so a pattern written against the normal output format silently matches none of them. Twenty replies sat in the capture looking, to a careless grep, like nothing at all.

The method, which is the only part of this project I would defend without qualification.

Line-card BFD for plain Linux, part 9: echo mode, and instrumentation that cannot see its own failure

awasey8905@gmail.com (Abdul Wasey) — Sat, 25 Jul 2026 00:00:00 GMT

Echo was the feature I expected to be easy.

It is the one part of BFD where the packet belongs to neither endpoint’s control plane. The originator sends a packet addressed to itself, the neighbour’s forwarding plane loops it straight back without its BFD daemon ever seeing it, and the originator times the round trip. It tests the neighbour’s data path rather than its software, which is exactly the sort of thing a dataplane engine should be good at. Every other milestone had been about dragging work down into the kernel; this one looked like it belonged there already.

It turned out to be the only feature in the project that the architecture forbids outright, and working out why took longer than any of the code.

XDP cannot originate packets, and there is no way around it

XDP_TX is a verdict on a frame that has just arrived. That is precisely what makes the control path work here, the peer’s packet is the clock, but an echo has no inbound packet to clock off.

There is no timer callback that can transmit. No helper produces a second packet from one. The redirect helpers move the frame rather than copy it. The kernel-side escape is bpf_clone_redirect, which exists only for TC and lwt, and TC never sees anything at the echo cadence here because control packets are XDP_TX’d straight past it.

Putting echo transmission in the kernel would have meant moving the control bounce out of XDP into TC: skb allocation in the hot path, and a rewrite of the one mechanism this whole project rests on.

That is the trade in full. Gain a kernel-side echo originator, lose the RX-clocked control path that is the entire result. I declined it, and I want to be honest that declining took a while, because the alternative is admitting a feature does not fit and shipping something less than you intended. Writing down why took longer than the feature would have.

So the milestone split along the line the hardware draws

Answering a neighbour’s echo is a reflection, and a reflection has a packet in hand, so it lives entirely in XDP: MAC swap, TTL decrement, checksum recompute, XDP_TX, no session lookup, payload untouched. It is the same trick as the control path, applied to somebody else’s packet.

Originating one has no such luxury, so it runs from userspace over a raw socket. A raw socket rather than a normal UDP one, because a self-addressed packet through the latter is routed to loopback and never reaches the wire at all.

That split is not a compromise I am papering over. One half is a production capability and the other is a diagnostic, they have different reliability properties, and the documentation says so in those words.

The reflector’s argument is capability rather than speed. With ip_forward=0 the host stack discards a self-addressed echo as a martian, so a non-router host cannot participate in echo at all. The reflector lets it, without advertising the box as a forwarder.

The guard matters as much as the mechanism, and it is four lines:

if (iph->ttl != 255)      /* GTSM: single-hop echoes only */
        return XDP_PASS;
/* A classic echo is self-addressed to the originator. */
if (iph->saddr != iph->daddr)
        return XDP_PASS;
/* Reflect only for a peer of an echo-active session; otherwise
 * this is an arbitrary 3785 packet (amplification vector). */
struct bfd_addr esrc;
key_set_v4(&esrc, iph->saddr);
if (!bpf_map_lookup_elem(&echo_peers, &esrc))
        return XDP_PASS;

Without that last lookup this is a reflector for anyone who sends a self-addressed packet to port 3785, which is an amplification primitive with my name on it. The map is written only for peers of sessions that have echo enabled, so the box answers exactly the neighbours that asked and nobody else. XDP_PASS rather than XDP_DROP on the rejects, because an unrecognised packet on 3785 is not necessarily hostile and the stack can have it.

Two spec errors, surfaced the moment there was code

The design document proposed demuxing returned echoes by source address and port, following the unaffiliated-echo draft. That cannot work. The return is still addressed to our own local address, which names no session, and several sessions may share one.

The discriminator written into the payload is the only thing that identifies the session, and classic echo never loops Your Discriminator, so it survives the trip untouched. Which makes the return path a lookup on the payload rather than on the headers:

__u32 ed = bpf_ntohl(eb->my_disc);
struct session_key *ek = bpf_map_lookup_elem(&echo_disc, &ed);
if (!ek)
        return XDP_PASS;
struct session_state *es = bpf_map_lookup_elem(&bfd_sessions, ek);
if (!es)
        return XDP_PASS;
es->echo_last_seen_ns = bpf_ktime_get_ns();

Two lookups where the address would normally have given it in one: the discriminator names a session key, the key names the session. That indirection is the whole cost of the addresses being useless here.

The second was subtler and cost a debugging round. Every return was being dropped before the echo branch was reached, because the v4 parser applies GTSM first and returns arrive at TTL 254 by definition. The symptom was the reject counter climbing at exactly the echo rate while the echo counter stayed flat, which is the kind of thing a counter tells you and a log does not.

Detection is advisory, permanently

The sweep marks each echo-active session alive or not and reports it, but never feeds the session state machine.

With userspace transmission, a local scheduling stall is indistinguishable from a path failure: echoes stop leaving, returns stop arriving, the timestamp goes stale. Wiring that into the state machine would convert our own scheduling delay into a teardown, which is the exact failure this engine was built to avoid.

Verified from both sides: disabling forwarding on the neighbour stops the returns, loss climbs one for one, echo liveness flips, and all 64 control sessions stay up.

The sharpest lesson in the milestone

Under load, echo transmission stalled for 2.6 seconds at a time. The loss counter read zero throughout. The liveness flag read healthy.

Both were correct. Both were useless.

Loss only increments when an echo is outstanding as the next one falls due, and during a total stall nothing ever falls due. The liveness figure is printed on transmit, and transmit is what stopped.

Instrumentation driven by the thing being measured cannot observe that thing failing.

A windowed inter-send gap, sampled independently, showed it immediately: 13ms idle against a 10ms interval, already 30% over budget with nothing running.

And a lesson about my own habits

Measuring the cost of the always-on additions produced a second lesson.

Flap count had been the working metric since the bake-off. At 64 sessions it turned out to vary from 0 to 20 across runs of identical reference code, which is wider than any effect worth measuring.

A full day went into chasing that variance. The cause was a debug bfd peer line persisted in the neighbour’s config file, surviving every restart, contaminating every run including the supposed baselines. Several confident conclusions were drawn and retracted in the process, each overturned by the next run.

The metric was replaced with the per-session maximum transmit gap, which yields 64 numbers per run instead of one rare event. The answer became a bound rather than a claim: the reference build’s own median spans 22.3 to 27.1ms across two runs, the echo build sits at 23.2 inside that spread, so the additions cost less than roughly 5ms of median. Which is not the same as zero, and the docs say that too.

The upstream bug had two halves facing each other

RFC 5880 requires that echo not be transmitted faster than the neighbour advertises it can receive. For an offloaded session bfdd never performs that negotiation, which I wrote up separately.

But that fix has no input unless the dataplane reports the neighbour’s advertisement upward, and this engine was hardcoding it to zero. The engine also had the mirror of the same bug outbound: it advertised its own echo receive interval as zero, which the RFC defines as “cannot receive echo packets”, so no conforming neighbour would ever have echoed at it.

Which explains, in retrospect, why every reflector test until then had used a hand-built frame.

With both halves fixed the chain closes and can be watched end to end: the neighbour advertises 200ms, the daemon negotiates it against a locally configured 50ms, and the wire cadence moves from 50 to 200. And the reflector finally answered a real implementation rather than a scapy script: 433 echoes, 433 reflected, 12 microseconds minimum turnaround, on a host that would have dropped every one of them as a martian.

Multihop, which looks like the smallest milestone and taught the same lesson the project keeps teaching.

The RFC says negotiate, but nobody was listening for offloaded sessions

awasey8905@gmail.com (Abdul Wasey) — Sat, 25 Jul 2026 00:00:00 GMT

Echo mode is the part of BFD where you send a packet addressed to yourself, the peer’s forwarding plane loops it back, and you time the round trip. It tests the peer’s data path rather than its control plane, which is the whole point.

RFC 5880 section 6.8.9 puts a limit on it: you must not transmit echo packets faster than the interval the remote system advertises in its Required Min Echo RX field. The effective interval is the larger of what you want and what the peer says it can take.

While implementing echo in xdp-bfd, I found that for offloaded sessions bfdd never performed that negotiation.

Where the negotiation lives, and why it was unreachable

For a normal session, bfdd does the calculation in bs_echo_timer_handler(). That function has exactly two callers:

the control packet receive path in bfd_packet.c, which bfdd does not execute for offloaded sessions, because the dataplane is processing control packets
bfd_set_echo(), where the call sits behind if (bs->bdc == NULL)

bs->bdc is the dataplane context. It is non-NULL precisely when the session is offloaded. So the guard excludes exactly the sessions that reach this code through the first path being unavailable.

The result is that bs->echo_xmt_TO stays at 0, and _bfd_dplane_session_fill() puts the locally configured bs->timers.desired_min_echo_tx into the min_echo_tx field of DP_ADD_SESSION. The dataplane faithfully transmits at that rate.

Configure the peer to advertise 200ms and the local side to want 50ms, and you get echoes every 50ms: four times faster than the peer said it could receive them.

The information was already there

The frustrating part, and the reason the fix is small, is that bfdd already has the peer’s advertisement. bfd_dplane_session_state_change() receives state->required_echo_rx from the dataplane and stores it into bs->remote_timers.required_min_echo.

It was being recorded and never used. show bfd peer even displays it correctly, which makes this genuinely hard to spot from the CLI: the operator sees the peer’s 200ms reported back accurately while the wire carries 50ms. Only the value sent to the dataplane, and the actual packet cadence, show the problem.

The fix

A bfd_dplane_echo_negotiate() that takes the larger of the two values, called from the state-change handler and from bfd_set_echo(), so that enabling echo on an already-established session negotiates too rather than waiting for the next state change:

void bfd_dplane_echo_negotiate(struct bfd_session *bs)
{
        uint32_t negotiated;

        if (bs->bdc == NULL || !CHECK_FLAG(bs->flags, BFD_SESS_FLAG_ECHO))
                return;

        negotiated = bs->remote_timers.required_min_echo >
                             bs->timers.desired_min_echo_tx
                     ? bs->remote_timers.required_min_echo
                     : bs->timers.desired_min_echo_tx;

        if (bs->echo_xmt_TO == negotiated)
                return;

        bs->echo_xmt_TO = negotiated;
        bfd_dplane_update_session(bs);
}

The whole fix is that comparison, and the reason it took a bug report rather than a glance is that both of its inputs were already sitting in the session struct. Nothing had to be plumbed anywhere. The two guards are what keep it inert: it does nothing for a session that is not offloaded or has echo disabled, and it does not push an update to the dataplane unless the value actually changed.

min_echo_tx then carries the negotiated interval, and the field’s comment says so explicitly, since the next person to read it deserves to know it is not the configured value.

Two things I flagged for reviewers, because both look like behaviour changes and neither is:

The wire format does not change. min_echo_tx now carries a negotiated value instead of a configured one, but no dataplane could have implemented echo correctly before this, because it was never told what the peer required. There was no working behaviour to break.

The fix is inert against a dataplane that does not populate required_echo_rx in BFD_STATE_CHANGE. In that case remote_timers.required_min_echo stays 0, the negotiation is a no-op, and the configured value is used exactly as before. Which also means testing bfdd on its own shows no change at all, and I said so explicitly rather than letting a reviewer discover it.

Verified against a dataplane that does report the field: observed cadence moved from 50ms to 200ms against a peer advertising 200ms, with no other configuration change. Merged as #22805.

What I took from it

This bug needed two implementations to be visible. The control plane had the peer’s requirement and did not use it. The dataplane used what it was told and had no way to know better. Each side was locally reasonable.

It also only shows up on the wire. Every piece of CLI output was correct throughout. If I had been checking show bfd peer rather than counting packet intervals, I would have concluded it worked.

Line-card BFD for plain Linux, part 8: IPv6, and the bake-off argument in miniature

awasey8905@gmail.com (Abdul Wasey) — Sat, 18 Jul 2026 00:00:00 GMT

Part 7 got the engine to sixty-four sessions. IPv6 was the next item on the list, and on the face of it the least interesting one: parse a different header, branch where the protocols differ, done.

There was a question buried in it though, and it had been nagging since the bake-off.

The entire argument for this project rests on one claim: that moving transmission into the kernel is what buys the resilience, rather than any of the other things that changed along the way. That claim came from comparing different programs on different runs. A skeptic could reasonably say the XDP build was better for some other reason, or that the load was not identical, or that six weeks of incidental fixes did the work.

Adding a second address family creates the chance to answer that properly. If v6 transmits from userspace while v4 rides the kernel path, on the same engine, on the same box, at the same instant, under the same load, then everything is controlled except the one variable in question.

So the milestone had two jobs: make the engine dual-stack, and settle whether the kernel reply is actually load-bearing.

The structural move came first, and alone

The session key widened from two __be32 fields to two 16-byte addresses, with v4 stored v4-mapped (::ffff:a.b.c.d), so both families share one hash map, one slot-socket pool, and the same fast path with no possibility of key collision.

That step contains no v6 code at all. It exists to prove the layout change against the live v4 session before anything interesting is built on it. A 10,000-packet regression run, RX-clocked lockstep intact through the new key.

Each of the five landing steps carried a v4 regression check, so the shared code never regressed the working path.

The v6 parse path

A branch on ethertype: fixed 40-byte ipv6hdr, GTSM as hop_limit 255, and a deliberate refusal to walk extension headers. UDP hidden behind an extension chain never reaches a session, while non-UDP first headers are XDP_PASSed to the stack so ICMPv6 neighbour discovery survives an attached program.

The kernel reply is family-branched where the protocols actually differ. v4 keeps udp->check = 0 and the IP-checksum trim recompute. v6 has no IP checksum but a mandatory UDP one, so the echo recomputes it over the swapped pseudo-header as a 34-word fold.

The ordering constraint is the part worth writing down: the fold must complete before bpf_xdp_adjust_tail, which invalidates every packet pointer, and must never read past the 24 BFD bytes that survive the trim.

if (ip6) {
        __u32 csum = 0;
        __u16 *w = (__u16 *)&ip6->saddr;
        for (int i = 0; i < 16; i++)   /* saddr + daddr */
                csum += w[i];
        csum += udp->len;              /* pseudo length */
        csum += bpf_htons(IPPROTO_UDP);
        w = (__u16 *)udp;              /* UDP hdr, check == 0 */
        for (int i = 0; i < 4; i++)
                csum += w[i];
        w = (__u16 *)bfd;              /* 24-byte payload */
        for (int i = 0; i < 12; i++)
                csum += w[i];
        csum = (csum & 0xffff) + (csum >> 16);
        csum = (csum & 0xffff) + (csum >> 16);
        __u16 c = ~csum & 0xffff;
        udp->check = c ? c : 0xffff;   /* RFC 768: 0 -> 0xffff */
}

if (excess > 0 && bpf_xdp_adjust_tail(ctx, -excess))
        return XDP_DROP;

Thirty-four words, fixed count, every pointer bounds-proven above so the verifier accepts it. The correctness of the whole block lives in the order of those two statements: fold first, trim second. Reverse them and you are folding over freed pointers, which the verifier will not catch because the pointers were valid when it checked them.

The oversized-frame case, 16 trailing bytes echoed back as a valid 24-byte reply with a recomputed checksum and a patched payload_len, was verified against injected traffic rather than just reasoned about.

The experiment

Here is the answer to the question this milestone opened with.

For a window during the landing, the engine was genuinely dual-stack in parsing but only v4 had the kernel reply. That window is the experiment. Both families ran the L3+L4 ladder concurrently at 3x300ms: v4 on kernel RX-clocked TX, v6 transmitting from userspace at RFC pacing.

v4: 0 flaps.
v6: 19 flaps, TX gaps to 1900ms.

The RT throttle’s ~950ms per second starvation exceeds the 900ms detect budget once per cycle, each flap recovering autonomously in about 3ms.

Same run. Same instant. Same engine. Same host, same load, same code except for one thing: which side of the kernel boundary the transmit clock lives on.

With the kernel reply enabled, the rerun of the same ladder: v6 0 flaps, TX max 300.5ms, indistinguishable from v4.

That settles it. The resilience is not an accident of the rewrite, or of six weeks of incidental fixes, or of a favourable run. It is the kernel reply, and here is the same engine failing and then not failing with only that changed.

You rarely get a controlled experiment that clean outside a textbook, and I got it by accident, from the ordinary business of landing a feature in stages.

Scale, and closing an upstream loop

The v6 spoof harness repeated the m5 validation from a third host: wrong your_disc and hop_limit 64 both XDP_DROPped and counted, correct credentials passed.

The mixed-family scale run closed the milestone: 32 v4 plus 32 v6 at the 64-session cap, 3x10ms timers, through the same L3+L4 ladder. Zero flaps in either family, zero wire transitions, and per-slot maximum TX gaps sitting in a single 13 to 14.5ms band with the two families statistically indistinguishable.

The v4-only baseline had shown correlated stall flaps at this session count. The dual-stack engine beats its own earlier result.

The scale work also found the 8KB output buffer bug in bfdd that stranded sessions silently, and the validation session that followed found two more in the same file. All three were invisible without a real dataplane at scale, which is the recurring theme of this project’s relationship with FRR.

Echo mode, where the feature I most wanted to put in the kernel turned out to be the one XDP fundamentally cannot do.

A read of zero bytes that looked exactly like a closed socket

awasey8905@gmail.com (Abdul Wasey) — Sat, 18 Jul 2026 00:00:00 GMT

This is my favourite of the FRR bugs I found, because the failure alternates. Run show bfd peers counters and it works. Run it again and it tears down the dataplane connection partway through. Run it again and it works. Every time, period two.

Anything with period two is a state machine with somewhere to hide, and in this case the hiding place was the input buffer.

Two defects, and neither is enough on its own

First: consumed space is never reclaimed on the synchronous path.

bfd_dplane_expect() calls stream_pulldown() only when reads >= 3 within a single call. A counters query consumes exactly one message per call, the reply whose id matches, and then breaks. So the pulldown never fires.

The event-driven reader, bfd_dplane_read(), does pull down, but only when a read event fires. Between two CLI invocations the dataplane sends nothing unprompted, so no read event happens and nothing reclaims the space.

After a successful sweep of N sessions, endp is parked at N times the reply size with the buffer logically empty. The bytes are consumed. The cursor does not know that.

Second: a full buffer is misdiagnosed as the peer closing.

When the buffer fills mid-message, the parse loop’s goto read_again jumps back past the buffer-full guard at the top of the function. stream_read_try() is then called with size 0. Its guard is STREAM_WRITEABLE < size, which evaluates to 0 < 0, which is false, so it proceeds and issues the equivalent of read(fd, buf, 0).

A zero-length read returns 0 by definition. And the rv == 0 path means “the socket closed”, so bfdd calls bfd_dplane_ctx_free() and tears down a perfectly healthy connection, with the tail of a valid reply still unread in the kernel buffer. That unread data is why the wire shows an RST rather than a FIN.

The comment above the entry guard shows the bogus-close hazard was already known to whoever wrote it. The mid-loop re-read path just was not covered by it.

The arithmetic is exact, which is how I knew I had it

64 sessions, 80-byte counters replies, an 8192-byte buffer:

A successful sweep leaves 64 x 80 = 5120 bytes of dead space.
The next sweep has 8192 - 5120 = 3072 bytes of headroom, which is 38 full replies.
Reply 39 can only partially append. Buffer exactly full. Zero-length read. Forged EOF. Teardown.
The reconnect calls stream_reset(), which clears the residue, so the cycle starts over. Period two.

64 + 39 = 103 replies per connection, deterministically, every time. When a prediction like that lands on the nose, you are done guessing.

Why this is worse than a crash

The failing invocation still prints a full peer list. Every session past the failure point shows stale or zero counters. The operator gets wrong data presented as good data, with no error, unless debug bfd distributed happens to be on.

Each occurrence also drops the connection and re-triggers the entire session registration burst on reconnect. Monitoring that polls this command hits it persistently.

The fix

Pull the input buffer down at function entry, so consumed space is reclaimed regardless of which caller you are. And never issue a zero-length read: reclaim first, and if the buffer is still full of an incomplete message, treat that as the protocol error it actually is, namely a single message larger than the entire buffer.

stream_pulldown(bdc->inbuf);

/*
 * Never issue a zero-length read: `read()` returns 0 and would be
 * misdiagnosed below as the peer closing the connection. If there
 * is no headroom, reclaim consumed space first; a buffer that is
 * still full after that holds a message larger than the buffer,
 * which is a protocol violation.
 */
if (STREAM_WRITEABLE(bdc->inbuf) == 0) {
        stream_pulldown(bdc->inbuf);
        if (STREAM_WRITEABLE(bdc->inbuf) == 0) {
                zlog_err("%s: input buffer full with incomplete message", __func__);
                bfd_dplane_ctx_free(bdc);
                return -1;
        }
}

The unconditional stream_pulldown() at the top is the actual fix: it runs for every caller, on every call, so no path can accumulate dead space any more. The block below it is the belt to that braces, and it turns the impossible case into a loud one instead of a zero-length read.

A review comment pushed that second half further. My first version returned -1 on the buffer-full branch without freeing the context, which leaves a wedged connection behind. It now calls bfd_dplane_ctx_free() first, matching the socket-closed and socket-error paths: a peer claiming a message larger than the whole 8KB buffer is a protocol violation the parser cannot recover from, so tearing the connection down and reconnecting is the right answer.

While there I noticed the pre-existing bad-version check a few lines up has exactly the same gap, return -1 with no free. I left it out rather than widen a bug fix into unrelated cleanup, and said so on the PR with an offer to send a follow-up. Scope discipline is easier to defend than to practise.

Validation:

Responder harness, 64 sessions: unpatched delivers exactly 103 replies per connection before the RST, repeating every two invocations. Patched delivers 384 out of 384 across six consecutive invocations on one uninterrupted connection, zero resets.
Against a real dataplane with 64 established sessions: unpatched alternates strictly, 39 of 64 across ten trials, each failure logging bfd_dplane_expect: socket closed. Patched completes ten consecutive sweeps with zero disconnects.
Built together with the shutdown drain from #22692 to confirm they coexist.

The review asked for a test, and the test needed a dataplane

Donald Sharp’s response to the fix was, reasonably, “please write a topotest that shows this problem is fixed.”

That is harder than it sounds. The bug lives in bfd_dplane_expect(), which only runs when bfdd is in distributed mode connected to an external dataplane over the bffdp protocol. A standalone bfdd never reaches the code at all, so there was nothing in the FRR test tree that could trigger it. I had a Python responder that worked, but wiring a Python helper into a topotest is a one-off.

He pointed me at fpm_listener in fpm_testing_topo1, which solves the identical shape of problem for zebra’s FPM: a mock receiver on the far end of an offload socket that dumps its state on SIGUSR1 so a test can inspect it.

So the PR grew a second and third commit:

bfdd/bfd_dplane_listener, a minimal in-tree stand-in for a dataplane. It accepts the bffdp connection, tracks the sessions bfdd registers and reports them up, answers counter requests, and dumps what it received on SIGUSR1. It runs no BFD state machine and sends no BFD packets. A session is simply declared up once registered, which is enough for bfdd to treat it as established.

bfd_dplane_counters_topo1, the test itself: eight peers, twenty counter sweeps, then assertions that every request was answered, the connection is still up, and bfdd never reconnected. Against the unpatched daemon it fails with 103 of 160 counter requests were answered, which is exactly where the buffer boundary sits.

Getting that merged-shaped took a round of unglamorous work that I would not have predicted:

The listener needs XREF_SETUP(), because anything in sbin_PROGRAMS gets an xrelfo pass. Missing it broke the build.
RPM needs an %exclude in redhat/frr.spec.in; Debian needs an entry in debian/frr-test-tools.install. The two distros handle fpm_listener differently, so both were needed and neither was optional.
OpenBSD builds with clang and -Werror, which flagged sigterm_handler under -Wmissing-noreturn. Fixed with FRR_NORETURN, the same way fpm_listener declares its own.

Open as #22694.

What I took from it

read(fd, buf, 0) returning 0 is correct. Treating a 0 return as EOF is correct. Both of those are true and the combination is a bug, which is the shape a lot of the interesting ones have.

The guard that would have caught it existed. It was at the top of the function, and the failing path jumped over it with a goto. A precondition you can skip is a precondition you do not have.

The larger lesson was about what a fix costs. The actual repair is a handful of lines. Making it provable meant contributing test infrastructure that did not exist, and then chasing it across three packaging systems and a compiler that nobody in the conversation was using. The reviewer was right to ask. Untestable code paths stay broken, and the dataplane paths in bfdd had been untestable since they were written.

Deleting sessions into a buffer nobody would ever flush

awasey8905@gmail.com (Abdul Wasey) — Fri, 17 Jul 2026 00:00:00 GMT

This one is the direct sequel to the 8KB buffer bug, and it exists partly because that fix landed.

When bfdd shuts down it tears down every session. For dataplane sessions, bfd_session_free() calls bfd_dplane_delete_session(), which enqueues one DP_DELETE_SESSION message per session into the dataplane client’s output buffer.

Nothing ever writes those bytes to the socket.

Why they never leave

Enqueuing schedules a write event. On the shutdown path, the event loop does not run again, so that event never fires. bfd_dplane_ctx_free() sees bglobal.bg_shutdown set, goes straight to free_resources, closes the socket, and frees bdc->outbuf with the messages sitting in it.

What the dataplane actually receives depends on which version you are running, and this is the uncomfortable part:

On 10.5.1, before #22645, enqueue only schedules the write. Zero deletes reach the dataplane. The whole burst is silently lost.
On current master, with #22645, the synchronous flush I added fires once when the buffer fills mid-burst. So exactly one buffer’s worth is delivered, and the tail is freed unsent. At 140 bytes per message that is floor(8192/140) = 58 delivered, and everything past 58 dropped.

I would argue the second case is worse. Losing everything is at least unambiguous: the dataplane sees no teardown at all and its existing reconnect logic handles it. Losing an arbitrary subset means the dataplane observes a partial teardown, in hash order, with no signal that the rest were also meant to go. It looks like a successful operation.

So my own fix converted a clean failure into a misleading one. It did not cause this bug, which was there the whole time, but it changed its shape, and I would rather say that plainly than let someone else find it.

#22645 also cannot cover this case. After the last delete is enqueued there is no further enqueue to trigger a flush, and no event-loop iteration left to service the scheduled write.

Reproducing without a dataplane

Same approach as before, and the reason both reports were actionable: you do not need an offload engine. A TCP socket that accepts a connection, parses the bfddp message stream, and counts messages per type is enough. Configure 64 sessions, let the registration burst land, stop bfdd cleanly, and count the deletes.

The sessions never come up against a sink that does not answer, and that does not matter. The shutdown delete walk runs for every configured dataplane session regardless of state.

Unpatched, the sink reports 58 deletes and 8120 bytes: exactly one full buffer.

The fix

A synchronous drain in bfd_dplane_ctx_free(), before socket_close(), run only when bg_shutdown is set:

/* Nothing to write or nowhere to write it. */
if (bdc->sock == -1 || bdc->connecting)
        return;

while ((remaining = STREAM_READABLE(bdc->outbuf)) > 0) {
        rv = write(bdc->sock, stream_pnt(bdc->outbuf), remaining);
        if (rv <= 0)
                break;
        bdc->out_bytes += (uint64_t)rv;
        stream_forward_getp(bdc->outbuf, (size_t)rv);
}

remaining = STREAM_READABLE(bdc->outbuf);
if (remaining)
        zlog_warn("%s: %zu bytes of data plane messages lost on shutdown", __func__,
                  remaining);

It is deliberately best-effort. A single pass writing whatever the socket will take, giving up if it would block or errors, and saying so in the log if anything is left. The event loop is already gone by this point and the socket belongs to the frrevent machinery, so re-arming a write event is not available and blocking on shutdown would be worse than the problem. Unsent deletions are timed out by the dataplane instead, which is a mechanism that already has to exist for the case where bfdd is killed rather than stopped.

The warning matters as much as the loop. The original bug was not that messages were lost, it was that they were lost silently.

I deliberately did not reuse bfd_dplane_flush(). Its error path frees the very context this code is being called from, which would be a use-after-free.

Results, with the sink and with a real engine:

64 sessions, clean stop: unpatched 58/64, patched 64/64, byte count exact at 140 bytes per message, no loss warning.
128 sessions: patched delivers 128/128 and 17920 bytes, two full buffer-loads plus a tail, which confirms the drain loops rather than flushing once.
Against a live dataplane holding 64 established sessions, 32 IPv4 and 32 IPv6: unpatched, a clean restart stranded 6 sessions as orphans. Patched, all 64 deletes arrive, zero orphans across the restart, and all 64 re-register with Output full events: 0. A second restart in the same run behaves identically, which shows the registration-side behaviour from #22645 is undisturbed.

Open as #22692.

What I took from it

Shutdown paths get written as if the work is already done. The event loop is usually still there to finish anything outstanding, so “enqueue and return” reads as complete. It is complete right up until the moment you are enqueueing because you are shutting the loop down.

The other thing: fixing a bug is a good time to look for the one next to it. The buffer flush I added made this failure visible by changing 0 into 58, and 58 is a much stranger number to see in a log than 0.

How 8KB of buffer silently lost BFD sessions in FRR

awasey8905@gmail.com (Abdul Wasey) — Wed, 15 Jul 2026 00:00:00 GMT

While building xdp-bfd, an offload engine that runs the BFD fast path in XDP, I needed FRR’s bfdd to hand session lifecycle over to my dataplane. This turned up during the 64-session scale run. Below roughly twenty peers it worked. Above it, sessions started disappearing, and nothing in the logs said why.

What the dataplane protocol does at connect time

When bfdd connects to a distributed-BFD dataplane, it walks every configured session and enqueues one registration message per session into the dataplane client’s output buffer. That buffer is 8KB, and the whole walk happens in a single event-loop pass, before the write event ever gets a chance to run.

At roughly 140 bytes per message, 8KB holds about 58 registrations. With 64 sessions configured, the last six never made it.

Why the failure was invisible

A failed enqueue was not the end of it. For each session past the limit, bfdd had already disabled the software implementation, on the assumption that the dataplane was taking over. It then reset bs->bdc to NULL when registration failed.

That combination is the actual bug. The session was:

never delivered to the dataplane, so the dataplane did not know it existed
no longer handled in software, because that path had been switched off
left with a null dataplane context, so every subsequent operation on it became a silent no-op

The session still appeared in show bfd peers. It simply never ran anywhere. No warning was logged, because from the enqueue path’s point of view a full buffer is an ordinary condition.

Reproducing it without a dataplane

The useful part of the investigation was making it reproducible for someone who does not have an offload engine sitting around. A dataplane does not have to do anything for this bug to show up: it only has to accept a TCP connection and count bytes.

A socket that accepts the connection and reads without ever replying is enough. Configure N sessions, connect, and count the bytes that arrive. The count stops at exactly one buffer’s worth, every time, and the cutoff is deterministic.

That mattered more than the fix. A bug report with a reproducer that needs a specific piece of unreleased software is a bug report nobody can act on.

The fix

Two commits, each useful on its own.

The first makes the enqueue path try to flush before giving up. During the registration burst the socket is almost always writable, and the packet capture showed zero TCP backpressure, so the common case now simply succeeds:

if (buflen > STREAM_WRITEABLE(bdc->outbuf) &&
    !(bdc->client && bdc->connecting)) {
        /*
         * On socket failure or close the context is freed by
         * the flush. The buffer is non empty here, so a zero
         * return means exactly that: don't touch `bdc` again.
         */
        if (bfd_dplane_flush(bdc) == 0)
                return -1;
}

There is a subtlety in the error path. A zero return from the flush with a non-empty buffer means the context was freed on a socket error, so it must not be touched again. Clients that are still connecting are skipped entirely, because flushing would spin on EAGAIN.

A review bot flagged this as possibly leaving queued messages with no scheduled writer, and a maintainer asked whether it needed fixing. It does not, and being able to say exactly why is the useful part: bfd_dplane_flush() loops on STREAM_READABLE() and treats EAGAIN, EWOULDBLOCK, and EINTR as retry, so it cannot return with data still buffered. Its only two exits are a fully drained buffer, at which point cancelling the write event is correct because there is nothing left to write, or freeing the context on socket error. After a successful flush the whole buffer is writeable, so the space check that follows can only fail for a single message larger than 8KB. The largest bfddp message I measured is 203 bytes.

The second commit stops a failed registration from stranding the session. On failure it logs a warning and returns the session to the software implementation, which mirrors the recovery that already existed when a dataplane detaches. Software BFD is worse than offloaded BFD, but it is dramatically better than nothing at all, which is what the session had before.

Result

With 128 sessions configured and a byte-counting sink, the unpatched build delivers exactly 58 registrations, 8120 bytes, and silently loses the other 70. The patched build delivers all 128, with the byte count exact and Output full events at zero.

End to end against the engine, the reconnect burst that previously registered 40 of 64 sessions now registers all 64.

The fix is merged to FRR master as #22645. It also changed the shape of a second bug on the shutdown path, which I wrote up separately. Packaged releases up to 10.5.1 still carry the bug, so if you are running distributed BFD with more than about twenty peers on a released build, add peers through vtysh after the dataplane connects rather than putting them in frr.conf.

What I took from it

The bug was not in the buffer size. 8KB is a reasonable buffer. The bug was that a resource limit was being treated as an unremarkable condition, in a code path where crossing it meant a session belonged to nobody.

Failure paths that quietly reduce functionality are worse than failure paths that crash, because a crash gets reported. This one produced a router that believed it was monitoring a link it was not monitoring, which is precisely the thing BFD exists to prevent.

Line-card BFD for plain Linux, part 7: one session to sixty-four

awasey8905@gmail.com (Abdul Wasey) — Mon, 13 Jul 2026 00:00:00 GMT

Part 6 took session continuity and validation off the list from part 5. The next item on it is the one I had been quietly avoiding, because it invalidates every number published so far.

Everything measured up to this point, the whole bake-off, the final matrix, the FRR integration, has been one session. The maps and the daemon were sized for 64 from the hardening work onward, and I had written “multi-session capable” in the docs on the strength of that sizing. Sizing a table for 64 and testing one entry in it are not the same claim, and the distance between them turned out to be five bugs wide.

There is also a reason to expect trouble specifically here. The whole design rests on the peer’s packet being the clock: a session’s timing comes from its own inbound stream, processed in softirq. With one session that is a clean story. With sixty-four it is sixty-four streams sharing one receive path, and the question of whether they stay independent is exactly the sort of thing that is obvious in hindsight and untested in fact.

Four things that only worked because there was one of them

The receive path matched sessions on the peer’s IP address alone, and never recovered the local destination address at all. With one session that is sufficient and looks like a design. With two sessions to the same peer from different local addresses it is a coin flip. Matching now happens on the address pair, using IP_PKTINFO with recvmsg to recover the destination, and it only matters for bootstrap packets where your_disc is still zero, since established sessions demux on the discriminator as before.

Source ports were next. RFC 5881 says they should be unique per session, which I had ignored because with one session it is a distinction without a difference. Each slot now transmits from 49152 + slot, which meant moving userspace transmission to per-slot sockets, opened lazily and kept for the life of the process so that a reused slot reuses its socket and teardown needs no file descriptor bookkeeping.

The interesting one is map ownership, and it is the same class of mistake as the timestamp race in part 4: two things touching one value from two contexts, correct in every test until they are not.

The kernel had been clearing cfg->poll in place when the peer’s Final arrived, while userspace pushed the whole tx_cfg struct whenever it updated its mirror. A push landing at the wrong moment could resurrect a Poll sequence that had already finished. What makes it worth writing about is that the repair is not a lock:

/* Poll termination (RFC 5880 s6.8.4): peer answered our P with
 * F. tx_cfg is userspace-owned, so ack via kernel-owned
 * final_seq instead of clearing cfg->poll in place (a racing
 * userspace mirror push could resurrect the finished poll). */
if (cfg && cfg->poll && (bfd->flags & BFD_F_FINAL))
        st->final_seq = cfg->poll_seq;

tx_cfg became strictly userspace-owned. The kernel acknowledges the Final by writing final_seq into session_state, a map it already owns and userspace already polls, and the transmit path from part 4 sets the Poll bit only while final_seq and poll_seq disagree. That is the st->final_seq != cfg->poll_seq condition in the payload rebuild, which I glossed over at the time.

Every map value ended up with exactly one writer. The race is not guarded against, it is unrepresentable, which is a much better place to leave a concurrency bug than behind a lock you have to remember to take.

The last one is smaller but has the same shape. The receive path sets a session’s alive flag and the sweep clears it, and those run on different CPUs, so plain byte writes could emit the same liveness event twice. It became a __u32, because BPF atomics are 32 or 64-bit only, with both transition sites using __sync_val_compare_and_swap and emitting only when the swap wins.

Two sessions, then sixteen, then sixty-four

Two concurrent sessions established, slot ports visible bound and on the wire. Poll sequences driven through the new final_seq path by changing the transmit interval to 20ms and back while Up: both negotiated, zero down events, and zero residual Poll bits on the wire afterwards.

The test I cared about most was kill isolation, because one session dying must not disturb its neighbour and until then that had been an assumption rather than a measurement. FRR was stopped dead on one neighbour, with no farewell packet. The sweep declared that session down after 31.2ms of silence, inside the same 30 to 33ms envelope as always, while the other session’s uptime ran on uninterrupted.

Under two-session L3 stress, the same condition that flaps stock bfdd 44 times, both sessions held with zero flaps: p50 8.75 / p99 10.02 / max 14.7ms on one and p50 8.74 / p99 10.02 / max 12.6ms on the other. Two RX-clocked streams do not degrade each other, which is not obvious in advance, since they share a receive path.

Sixteen sessions next, eight per peer via secondary addresses: 0 flaps under L3, and a mass-kill of 8 detected at 30.5 to 32.8ms with no batch drift.

Sixty-four, and five bugs

The full-capacity run found five things. Four were mine, and none of them could have appeared earlier.

The slot ports I had just introduced collided with bfdd’s own port allocator, which is the sort of bug that only exists when two programs are running together at scale; moving the slot range up to 65472 and above resolved it. AdminDown turned out to have no exit, so a session could enter it and stay there. Sourcing from INADDR_ANY broke the peer address demux I had also just introduced, so per-slot sockets now bind their local address explicitly. And the mark-and-sweep reconciliation added for graceful restart in the previous milestone was unmarking incorrectly, tearing down sessions on a reconnect where the discriminators had not changed at all.

The fifth was not in my code. bfdd’s dataplane client silently and permanently strands sessions when the registration burst overflows its 8KB output buffer, which became FRRouting#22638, fixed in #22645. It is the first of three bugs this scale work eventually found in that one file.

The pattern that started with the UNIX socket bug held: every time the engine pushed FRR’s dataplane path somewhere it had not been pushed before, something fell out.

The design’s own boundary

The 32-session mass-kill was detected in 30.1 to 32.9ms with no drift, the same envelope as one session. Good.

But under L3 stress there was one correlated flap of 19 sessions at once, from a single RX-softirq stall.

That is worth stating plainly rather than burying, because it is the honest limit of the design at this scale. RX-clocked TX shares fate with softirq latency, as part 5 already said for one session. At 64 sessions that fate is correlated: a single stall in the receive path delays every session’s reply at once, so they do not fail independently. Recovery was autonomous and took about 50ms, but 19 sessions flapping together is a different operational event from one session flapping.

It is the quantified false-flap boundary of the architecture, and finding it was the point of the milestone. Part 1 argued that the maximum matters more than the percentile; this is the same argument turned on my own engine, and the per-session maximum is what exposed it.

What I took from it

“The maps are sized for 64” is not a claim about behaviour, and I had let it stand in the documentation as though it were. Four of the five bugs here were mine, all latent for weeks, all invisible to every test I had run, because a single session cannot exercise demux, cannot collide with a port allocator, and cannot race another session’s map writes.

The engine did not get faster in this milestone. It got true.

IPv6, which turned out to contain the cleanest controlled experiment in the project, and which found the second and third bugs in the same bfdd file as the first.

Line-card BFD for plain Linux, part 6: hardening, and a review that found real edges

awasey8905@gmail.com (Abdul Wasey) — Sun, 12 Jul 2026 00:00:00 GMT

Part 5 ended with a working benchmark and an embarrassing list: no session continuity, no real validation, one session tested out of sixty-four, and three RFC 5880 features missing outright.

This part takes the first two off that list. It is the unglamorous middle of the project, where nothing gets faster and the interesting question is no longer “does it work” but “does it work when someone is trying to break it, or when the control plane restarts underneath it”.

Nothing here changes the steady-state path, so the numbers from part 5 stand unchanged throughout.

Surviving a control-plane restart

The most visible gap was that restarting bfdd dropped every session. The dataplane tore them down when the control plane disconnected and bfdd re-added them on reconnect, which is correct in the sense that nothing lies about state, and useless in the sense that a routine daemon restart takes your routes with it.

The fix is a hold mode, --dp-hold, that keeps wire sessions alive across the gap. On disconnect the sessions are orphaned rather than deleted. When bfdd comes back and re-adds them, they are adopted by address pair with discriminator continuity, and a mark-and-sweep pass reconciles anything that did not come back. I took the shape of this from Rafael Zalamena’s guidance on the FRR dev list, since the protocol’s intent matters more than my reading of it. Two back-to-back FRR restarts now produce zero peer-visible events.

It is not the default, and should not be. Drop-and-recreate is what the protocol expects unless an operator has decided otherwise, so continuity is something you ask for.

The same milestone added self-initiated Poll sequences on parameter change, per RFC 5880 s6.8.3, along with transitional transmission during the change. Before that, altering a timer mid-session could flap it against either side, which is a poor reward for reconfiguring something.

Refusing traffic properly

The validation story until this point was thin. It now enforces GTSM at TTL 255, demux validation per s6.8.6, and session creation gated on control-plane config so nothing can conjure a session by sending packets at the box. Everything that fails lands on XDP_DROP, so a rejected packet never reaches userspace at all rather than being filtered somewhere later.

The demux check is the one that does the real work against spoofing:

__u8 rstate = BFD_STATE(bfd);
if (cfg && cfg->my_disc) {
        __u32 ydisc = bpf_ntohl(bfd->your_disc);
        if (ydisc != cfg->my_disc &&
            !(ydisc == 0 && rstate <= 1)) {
                count(3);
                return XDP_DROP;
        }
}

A packet has to name our discriminator back at us. The exception is the reason it is not simply an equality test: a peer that has lost its state and is restarting legitimately sends your_disc of 0, so that is allowed, but only while it also reports Down or AdminDown. Without the state condition, zero becomes a wildcard anyone can send.

What makes this a validation fix rather than a parsing detail is what happens on success. A packet that gets past here refreshes the session’s liveness and gets echoed back by the transmit path. Anything that can forge its way through is therefore able to keep a dead session up indefinitely, which is precisely the failure BFD exists to prevent. All of it was tested by injecting from a third host.

One measurement note cost me an hour and is worth passing on. XDP_DROP consumes the frame before any capture hook sees it, so tcpdump shows you nothing at all. The only trustworthy signal that a drop happened is the map counters read through bpftool. A test that “sees no packets” is otherwise indistinguishable from a test that sent none, and I briefly believed the wrong one.

The review pass

An external code review of the three source files, taken seriously enough to answer each point on the wire.

The structural item was the shared ABI header. The BPF map value structs had been living as three hand-synchronised copies, one per file that touched them, which works right up until one side gains a field the others do not and the two halves start reading different offsets out of the same bytes. They became a single included definition.

Two of the flagged edges turned out to be real, and both are the same kind of mistake.

The first is IP options. Options push the UDP header to a variable offset, and the parser was reading the GTSM and discriminator checks from fixed positions. A packet carrying options therefore walked straight past both guards from the previous section, the two that exist specifically to stop that packet, and arrived in userspace unvalidated.

/* IP options (ihl != 5) on a UDP packet: a single-hop BFD
 * control packet never carries them. Passing would skip the
 * GTSM/your_disc checks below (UDP header sits at a variable
 * offset with options) and leak the packet to the userspace
 * socket unvalidated - the same bypass class as an XDP_PASS
 * reject. Drop it. */
if (iph->ihl != 5) {
        count(3);
        return XDP_DROP;
}
udp = (void *)(iph + 1);

The fix is three lines, and the last line is the reason they are needed: udp = (void *)(iph + 1) assumes a 20-byte header. That assumption is fine, and it is load-bearing, and nothing had been enforcing it. A single-hop BFD control packet never legitimately carries options, so refusing them outright is cheaper and safer than learning to parse them.

The second is bfddp framing. On a framing error the reader reset its buffer and carried on reading the same stream, which means it can resync onto arbitrary mid-stream bytes and start interpreting them as messages. It now drops the connection cleanly, and because --dp-hold already exists that drop turns into a hitless reconnect rather than an outage.

Two RFC-correctness fixes rode along: the s6.8.7 jitter cap at 90% when detect_mult is 1, and trimming an over-length echoed frame back to 24 BFD bytes with a recomputed checksum.

Every change was checked the same way as everything else here, with an injection harness and a capture. None touched the steady-state path, so the resilience numbers from part 5 stand unchanged.

The rejected suggestions, and the reasoning for rejecting them, are in the repo under docs/refactor-abi. Writing down why you did not take a suggestion is worth as much as the changes you did take, and it is the part people skip.

What I took from it

The two review comments I was most inclined to wave away were the two that were real. Both were of the form “what if the header is not where you think it is”, which is exactly the class of assumption that a fast path is built on and a parser has to defend.

Sixty-four sessions, which is the next item on that list and the one that found the most bugs.

Two bytes of padding, and a transport that never worked

awasey8905@gmail.com (Abdul Wasey) — Thu, 09 Jul 2026 00:00:00 GMT

I was pointing xdp-bfd at FRR’s distributed BFD dataplane socket and wanted the UNIX transport rather than TCP, since both ends were on the same host. It never connected. Every three seconds:

bfdd[13438]: bfd_dplane_client_connect: data plane connection failed: Invalid argument

The TCP transports worked perfectly against the same engine. Only unixc: failed.

The address was fine, the length was not

strace gave it away immediately:

connect(16, {sa_family=AF_UNIX, sun_path="/tmp/frr-repro.sock"}, 112) = -1 EINVAL

The path is right. The 112 is not. On x86_64 with glibc:

sizeof(struct sockaddr_un)  = 110
sizeof(struct sockaddr_in6) = 28
sizeof(union addr)          = 112

bfd_dplane_client_init() copies the caller’s address into a union of sockaddr, sockaddr_in, sockaddr_in6, and sockaddr_un, then records the length. It records the wrong one:

if (salen <= sizeof(bdc->addr)) {
        memcpy(&bdc->addr, sa, salen);
        bdc->addrlen = sizeof(bdc->addr);   /* should be salen */

It stores the size of the whole union instead of the size of the address it was given. The union is 112 bytes because sockaddr_in6 forces 8-byte alignment, padding it past the 110 that sockaddr_un actually occupies.

Why only AF_UNIX noticed

This is the part worth keeping. The kernel’s AF_UNIX path validates the address length strictly. unix_validate_addr() in net/unix/af_unix.c rejects any addrlen greater than sizeof(struct sockaddr_un) outright, so 112 is refused with EINVAL before anything else happens.

Inet sockets do not. Pass an oversized addrlen to an AF_INET connect and it is tolerated, because the kernel only needs the leading bytes it understands. So the exact same bug, on the exact same line, is invisible on the TCP transports and fatal on the UNIX one.

That asymmetry is why this survived so long. git log -S addrlen -- bfdd/dplane.c shows the logic unchanged since client mode was introduced in 6655b43d51. As far as I can tell, unixc: never worked on Linux, in any release.

The fix

memcpy(&bdc->addr, sa, salen);
bdc->addrlen = salen;

One line. The truncating branch keeps sizeof(bdc->addr), which is correct there, because that path really did fill the whole union.

I rebuilt bfdd from the 10.5.1 source with only that change and it connected to the same listener that stock bfdd had been refusing for hours. Merged to master as #22621, reviewed by the dataplane author.

What I took from it

I spent longer than I should have looking at permissions, socket paths, and my own listener, because “Invalid argument” on a connect to a socket that plainly exists reads like the address is wrong. It was not the address. It was the number sitting next to it.

The wider lesson is about the shape of the bug rather than the bug. A single sizeof on a union that holds several address families is a length that happens to be right for the largest member and wrong for the rest. It survived because the family that cares about it strictly was the one nobody was testing.

Packaged releases up to 10.5.1 still carry this, so use the TCP transport with them.

Line-card BFD for plain Linux, part 2: the folklore is partly wrong

awasey8905@gmail.com (Abdul Wasey) — Tue, 07 Jul 2026 00:00:00 GMT

Part 1 measured FRR’s bfdd under a stress ladder and found it flapping 44 times in 120 seconds under timer pressure, with a p99 that looked perfectly healthy throughout.

Before writing any kernel code I wanted a second data point on the “userspace BFD cannot hold aggressive timers” claim. So I wrote the dumbest possible BFD daemon: 160 lines, one thread, a recv() with a 2ms timeout as the main loop clock, the RFC 5880 state machine, nothing else.

The plan was to watch it die the same way bfdd did, and then be justified in building the XDP path.

It did not die

Under the exact timer and hrtimer stress that flapped bfdd 44 times in 120 seconds, the naive loop ran clean. Zero flaps, p99 gap 13ms against a 30ms budget.

I ran it twice because I did not believe it.

Why, and it is not luck

It is scheduler mechanics.

A task that wakes 500 times a second, does microseconds of work, and sleeps again accumulates almost no vruntime. So whenever it wakes, CFS considers it the most deserving thing on the runqueue and it preempts the CPU hogs nearly instantly.

bfdd is the opposite shape. It is a heavyweight event loop serving zebra IPC, config machinery, and many timers, waking on exact intervals. The timer-subsystem stress was hitting precisely its wakeup path.

So “userspace” was never the problem as a category. The architecture of the wakeup path was. The folklore needed rewriting: userspace BFD reliability depends on winning a scheduling war, and your event-loop architecture decides how well armed you are.

Which raises the obvious next question. Is there a war no userspace architecture wins?

Yes: SCHED_FIFO

Four real-time priority hogs on four vCPUs, and the naive loop finally broke: 25 flaps in 60 seconds, max TX gap 324ms.

The kernel’s RT throttling reserves about 50ms per second for normal tasks when RT work saturates the CPUs. No amount of CFS-friendliness helps when you only exist for one breath per second.

This matters because it is not an artificial condition. On a loaded router, packet forwarding, softirq storms, and other RT-priority work outranking a BFD daemon is the normal state of affairs, not the exception.

The lesson hiding inside the failure

During the RT runs, the naive daemon’s own log showed zero detect timeouts. From its perspective, the peer’s packets kept arriving on schedule.

They had not. They had been queueing in the socket buffer while the daemon was starved. When it finally got CPU, it drained the backlog and its state machine saw a smooth stream of arrivals.

The peer, watching the actual wire, saw 300ms of silence and correctly declared the session dead.

Process logs lie under exactly the conditions you most need them. This is why every number in this series comes from tcpdump on the hypervisor bridge instead of from inside the guest, and it is a theme that recurs until the end of the project.

The question had sharpened. Not “userspace versus kernel”, but “which transmit architectures survive RT starvation, and at what cost”. I tested five, and the most instructive result was the one that failed while producing the best p99 of any backend.

Line-card BFD for plain Linux, part 3: the TX bake-off

awasey8905@gmail.com (Abdul Wasey) — Tue, 07 Jul 2026 00:00:00 GMT

Part 2 established that userspace BFD survives ordinary load and loses to SCHED_FIFO. So the question became: which transmit architectures survive RT starvation, and at what cost?

Five backends, identical conditions: 4x SCHED_FIFO prio-50 hogs, 60 seconds, the 3x10ms session, wire-truth capture from the hypervisor.

TX backend	flaps	p50	p99	max gap
FRR bfdd	continuous	8.8	287	960ms
naive userspace loop	25	10.0	12.0	324ms
userspace + `chrt -f 90` + pinned core	0	10.0	13.0	15.0ms
SO_TXTIME + etf, one packet in flight	48	10.0	13.2	1517ms
SO_TXTIME + etf, pipelined 5 deep	48	10.0	10.1	994ms
XDP RX-clocked	0	8.75	11.0	12.5ms

The chrt row, and its asterisk

Running the daemon at RT priority 90, above the prio-50 hogs, works completely. Zero flaps, 15ms max.

The asterisk is that you won the priority war because nothing else contested it. On a real router you cannot assume that. Forwarding work, softirq processing, and other components are fighting for the same priorities, and a priority arms-race between your failure detector and your data plane is not a design, it is a standoff.

The etf rows are the most instructive failure of the project

The theory is genuinely attractive. SO_TXTIME lets userspace enqueue a packet with an explicit future launch time, and the etf qdisc releases it at that nanosecond via hrtimer. Wire timing becomes independent of when userspace got scheduled.

And it delivers on exactly that promise. The pipelined variant produced the best p99 of every backend tested, 10.1ms.

It also flapped 48 times. Worse than doing nothing special at all.

Zero packets were dropped by the qdisc in those runs. etf did its job perfectly on every packet it was given. The problem is that a starved daemon gives it nothing.

Timestamping fixes jitter. It cannot manufacture liveness.

Pipelining five packets ahead buys 50ms of starvation tolerance. The failure signature in the gap data is unmistakable: max gaps clustering at 915 to 995ms, which is the RT throttle’s breathing pattern. Userspace gets its ~50ms of CPU once per second, and a 50ms pipeline against a 950ms drought loses every time.

Deepening the pipeline does not fix it either, and this is the part I find genuinely interesting. Pre-built packets carry pre-built protocol state. A pipeline deep enough to bridge the drought is a second of frozen, potentially stale announcements. etf’s tolerance equals pipeline depth, and pipeline depth is bounded by state staleness, not by etf.

Two operational landmines

Documented so nobody steps on them twice.

A software etf qdisc silently drops every packet on its band that lacks a launch timestamp. Including ARP. Install it on all queues of an interface and you blackhole the interface. Neighbour resolution dies quietly and everything above it follows. Scope the tc filter to exactly the timestamped flow, and tear the qdisc down with the experiment.

Pipelined sending splits your traffic into two scheduling classes, urgent (handshake, Poll answers) and pipelined (steady state). Getting a packet into the wrong class produces second-scale protocol latency with zero packet loss. My first pipelined build scheduled Poll Finals a full second into the future and produced a perfectly periodic renegotiation loop.

Both bugs were invisible in logs and obvious in the pcap.

What the bake-off left standing

One conclusion: to survive RT starvation without priority privileges, packet transmission has to leave userspace entirely.

Which is a problem, because XDP cannot originate a packet.

Line-card BFD for plain Linux, part 4: RX-clocked TX

awasey8905@gmail.com (Abdul Wasey) — Tue, 07 Jul 2026 00:00:00 GMT

Part 3 ended with one conclusion standing: transmission has to leave userspace entirely.

The obvious way to do that in BPF does not exist. XDP is an ingress hook. You can act on packets that arrive, but you cannot originate one. bpf_timer callbacks run without a packet context, so there is nothing to XDP_TX from a timer. I checked again on kernel 7.0 hoping something had landed. It had not. The only kernel-side send path, BPF_PROG_RUN with live frames, is triggered by a userspace syscall, which puts the scheduler right back in the loop you were trying to escape.

Stop trying to originate

In steady state, a BFD peer hands you a packet every 10ms. Every one of those packets is a packet context.

So instead of building packets, rewrite the one you just received, in place: swap the MAC addresses, swap the IP addresses, set our source port, rebuild the 24-byte BFD payload from a config map that userspace keeps current, and return XDP_TX. The frame goes back out the same interface about 30 microseconds after it arrived, entirely in softirq.

The payload rebuild is the whole of it:

/* BFD payload from config. P while a Poll sequence is
 * active, F when answering the peer's P; never both. */
bfd->vers_diag   = (1 << 5) | (cfg->diag & 0x1f);
bfd->flags       = ((cfg->state & 0x3) << 6) | send_final;
if (!send_final && cfg->poll && st->final_seq != cfg->poll_seq)
        bfd->flags |= BFD_F_POLL;
bfd->detect_mult = cfg->mult;
bfd->len         = 24;
bfd->my_disc     = bpf_htonl(cfg->my_disc);
bfd->your_disc   = bpf_htonl(cfg->your_disc);
bfd->min_tx      = bpf_htonl(cfg->min_tx_us);
bfd->min_rx      = bpf_htonl(cfg->min_rx_us);
bfd->min_echo_rx = bpf_htonl(cfg->min_echo_rx_us);

st->tx_pkts++;
return XDP_TX;

There is no allocation, no socket, and no syscall in there. Every field comes either from the frame that just arrived or from a map that userspace wrote at its leisure, which is exactly why none of it cares whether userspace is currently running.

If the received packet had the Poll bit set, send_final is set and the reply carries Final. Poll sequences get answered for free, faster than any userspace implementation could.

Our transmit clock is now the peer’s transmit clock. There is no timer to service, no wakeup to miss, no process to starve. SCHED_FIFO hogs can pin every core at 100% and the replies keep flowing, because softirq processing preempts them all.

On the wire this shows up as a signature. Every userspace backend in the bake-off produced a p50 gap of 10.00ms, its own timer. The XDP path produces 8.75ms: the peer’s RFC-jittered distribution, echoed back.

Detection needed the mirror-image trick

XDP cannot see silence any more than it can originate. A dead link delivers no packets, so the program that would notice never runs.

The fix is one global bpf_timer sweeping the session map every 5ms, comparing now minus last-seen against each session’s negotiated detect time, and pushing an event to userspace through a ring buffer when a session goes quiet.

Detection latency gets quantised by the sweep interval, measured at 33 to 34ms against the 30ms RFC detect time, under full stress, and the sweep runs regardless of what userspace is doing.

The same bug, twice, by the same author

Worth confessing because it only shows up under load.

The sweep snapshots “now”, then walks sessions. A packet can arrive on another CPU between the snapshot and the check, stamping last-seen newer than now. Unsigned subtraction wraps to 18 quintillion milliseconds, which comfortably exceeds any detect time, and you get a phantom session-down.

The guard is two lines, and they are the reason the sweep can be trusted:

__s64 delta = (__s64)(now - st->last_seen_ns);
if (delta < 0)
        return 0;   /* packet raced past our now-snapshot */
if ((__u64)delta > detect_ns &&
    __sync_val_compare_and_swap(&st->alive, 1, 0) == 1)
        emit(k, st, now, 0);

The cast to __s64 is the whole fix. Subtract in unsigned and a two-microsecond race becomes an eighteen-quintillion-millisecond silence; subtract in signed and it becomes a small negative number you can recognise and ignore.

I fixed it there, then three weeks later wrote the identical bug into the userspace map-polling path and got the identical 18-quintillion log line.

Concurrent readers of monotonic timestamps get you exactly once per privilege level, apparently.

What userspace keeps

Everything that does not need to be fast: the RFC 5880 state machine, session bring-up, the 1-second slow-rate transmission the RFC requires while a session is down, and exactly one packet at the moment of transition to Up.

That last one earned its place the hard way. My first version suppressed all userspace TX the moment the session entered Up. Clean division of labour: kernel speaks, userspace shuts up.

But the transition to Up is often triggered by a packet carrying the peer’s Poll, and the Final answering it was the exact packet being suppressed. The kernel could not send it either: the triggering packet was already consumed, and XDP only speaks when the next one arrives, which the peer, waiting on its unanswered Poll, was sending at the 1-second slow rate.

The result was a beautifully stable failure loop with a 32.6ms period. Diagnosed, like everything else in this project, not from the logs, which showed a healthy state machine, but from the pcap, which showed a missing packet.

One structural limitation, stated plainly

RX-clocked TX requires the peer to have its own clock. Two RX-clocked implementations facing each other would echo each other into silence. Nobody sends first after a gap.

The userspace slow-rate path doubles as the recovery spark, and detection of a dead peer never depended on receiving anyway. But the design assumes an asynchronous peer, and that assumption should be written on the box.

The results, and handing session control to FRR.

Line-card BFD for plain Linux, part 5: results, and putting FRR in the loop

awasey8905@gmail.com (Abdul Wasey) — Tue, 07 Jul 2026 00:00:00 GMT

Part 4 described the RX-clocked design. This is what it measured.

The full matrix: the complete stress ladder plus a five-minute soak, running at ordinary process priority, interoperating with stock FRR on the far end.

Level	p50	p99	max	flaps
L1 fair CPU	8.97	10.03	13.3	0
L2 sched churn	8.83	10.01	12.0	0
L3 timer storm	8.72	10.11	28.0	1
L4 RT hogs	8.72	10.01	11.5	0
5-min soak	8.77	10.01	10.7	0

For comparison, under that same L3 stress, bfdd flapped 44 times.

The one flap deserves telling in full

It is more informative than a column of zeros would have been.

Under the hrtimer storm, one echoed reply out of roughly 66,000 was delayed 28ms by softirq latency. The peer’s 30ms detect timer fired at the margin, correctly.

What followed, reconstructed packet by packet from the capture: the peer’s Down arrived, userspace noticed via the map within 3ms, transitioned, and answered the peer’s re-establishment Poll with Final 74 microseconds after it arrived. Down, Init, Up, Poll, Final. The complete RFC handshake, in 3.8 milliseconds.

Total session downtime across eleven minutes of hostile load: under 4ms.

So the honest characterisation is not “immune”. Kernel-path TX is immune to scheduler starvation but shares fate with softirq latency. Under extreme timer pressure it grazed the detect budget once and self-healed in less time than a single packet interval.

That is a different universe from the userspace failure mode. But it is a universe with physics in it.

Making FRR drive it

bfdd has a distributed-BFD mode where it delegates all session processing to an external dataplane over a small TCP or UNIX socket protocol called bfddp, designed so vendors can attach hardware dataplanes without patching FRR.

Implementing it took one protocol shim. bfdd creates and owns the sessions, assigns discriminators, and displays state and counters. The packets ride the XDP path. show bfd peers and show bfd peers counters report a session whose packet counts are read out of a BPF map.

The same SCHED_FIFO stress against the FRR-driven session: 0 flaps, p50 8.99, p99 11.01, max 13.01ms, uptime uninterrupted in FRR’s own CLI.

Putting FRR in the control loop cost nothing.

And the integration immediately found a bug

bfdd’s UNIX transport for that dataplane socket, --dplaneaddr unixc:, failed every connect attempt with EINVAL.

That turned into the first of six upstream bug reports, and I wrote it up separately, because the reason it had survived since the feature was introduced is more interesting than the one-line fix.

It set a pattern that held for the rest of the project: every milestone that pushed the engine somewhere new found something in bfdd that only a real dataplane at scale could reach.

Where this left things

At this point I wrote the results up and stopped, because the interesting question, “can plain Linux get line-card behaviour”, had an answer. It is worth being precise about how narrow that answer was.

What existed was lab-grade, single-hop, IPv4-only, with:

No authentication, echo, or demand mode. Three of RFC 5880’s features simply absent.
No session continuity. Sessions dropped and re-established across a bfdd restart. The dataplane tore them down on control-plane disconnect and bfdd re-added them on reconnect, which is correct but not hitless.
One session, ever. The maps were sized for 64. Exactly one had been tested.
Validation gaps. No spoofing defence worth the name, and no proof that malformed traffic could not reach userspace.
An asynchronous-peer requirement, structural to the RX-clocked design.
Numbers from VMs. The stress was applied inside the guest and hit every backend identically, so the comparisons are load-bearing, but the absolute figures still await bare metal.

A benchmark that only holds for one unauthenticated IPv4 session against a cooperative peer is a demo, not an implementation. Everything from here on is the unglamorous work of closing that list, and each item closed found something new, which is why the series keeps going.

First the correctness debt: session continuity, RFC 5880 validation, and an external review whose two most annoying suggestions turned out to be real bugs.

Line-card BFD for plain Linux, part 0: what all of this means

awasey8905@gmail.com (Abdul Wasey) — Mon, 06 Jul 2026 00:00:00 GMT

Who this is for. If you already know what BFD and XDP are, skip straight to part 1; nothing here will be new. This part exists for everyone else: people who found the series through the Linux or eBPF side and do not do routing, people who do routing and have never written a BPF program, and anyone who wants to follow the argument without first reading three RFCs. Nothing later in the series assumes you read this, but everything later assumes you know these words.

The problem in one paragraph

Two routers are connected. One of them dies, or the link between them breaks. How fast does the other one find out? If the answer is “thirty seconds”, every packet sent during those thirty seconds is dropped into a hole. The job of the protocol this series is about is to make that answer “thirty milliseconds” instead, and the whole project is about what it takes to keep that promise on an ordinary Linux box that is busy doing other things.

The routing side

BFD (Bidirectional Forwarding Detection). A deliberately stupid protocol whose only job is to notice that a neighbour has stopped answering. Two systems send each other small UDP packets at a fixed interval, and if enough consecutive packets go missing, the link is declared down. It is defined in RFC 5880, with RFC 5881 covering the single-hop case and RFC 5883 the multihop one.

It is stupid on purpose. Because it does one thing, it can do it far faster than the routing protocols themselves, which is why they lean on it.

Detect time, and “3x10ms”. A session is configured with a transmit interval and a multiplier. 3x10ms means “send every 10 milliseconds, declare the link dead after 3 missed”, so a 30ms detect time. Aggressive timers mean fast detection and less tolerance for lateness. That trade is the entire subject of this series.

Flap. A session going down and then back up. Usually the link was fine and the packets were merely late, which makes a flap worse than useless: it tears down working routes.

Session. One BFD conversation between two endpoints. A router with many neighbours runs many sessions at once, which is why the later parts care about what happens at 64 of them.

FRR (FRRouting). The open-source routing suite most Linux routers run. It implements BGP, OSPF, IS-IS, and the rest as separate daemons. bfdd is its BFD daemon, and it is both the thing measured in part 1 and the source of the six upstream bugs this project ended up reporting.

Distributed BFD, and the “dataplane”. FRR can hand the actual packet handling to an external program over a small protocol called bfddp, keeping only session setup and reporting for itself. It exists so hardware vendors can attach their own forwarding hardware. This project attaches to it as a software dataplane, which is why FRR bugs kept turning up: that code path had rarely been driven hard by anyone.

Echo mode. A BFD feature where you send a packet addressed to yourself, the neighbour’s forwarding hardware loops it straight back without its software ever seeing it, and you time the round trip. It tests the neighbour’s data path rather than its software. Covered in part 9.

GTSM, TTL 255. Every IP packet carries a hop counter that decreases each time a router forwards it. If you insist that arriving packets still read 255, you know nobody forwarded them, so the sender must be directly attached. It is a cheap way to reject spoofed traffic from far away.

Discriminator. A number each side picks to identify its end of a session, so packets can be matched to the right conversation.

The Linux side

Kernel space and user space. The kernel is the part of the operating system that owns the hardware. Ordinary programs, including routing daemons, run in user space and must be scheduled onto a CPU before they can do anything. That “must be scheduled” is where the trouble starts.

The scheduler, and starvation. When more work wants CPU than there is CPU, the kernel scheduler decides who runs. A program that is not chosen does not run at all, and a BFD daemon that is not running is not sending packets. Being starved for 300ms is enough to kill a 30ms session.

SCHED_FIFO, and RT throttling. Linux has real-time priorities. A SCHED_FIFO task outranks all ordinary tasks and runs until it yields. To stop a runaway real-time task locking the machine, the kernel reserves a small slice, roughly 50ms per second, for everyone else. If real-time work saturates every core, an ordinary program gets that one slice and nothing more. This is the condition no user-space design in part 3 survives.

Softirq. Work the kernel defers slightly, then runs at higher priority than any ordinary program, including most packet processing. Code that runs in softirq context keeps running when user-space programs are starved, which is exactly the property this project needs.

eBPF. A way to load small, verified programs into the running kernel. The kernel checks them for safety first, so they cannot crash it or loop forever. They are how you add behaviour to the kernel without patching it.

XDP (eXpress Data Path). An eBPF hook at the earliest possible point of packet receive, before the kernel has built any of its normal packet structures. An XDP program looks at a packet and returns a verdict.

If you have never seen one, the shape is the whole idea. This is not from the project, it is the smallest thing that shows what a verdict means:

SEC("xdp")
int hello(struct xdp_md *ctx)
{
        void *data     = (void *)(long)ctx->data;
        void *data_end = (void *)(long)ctx->data_end;
        struct ethhdr *eth = data;

        /* The verifier requires proving every read is in bounds. */
        if ((void *)(eth + 1) > data_end)
                return XDP_PASS;

        if (eth->h_proto == bpf_htons(ETH_P_IPV6))
                return XDP_DROP;   /* discard, never reaches the stack */

        return XDP_PASS;           /* carry on to the normal kernel path */
}

There are three verdicts that matter here: XDP_PASS to continue to the normal stack, XDP_DROP to discard the packet outright, and XDP_TX to send it straight back out the interface it arrived on.

Note the bounds check. Every read from a packet has to be provably in-bounds before the kernel will load the program, which is what makes it safe to run untrusted code in the receive path, and also what makes writing it feel unlike ordinary C.

The crucial limitation, and the hinge of this whole project: XDP can only react to a packet that has arrived. It cannot create one. That is why part 4 stops trying to transmit and starts bouncing the peer’s own packets back instead.

BPF maps. Shared tables that BPF programs and ordinary programs can both read and write. They are how the kernel half and the user-space half of this engine exchange configuration and state.

bpf_timer. A timer that fires inside the kernel and runs a BPF function. Used here to notice sessions that have gone quiet, since silence delivers no packet and therefore triggers no XDP program.

Offload. Moving work off the general-purpose CPU onto something that is always available. Hardware routers offload BFD to dedicated line cards. This project offloads it to the kernel’s receive path, which is the nearest thing an ordinary Linux box has to a line card.

The measurement side

p50, p99, and max. If you sort every measurement, p50 is the middle value and p99 is the value 99% fall below. They describe the typical case and the common bad case. Max is the single worst.

This distinction is not pedantry, it is the finding in part 1: a BFD daemon flapping badly enough to tear down routes had a completely healthy-looking p99, because the events that killed it were rare. Anything that summarises away the maximum hides this class of failure entirely.

Inter-packet gap. The time between one packet leaving and the next. For BFD this is the number that matters, because a long gap is what makes the far end give up.

pcap, and tcpdump. A packet capture is a recording of what actually crossed the wire. tcpdump makes one. Every number in this series comes from a capture taken outside the machine under test, on the virtualisation host, because a starved program’s own logs lie about what it sent and when.

stress-ng. A tool for deliberately loading a machine in specific ways: CPU, context switching, timers. The “stress ladder” in this series is four escalating levels of it.

Topotest. FRR’s integration test framework. It builds a small virtual network of routers and asserts things about their behaviour. Getting one written for a dataplane bug turned out to be most of the work in one of the upstream fixes.

Ready

That is the vocabulary. Part 1 starts by measuring whether the received wisdom about software BFD is actually true.

Line-card BFD for plain Linux, part 1: measuring the folklore

awasey8905@gmail.com (Abdul Wasey) — Mon, 06 Jul 2026 00:00:00 GMT

BFD is the failure detector under BGP, OSPF, and IS-IS. Two routers exchange small UDP packets at a negotiated interval, and if one side misses enough of them, typically 3 packets in 30ms at aggressive timers, it declares the link dead and the routing protocol withdraws routes.

The entire value of the protocol is timing. An implementation that sends late is worse than no BFD at all, because a false timeout tears down routes for a link that is actually fine. The failure detector becomes the failure.

The folklore

Operational folklore says software BFD cannot be trusted with aggressive timers under CPU load. It is why people configure 3x300ms instead of 3x10ms, why SONiC’s software BFD documentation caps timers at 300ms, and why hardware routers offload BFD to line cards. Juniper goes as far as dedicating an offload CPU with DPDK flow filters on the vSRX, which is a vendor telling you the forwarding plane cannot be trusted to deliver BFD packets to a busy control plane on time.

I wanted three things: to quantify the actual failure modes instead of repeating the folklore, to understand why userspace BFD fails when it fails, and to see whether plain Linux on a commodity NIC can get line-card behaviour using XDP.

This is a lab project, a few weeks of evenings on a Proxmox testbed. Every claim in it is backed by a packet capture in the repo.

The testbed, and one decision that mattered

Three Ubuntu 26.04 VMs (kernel 7.0, virtio-net with multiqueue) on an isolated Proxmox bridge: a device under test, an FRR peer, and a load generator that ended up mostly unused, because the interesting load is on the DUT itself.

The measurement instrument is tcpdump on the hypervisor bridge, not inside the guest. The host sees actual wire times and does not care what the guest’s processes believe.

That choice turned out to matter more than expected, and part 2 has a concrete example of process logs and the wire disagreeing completely.

The baseline

FRR 10.5.1’s bfdd, a single-hop session at 3x10ms (30ms detect time), and an escalating stress-ng ladder on the DUT. Gaps are DUT-to-peer inter-packet times from the host capture. Nominal spacing with RFC jitter is 7.5 to 10ms.

Level	Load	p50	p99	max gap	session
L1	4x CPU hogs (fair sched)	8.81	10.03	~10ms	survived
L2	8x CPU + context-switch churn	8.73	10.03	750ms	flapped
L3	CPU + timer/timerfd/hrtimer pressure	8.82	10.16	970ms	44 flaps in 120s
L4	4x SCHED_FIFO prio-50 hogs	8.82	287	960ms	~40% of packets never sent

The number that should worry you

It is not the 970ms.

It is the p99 sitting at 10.16ms in the same run. Nominal. Perfect.

The starvation events that kill sessions are rare, a handful per hundred seconds, so they are invisible to percentile monitoring. A dashboard graphing p99 packet spacing on that box would have shown a completely healthy BFD daemon while it flapped 44 times in two minutes.

If your alerting is percentile-based, you do not find out about this class of failure from your monitoring. You find out from the routing protocol, after the withdrawal.

That pattern, p99 fine and max fatal, is invisible in any summary statistic that is not a maximum, and it shows up repeatedly in the rest of this series.

Having measured the folklore, the obvious next step was to build the XDP path and prove the point. Instead I wrote the dumbest possible userspace BFD daemon first, expecting it to die the same way, and it refused to.

Teaching an NGINX directive to take a variable, so my VPN could skip the login

awasey8905@gmail.com (Abdul Wasey) — Wed, 17 Sep 2025 00:00:00 GMT

I run about thirty self-hosted services behind a single NGINX reverse proxy, all of them sitting behind a JWT auth gateway backed by Google OAuth. That is the right default for anything reachable from the internet.

It is a slightly irritating default when I am already on my own WireGuard tunnel, on my own network, and I just want to look at a dashboard. I wanted one rule: authenticate everything, except traffic arriving from the VPN subnet.

nginx is very good at this sort of thing. The geo module exists precisely to turn a client address into a variable:

geo $jwt_enabled {
    default        on;
    10.10.10.0/20  off;   # VPN subnet
}

Then hand $jwt_enabled to the directive that turns authentication on and off, and you are done in four lines.

Except you are not, because the directive would not take it.

Why a directive cannot always take a variable

This is the part I had not internalised before, and it is worth spelling out, because from the config file everything looks like a variable.

An nginx module declares each directive with a parse-time handler and a set of flags. The one here looked like this:

{ngx_string("auth_jwt_enabled"),
 NGX_HTTP_MAIN_CONF | NGX_HTTP_SRV_CONF | NGX_HTTP_LOC_CONF | NGX_CONF_FLAG,
 ngx_conf_set_flag_slot,
 NGX_HTTP_LOC_CONF_OFFSET,
 offsetof(auth_jwt_conf_t, enabled),
 NULL},

NGX_CONF_FLAG with ngx_conf_set_flag_slot means: at config load time, read the token, expect exactly the string on or off, and write a 1 or 0 into an ngx_flag_t field. By the time a request arrives there is no string left, no expression, nothing to evaluate. Just an integer that was decided when nginx started.

So auth_jwt_enabled $jwt_enabled; does not fail at runtime. It fails at startup, with a complaint about an invalid value, because $jwt_enabled is neither on nor off.

A variable is only meaningful to a directive that was written to defer.

The change

nginx already provides the machinery: complex values. A directive declared with ngx_http_set_complex_value_slot stores a compiled expression, and the module evaluates it per request with ngx_http_complex_value().

Three coordinated edits. The directive:

{ngx_string("auth_jwt_enabled"),
 NGX_HTTP_MAIN_CONF | NGX_HTTP_SRV_CONF | NGX_HTTP_LOC_CONF | NGX_CONF_TAKE1,
 ngx_http_set_complex_value_slot,
 NGX_HTTP_LOC_CONF_OFFSET,
 offsetof(auth_jwt_conf_t, enabled),
 NULL},

The field, from a flag to a pointer:

-  ngx_flag_t enabled;
+  ngx_http_complex_value_t *enabled;

And the decision, moved from startup to request time:

ngx_int_t enabled = 1;

if (jwtcf->enabled != NULL) {
  ngx_str_t cv;
  if (ngx_http_complex_value(r, jwtcf->enabled, &cv) == NGX_OK && cv.len > 0) {
    if (ngx_strncmp(cv.data, "off", 3) == 0) {
      enabled = 0;
    }
  }
}

NGX_CONF_FLAG became NGX_CONF_TAKE1, because the directive now takes one arbitrary argument rather than a boolean the parser recognises. Static configs keep working unchanged: auth_jwt_enabled on; is now a complex value that happens to be constant, which evaluates to the string on on every request.

Note which way the comparison runs. Only the literal off disables authentication. An empty value, a typo, a geo block that fell through to nothing: all of those leave enabled at 1 and the request gets authenticated. For a security directive that is the correct direction to fail, and it was worth being deliberate about rather than writing != "on".

The default flipped, and that broke everything

Here is the part I did not see coming.

ngx_flag_t fields use nginx’s NGX_CONF_UNSET sentinel, and the module’s merge function resolved unset to 0:

conf->enabled = prev->enabled == NGX_CONF_UNSET ? 0 : prev->enabled;

Unconfigured meant off. That is what made the module inert in a config that never mentioned it.

A pointer cannot use that sentinel. Unset is now NULL, inheritance is a plain pointer copy, and my runtime code starts from ngx_int_t enabled = 1 and only ever clears it on an explicit off.

Read those two together: NULL no longer means off, it means “nothing said otherwise”, and the code says the default is on. So after compiling my first version, JWT authentication was applied globally, on an ordinary nginx config with no JWT directives anywhere in it.

That is about as loud as a regression gets. Every site on the box demanding a token nobody had configured.

My fix was a guard at the top of the handler: if no key and no keyfile are configured, this module has nothing to validate against, so decline.

// Only activate JWT logic if key or keyfile_path is set
if (jwtcf->key.len == 0 && jwtcf->keyfile_path.len == 0) {
  return NGX_DECLINED;
}

Using “is a key configured” as the proxy for “is this module in use” is not elegant, but it is true: a JWT module with no key cannot do its job under any configuration.

The review

I opened the PR in mid-August and, after a couple of weeks, sent a polite nudge asking for even a quick reaction so I could refine the approach. The maintainer replied the same day, called it a good idea, and came back with formatting cleanup and tests.

He also removed my guard, saying he thought it was redundant because similar checks are handled further down, and asked me to add it back in the right place if it really mattered.

That is exactly the sort of comment worth taking seriously rather than defending, because it is checkable. I rebuilt with his commits and ran a plain config with no JWT directives against it. The global-auth problem did not reappear. His restructuring had moved the early-exit somewhere that covered the case my guard had been papering over, so the guard genuinely was redundant against the new shape of the code. I said so and agreed to drop it.

The tests he added are the neat part, because testing an IP-based feature in CI is awkward. Instead of a subnet, drive the variable from a request header:

map $http_test_auth_enabled $jwt_enabled {
    default  on;
    on       on;
    off      off;
}

location /enabled/variable {
    auth_jwt_enabled $jwt_enabled;
    ...
}

Then two curl cases: send Test-Auth-Enabled: on and expect 401, send off and expect 200. That tests the mechanism that actually changed, which is “can this directive read a runtime variable”, without needing a second network.

Merged and released as v2.4.0, about a month after I opened it.

What I took from it

The config file lies to you about what is a variable. $foo looks uniform everywhere it appears, but whether a directive can see it is decided in C, at module-definition time, by which parse handler the author picked. There is no way to tell from the outside except by trying it and reading the error.

The more useful lesson is about sentinels. Changing a field’s type quietly changed what “unconfigured” meant, and the default went from off to on in a security module. Nothing in the diff said “default changed”; it fell out of ngx_flag_t having a sentinel and a pointer not having one. When you change a representation, the values that were previously impossible are the ones to go looking at.

And the feature does what I wanted. On the VPN, dashboards open. Off it, Google OAuth. Four lines of geo and one directive that finally accepts what the rest of nginx has been handing it all along.

The disk was fine, the page was not: a two minute smartctl and a GUI that gave up

awasey8905@gmail.com (Abdul Wasey) — Tue, 15 Apr 2025 00:00:00 GMT

I plugged a 2.5 inch SSD into a Proxmox box through a USB-to-SATA adapter, went to Node → Disks in the web UI, and got a spinner followed by Communication Failure.

The disk itself was completely fine. lsblk listed it, fdisk read it, I mounted it and copied files to it. Every tool at the command line agreed the drive was healthy and present. Only the GUI disagreed, and it disagreed by hanging rather than by saying anything useful. journalctl, syslog and the pvedaemon logs had nothing.

That gap, everything works except the one thing that is supposed to show you that everything works, is what made it worth chasing.

Reading someone else’s Perl

Proxmox’s API is Perl, which is convenient here because it means the thing that hung is sitting on disk in readable form. Following the call path:

PVE/API2/Disks.pm handles the disk listing endpoint
which calls into PVE/Diskmanage.pm
where get_smart_data() shells out to smartctl

And the call site looked like this:

if (!$nosmart) {
    eval {
        my $smartdata = get_smart_data($devpath, !is_ssdlike($type));
        $health = $smartdata->{health} if $smartdata->{health};

        if (is_ssdlike($type)) { # if we have an SSD, try to get the wearout indicator
            my $wear_level = get_wear_leveling_info($smartdata);
            $wearout = $wear_level if defined($wear_level);
        }
    };
}

The eval catches errors. It does not catch slowness. And get_smart_data() invoked smartctl through run_command with no timeout at all, so if that command decided to take its time, the API request behind it simply waited.

The cheapest possible confirmation was to delete the problem:

if (!$nosmart) {
}

Then systemctl restart pvedaemon, reload the page, and the Disks view came up instantly. Not a fix, obviously, but it turned a theory into a fact in about thirty seconds.

The question that reframed it

I filed bug #6224 with that analysis, and the reply from Fabian Grünbichler was two short questions asked one after the other.

First: if you run the smartctl command manually, does it ever return?

It does:

# smartctl -a /dev/sdb
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.8.12-6-pve] (local build)

Read NVMe Identify Controller failed: scsi error unsupported field in scsi command

Then: how long does that take? Prefix it with time.

real    2m0.758s
user    0m0.013s
sys     0m0.000s

Two minutes. Not a hang. A return.

I had been describing this as something hanging indefinitely, and the whole time it was finishing, just far too late for anything to care. That difference matters enormously for the fix. A genuine deadlock needs to be found and broken. A slow return needs a deadline. They are not the same bug and they do not have the same remedy, and I would have gone on describing it wrongly if I had not been asked to put a number on it.

Two questions, no speculation, both answerable in a terminal. It is a good model for triage.

The root cause is a reasonable default meeting an unreasonable device

Daniel Kral found the rest of it, and it is more interesting than a plain bug.

smartmontools uses a 60 second default timeout for SCSI commands. That is not carelessness: it is sized so that a disk in a large JBOD has time to spin up before the command gives up. On the hardware that default was written for, it is the right number.

Devices behind the USB Attached SCSI driver are the ones most likely to trip it, because a USB-to-SATA bridge translating SCSI commands can fail in ways that consume the full timeout instead of erroring promptly. My 2m0.758s is that 60 second budget being spent twice, plus change.

So nothing here is broken in isolation. smartmontools waits as long as its slowest supported hardware needs. The adapter fails slowly rather than quickly. Proxmox calls the command without a deadline because it has no reason to expect one is needed. Put all three together and a web page stops loading.

The fix

A three-patch series. The one that matters is a single word:

 my $returncode = eval {
-	run_command($cmd, noerr => 1, outfunc => sub {
+	run_command($cmd, noerr => 1, timeout => 10, outfunc => sub {

Ten seconds instead of sixty, on the reasoning that a healthy drive answers almost immediately and anything slower is not worth blocking a UI over. The other two patches make the failure land properly once it can happen: one separates the error path so get_smart_data() dies with the real message, and one masks that error into an unknown health value, because the web UI expects a health field and was being handed an error string instead.

That last one is the tell that a timeout was never expected here. The moment the call could fail, the error handling around it turned out to be wrong too.

The patch is in pve-storage today. Amusingly, the bug is still marked PATCH AVAILABLE rather than resolved, so the tracker is more pessimistic than the source tree.

What I took from it

The debugging lesson is the timing question. “It hangs” is a description of your patience, not of the program. time turns it into a measurement, and the measurement pointed straight at a 60 second constant that a human had chosen deliberately for a reason that was sound.

The engineering lesson is that any call into an external command is a call into someone else’s timeout policy, whether or not you have thought about it. Proxmox did not choose 60 seconds. It inherited it from smartmontools, which inherited its requirements from JBOD spin-up times, and the UI inherited the consequences from both.

I also did not write the fix, and that is fine. Filing something with the call path traced, the offending call identified, a workaround that isolates it, and then answering the maintainer’s questions with numbers, is a complete contribution. The people who own the code are much better placed to decide whether the right answer is ten seconds or thirty.

Abdul Wasey — Writing

Line-card BFD for plain Linux, part 11: every bug came from a capture, none from a log

Every bug came from a capture, none from a log

The three things that survived

What I would say to someone starting something similar

The honest limitations, restated

Enabling echo mode took the session down, and the kernel picked the wrong address

Who chooses the source

Why most people never see it

The fix, and why it is not a setsockopt

Writing a test for source address selection

What I took from it

Line-card BFD for plain Linux, part 10: multihop, and a test that validated nothing

Where the comparison can live

Then the milestone taught the same lesson the project keeps teaching

The reply had a subtler version of the same problem

One IPv6 asymmetry worth recording

Reading the validation took a trick

Next

Line-card BFD for plain Linux, part 9: echo mode, and instrumentation that cannot see its own failure

XDP cannot originate packets, and there is no way around it

So the milestone split along the line the hardware draws

Two spec errors, surfaced the moment there was code

Detection is advisory, permanently

The sharpest lesson in the milestone

And a lesson about my own habits

The upstream bug had two halves facing each other

Next

The RFC says negotiate, but nobody was listening for offloaded sessions

Where the negotiation lives, and why it was unreachable

The information was already there

The fix

What I took from it

Line-card BFD for plain Linux, part 8: IPv6, and the bake-off argument in miniature

The structural move came first, and alone

The v6 parse path

The experiment

Scale, and closing an upstream loop

Next

A read of zero bytes that looked exactly like a closed socket

Two defects, and neither is enough on its own

The arithmetic is exact, which is how I knew I had it

Why this is worse than a crash

The fix

The review asked for a test, and the test needed a dataplane

What I took from it

Deleting sessions into a buffer nobody would ever flush

Why they never leave

Reproducing without a dataplane

The fix

What I took from it

How 8KB of buffer silently lost BFD sessions in FRR

What the dataplane protocol does at connect time

Why the failure was invisible

Reproducing it without a dataplane

The fix

Result

What I took from it

Line-card BFD for plain Linux, part 7: one session to sixty-four

Four things that only worked because there was one of them

Two sessions, then sixteen, then sixty-four

Sixty-four, and five bugs

The design’s own boundary

What I took from it

Next

Line-card BFD for plain Linux, part 6: hardening, and a review that found real edges

Surviving a control-plane restart

Refusing traffic properly

The review pass

What I took from it

Next

Two bytes of padding, and a transport that never worked

The address was fine, the length was not

Why only AF_UNIX noticed

The fix

What I took from it

Line-card BFD for plain Linux, part 2: the folklore is partly wrong

It did not die

Why, and it is not luck

Yes: SCHED_FIFO