RX-clocked TX

Part 3 ended with one conclusion standing: transmission has to leave userspace entirely.

The obvious way to do that in BPF does not exist. XDP is an ingress hook. You can act on packets that arrive, but you cannot originate one. bpf_timer callbacks run without a packet context, so there is nothing to XDP_TX from a timer. I checked again on kernel 7.0 hoping something had landed. It had not. The only kernel-side send path, BPF_PROG_RUN with live frames, is triggered by a userspace syscall, which puts the scheduler right back in the loop you were trying to escape.

Stop trying to originate

In steady state, a BFD peer hands you a packet every 10ms. Every one of those packets is a packet context.

So instead of building packets, rewrite the one you just received, in place: swap the MAC addresses, swap the IP addresses, set our source port, rebuild the 24-byte BFD payload from a config map that userspace keeps current, and return XDP_TX. The frame goes back out the same interface about 30 microseconds after it arrived, entirely in softirq.

The payload rebuild is the whole of it:

/* BFD payload from config. P while a Poll sequence is
 * active, F when answering the peer's P; never both. */
bfd->vers_diag   = (1 << 5) | (cfg->diag & 0x1f);
bfd->flags       = ((cfg->state & 0x3) << 6) | send_final;
if (!send_final && cfg->poll && st->final_seq != cfg->poll_seq)
        bfd->flags |= BFD_F_POLL;
bfd->detect_mult = cfg->mult;
bfd->len         = 24;
bfd->my_disc     = bpf_htonl(cfg->my_disc);
bfd->your_disc   = bpf_htonl(cfg->your_disc);
bfd->min_tx      = bpf_htonl(cfg->min_tx_us);
bfd->min_rx      = bpf_htonl(cfg->min_rx_us);
bfd->min_echo_rx = bpf_htonl(cfg->min_echo_rx_us);

st->tx_pkts++;
return XDP_TX;

There is no allocation, no socket, and no syscall in there. Every field comes either from the frame that just arrived or from a map that userspace wrote at its leisure, which is exactly why none of it cares whether userspace is currently running.

If the received packet had the Poll bit set, send_final is set and the reply carries Final. Poll sequences get answered for free, faster than any userspace implementation could.

Our transmit clock is now the peer’s transmit clock. There is no timer to service, no wakeup to miss, no process to starve. SCHED_FIFO hogs can pin every core at 100% and the replies keep flowing, because softirq processing preempts them all.

On the wire this shows up as a signature. Every userspace backend in the bake-off produced a p50 gap of 10.00ms, its own timer. The XDP path produces 8.75ms: the peer’s RFC-jittered distribution, echoed back.

Detection needed the mirror-image trick

XDP cannot see silence any more than it can originate. A dead link delivers no packets, so the program that would notice never runs.

The fix is one global bpf_timer sweeping the session map every 5ms, comparing now minus last-seen against each session’s negotiated detect time, and pushing an event to userspace through a ring buffer when a session goes quiet.

Detection latency gets quantised by the sweep interval, measured at 33 to 34ms against the 30ms RFC detect time, under full stress, and the sweep runs regardless of what userspace is doing.

The same bug, twice, by the same author

Worth confessing because it only shows up under load.

The sweep snapshots “now”, then walks sessions. A packet can arrive on another CPU between the snapshot and the check, stamping last-seen newer than now. Unsigned subtraction wraps to 18 quintillion milliseconds, which comfortably exceeds any detect time, and you get a phantom session-down.

The guard is two lines, and they are the reason the sweep can be trusted:

__s64 delta = (__s64)(now - st->last_seen_ns);
if (delta < 0)
        return 0;   /* packet raced past our now-snapshot */
if ((__u64)delta > detect_ns &&
    __sync_val_compare_and_swap(&st->alive, 1, 0) == 1)
        emit(k, st, now, 0);

The cast to __s64 is the whole fix. Subtract in unsigned and a two-microsecond race becomes an eighteen-quintillion-millisecond silence; subtract in signed and it becomes a small negative number you can recognise and ignore.

I fixed it there, then three weeks later wrote the identical bug into the userspace map-polling path and got the identical 18-quintillion log line.

Concurrent readers of monotonic timestamps get you exactly once per privilege level, apparently.

What userspace keeps

Everything that does not need to be fast: the RFC 5880 state machine, session bring-up, the 1-second slow-rate transmission the RFC requires while a session is down, and exactly one packet at the moment of transition to Up.

That last one earned its place the hard way. My first version suppressed all userspace TX the moment the session entered Up. Clean division of labour: kernel speaks, userspace shuts up.

But the transition to Up is often triggered by a packet carrying the peer’s Poll, and the Final answering it was the exact packet being suppressed. The kernel could not send it either: the triggering packet was already consumed, and XDP only speaks when the next one arrives, which the peer, waiting on its unanswered Poll, was sending at the 1-second slow rate.

The result was a beautifully stable failure loop with a 32.6ms period. Diagnosed, like everything else in this project, not from the logs, which showed a healthy state machine, but from the pcap, which showed a missing packet.

One structural limitation, stated plainly

RX-clocked TX requires the peer to have its own clock. Two RX-clocked implementations facing each other would echo each other into silence. Nobody sends first after a gap.

The userspace slow-rate path doubles as the recovery spark, and detection of a dead peer never depended on receiving anyway. But the design assumes an asynchronous peer, and that assumption should be written on the box.

Next

The results, and handing session control to FRR.