The TX bake-off

Part 2 established that userspace BFD survives ordinary load and loses to SCHED_FIFO. So the question became: which transmit architectures survive RT starvation, and at what cost?

Five backends, identical conditions: 4x SCHED_FIFO prio-50 hogs, 60 seconds, the 3x10ms session, wire-truth capture from the hypervisor.

The chrt row, and its asterisk

Running the daemon at RT priority 90, above the prio-50 hogs, works completely. Zero flaps, 15ms max.

The asterisk is that you won the priority war because nothing else contested it. On a real router you cannot assume that. Forwarding work, softirq processing, and other components are fighting for the same priorities, and a priority arms-race between your failure detector and your data plane is not a design, it is a standoff.

The etf rows are the most instructive failure of the project

The theory is genuinely attractive. SO_TXTIME lets userspace enqueue a packet with an explicit future launch time, and the etf qdisc releases it at that nanosecond via hrtimer. Wire timing becomes independent of when userspace got scheduled.

And it delivers on exactly that promise. The pipelined variant produced the best p99 of every backend tested, 10.1ms.

It also flapped 48 times. Worse than doing nothing special at all.

Zero packets were dropped by the qdisc in those runs. etf did its job perfectly on every packet it was given. The problem is that a starved daemon gives it nothing.

Timestamping fixes jitter. It cannot manufacture liveness.

Pipelining five packets ahead buys 50ms of starvation tolerance. The failure signature in the gap data is unmistakable: max gaps clustering at 915 to 995ms, which is the RT throttle’s breathing pattern. Userspace gets its ~50ms of CPU once per second, and a 50ms pipeline against a 950ms drought loses every time.

Deepening the pipeline does not fix it either, and this is the part I find genuinely interesting. Pre-built packets carry pre-built protocol state. A pipeline deep enough to bridge the drought is a second of frozen, potentially stale announcements. etf’s tolerance equals pipeline depth, and pipeline depth is bounded by state staleness, not by etf.

Two operational landmines

Documented so nobody steps on them twice.

A software etf qdisc silently drops every packet on its band that lacks a launch timestamp. Including ARP. Install it on all queues of an interface and you blackhole the interface. Neighbour resolution dies quietly and everything above it follows. Scope the tc filter to exactly the timestamped flow, and tear the qdisc down with the experiment.

Pipelined sending splits your traffic into two scheduling classes, urgent (handshake, Poll answers) and pipelined (steady state). Getting a packet into the wrong class produces second-scale protocol latency with zero packet loss. My first pipelined build scheduled Poll Finals a full second into the future and produced a perfectly periodic renegotiation loop.

Both bugs were invisible in logs and obvious in the pcap.

What the bake-off left standing

One conclusion: to survive RT starvation without priority privileges, packet transmission has to leave userspace entirely.

Which is a problem, because XDP cannot originate a packet.