Getting transmit out of userspace

Part 1 ended with my simple daemon breaking under SCHED_FIFO load: 25 flaps in a minute. So I started trying other ways to get the packets out on time, running each one under the same test as before (four FIFO hogs, 60 seconds, 3x10ms, captured from the hypervisor).

Raising the priority

The first thing I tried was running my daemon at real-time priority 90, above the priority-50 hogs. That fixed it completely: zero flaps and a 15ms worst gap.

It only worked because nothing else on my test box wanted a higher priority, though. On a real router the forwarding path runs at high priority too, and I didn’t want BFD fighting it for the CPU. So I kept looking.

Trying SO_TXTIME

Next I tried SO_TXTIME, which I had high hopes for. It lets you queue a packet with a launch time, and the etf qdisc sends it at exactly that time. My thinking was that even if the daemon got to run late, the packets it had already queued would still go out on time.

The first problem showed up while setting it up. I put the etf qdisc on all of the interface’s queues, and the interface quietly stopped resolving neighbours. It turns out etf drops every packet on its band that has no launch time, and that includes ARP, so I had to limit the tc filter to just the BFD flow.

Once it worked, I ran the FIFO test with one packet queued at a time, and it flapped 48 times. So I changed it to queue five packets ahead. My first try at that kept renegotiating the session once a second: packets like the Final that answers a Poll need to go out right away, and I was scheduling them a full second in the future. The logs looked normal, and I only spotted it in the capture. With that fixed, the five-packet version got the best p99 of anything I’d tried, 10.1ms, and it still flapped 48 times.

Next I checked the qdisc stats to see if it was dropping packets. It wasn’t. Every packet I’d queued had gone out exactly on time. The gaps were happening after the queue ran dry, while the daemon was waiting for CPU to queue the next ones. Five packets only lasts 50ms, and the daemon was going almost a whole second between turns.

I did think about queuing a lot more, but every packet carries the session state from when it was built. With a second of packets queued up, I’d be telling the peer what the session looked like a second ago.

Where that left me

At this point I had tried three different ways of sending from userspace. Next to bfdd from part 1, they looked like this:

The only one that survived was the one I’d bumped to a higher priority. So I started looking at sending from inside the kernel instead, since that keeps running when userspace doesn’t. XDP was the obvious place to try, except that XDP only runs when a packet arrives. It can’t send one on its own.

Bouncing the peer’s packet back

My first thought was to find some way for XDP to send on a timer. BPF has timers that run in the kernel, but a timer callback doesn’t have a packet, so there’s nothing for it to send. I checked whether kernel 7.0 had changed that, and it hadn’t.

Then I noticed that the peer is already sending me a packet every 10ms. XDP can’t create a packet, but it can change one that just arrived and send it back out. So that’s what I did. When the peer’s packet comes in, my XDP program swaps the addresses around so it points back at the peer, writes my side of the BFD session into it, and sends it straight back. Userspace keeps a small config map up to date with what my side should say, and XDP just copies from it. Here’s the part that fills in the reply:

bfd->vers_diag   = (1 << 5) | (cfg->diag & 0x1f);
bfd->flags       = ((cfg->state & 0x3) << 6) | send_final;
if (!send_final && cfg->poll && st->final_seq != cfg->poll_seq)
        bfd->flags |= BFD_F_POLL;
bfd->detect_mult = cfg->mult;
bfd->my_disc     = bpf_htonl(cfg->my_disc);
bfd->your_disc   = bpf_htonl(cfg->your_disc);
/* ...timers the same way... */
return XDP_TX;

The reply leaves about 30 microseconds after the peer’s packet arrives, and none of it waits on userspace. I don’t have a send timer anymore, because the peer’s packets are the timer. If the peer’s packet has the Poll bit set, the reply gets Final, so even Poll sequences get answered without userspace being involved.

Then I ran it through the same FIFO test, at normal priority. It had zero flaps and a 12.5ms worst gap. The p50 was different too: all the userspace versions sat at exactly 10.00ms, because that’s their own timer, but the XDP version was at 8.75ms. That’s the peer’s timing, jitter included, coming straight back to it.

Noticing a dead peer

That covered sending, but I also needed to notice when the peer stops. A dead peer doesn’t send anything, so my XDP program never runs and has no way to notice. So I added a BPF timer that looks at every session every 5ms and marks any that have been quiet for longer than their detect time. Userspace then picks that up, takes the session down and tells bfdd. Under the timer stress from part 1 it caught a dead peer 33 to 34ms after it stopped, with a 30ms detect time. For a long time I thought the extra few ms came from the 5ms timer, but when I tested that later, the extra actually came from how often the userspace loop checked for the mark.

Under load, it sometimes marked a perfectly healthy session as dead. The log said the session had been silent for about 18 quintillion ms.

Here’s what was happening. The timer grabs the current time and then goes through the sessions. If a packet comes in on another CPU during that, the session’s last-seen time ends up a little newer than the time the timer grabbed. Subtracting the two should give a tiny negative number, but I was using unsigned math, so it wrapped around to 18 quintillion instead.

The fix was to subtract as signed and skip anything negative.

What stayed in userspace

With sending and detection in the kernel, I left everything else in userspace: the state machine, bringing sessions up, and the slow once-a-second packets the RFC wants while a session is down. My plan was for userspace to go quiet once a session was Up and let XDP handle it from there.

That plan didn’t survive testing. A session got stuck going round in a loop every 32.6ms, and the logs said the state machine was perfectly happy. In the capture there was one packet missing each time, and it was a Final. Sessions often go Up because the peer sent a Poll, so the Final answering it was exactly the packet userspace had just decided not to send, and XDP had already used up the peer’s packet by then. Now userspace sends that one packet as the session goes Up, and XDP takes over after it.

Since every reply rides on one of the peer’s packets, the peer does need its own timer for any of this to work. I used to assume two xdp-bfd engines facing each other would just sit there waiting for each other. When I actually tried it, they did the opposite: each reply set off the next one, and they bounced about 690,000 packets back and forth in five seconds, with a 10ms interval configured. It needs something like FRR, with its own timer, on the other end, and that’s still the case today.

Next, putting FRR in charge of it and hardening it.