Demand mode, auth and hitless restarts

After part 5, three big things were still missing, and they took up September. There was no demand mode, no authentication, and restarting xdp-bfd itself, for an upgrade or anything else, took every session down with it.

Demand mode

In demand mode, two peers stop sending regular packets once both are Up, and only check the path every now and then. bfdd had a setting for it, but the setting didn’t do anything. The flag was set to 0 in three places and never set to 1 anywhere. So I implemented it in FRR first, which has its own section in the FRR post, and then in xdp-bfd.

The trick from part 2 doesn’t work here, though. When the peer stops sending, there’s nothing for XDP to bounce. So while the peer is in demand mode, xdp-bfd stops replying from XDP and leaves the occasional Poll to userspace. While xdp-bfd is the side asking for quiet, it also holds off its own detection, or it would time out a peer that was only doing what it was asked.

At first it only worked in one direction. When xdp-bfd was the side asking for quiet, the peer just kept sending. It turned out xdp-bfd was going quiet within a microsecond of both sides being Up, before it had sent a single packet saying it wanted demand mode, so the peer had never actually been asked. Now it says so three times first.

Once both sides are quiet, something still has to check the path is there. The RFC leaves that optional and bfdd never does it, so xdp-bfd sends a Poll once a second.

Signing packets inside XDP

BFD authentication adds a signature to every packet, so the other side can tell it really came from you. In xdp-bfd the replies are built in XDP, so XDP had to be the one signing them, and it also had to check the signature on every packet coming in.

The kernel doesn’t give XDP programs a hash function to call, so I wrote SHA1 into the XDP program myself. The tricky part was that BPF programs only get 512 bytes of stack. The usual way of computing SHA1 keeps 80 values around at once, and that didn’t fit next to everything else. So my version only keeps the last 16 and works out the rest as it goes.

FRR’s keyed SHA1 also isn’t quite what the BFD RFC describes. It’s an HMAC, which is a different way of mixing the key in. xdp-bfd does it FRR’s way, since FRR is what it has to talk to, and I opened an issue in FRR about the difference.

Getting the keys from FRR

xdp-bfd could sign packets now, but bfdd had no way to hand it the keys. The message bfdd uses to pass a session to xdp-bfd had a TODO comment where the key was supposed to go. On top of that, when you set up an authenticated session in FRR and it got handed to a data plane, it went out with no authentication at all, while show bfd peer still said auth was on.

I fixed that in FRR, and it was merged in late September. bfdd now sends the keys that are usable now and the ones that will be later, along with when each is valid, not just the current key. That way xdp-bfd can switch to the next key on its own when one expires. Around the same time I also fixed two more auth bugs in bfdd itself, and those are in the FRR post.

On the test mesh, all 68 authenticated sessions came up.

Restarting without dropping anything

The other problem was restarts. Every time I restarted xdp-bfd, every session went down with it. On the test mesh with 1024 sessions, one restart caused 1022 flaps.

What I wanted was for a new copy of xdp-bfd to take over from the old one without the peers ever going unanswered. So I added a --pin option. With it, xdp-bfd pins its BPF maps and the attachment of its XDP program in the kernel, so they stay there after the process exits. When a new xdp-bfd starts, it picks up the pinned sessions, swaps its own XDP program into the pinned attachment in one step, and tells the old one to exit. The attachment and the session state never go away, so the peers keep getting answers the whole time. The old process was gone within 13ms and the new one had taken over in about 12ms, and this time the restart caused zero flaps.

With that, the test mesh was 1024 sessions on Proxmox VMs, covering everything from the earlier parts, and all of them came up apart from two I left pointing at peers that don’t exist on purpose.

Next: checking my own work.