Results, FRR, and hardening

With the XDP version from part 2 passing the FIFO test, I wanted to see how it did on the whole stress ladder from part 1. So I ran it through every level plus a five minute soak, at normal process priority, with stock FRR on the other end:

The only flap was under L3, the same load that made bfdd flap 44 times. I wanted to know what caused it, so I went through the capture around it. One reply had been stuck in softirq for 28ms, just long enough for the peer’s 30ms timer to run out, and the peer took the session down.

It came back by itself, though. My userspace saw the peer go Down and brought the session back up, and the whole thing took 3.8ms. Across all eleven minutes of testing, the session was down for less than 4ms. The scheduler wasn’t a problem anymore, but a slow softirq still could be, and that was the one time it happened.

Hooking it up to FRR

All of those runs had xdp-bfd managing the session by itself, with its own config. On a real router, though, BFD gets set up in FRR along with BGP and the rest, so I wanted FRR to be the one in charge and xdp-bfd to only handle the packets.

bfdd already has a mode for that. It can hand the packet side of its sessions to a separate program over a protocol called bfddp, which exists so hardware vendors can plug their own forwarding hardware into FRR. I added bfddp support to xdp-bfd so bfdd could use it the same way. You set up BFD in FRR like normal, and show bfd peers looks like normal, but the packets go through XDP, and even the counters FRR shows are read from the BPF map.

Since both were running on the same box, I first tried connecting them over a UNIX socket, and bfdd failed to connect every time with EINVAL. Over TCP it connected fine. The UNIX socket failure turned out to be a bug in FRR, and it was the first of quite a few FRR bugs I ran into, so those have their own post.

Once the two were talking, I ran the FIFO test again with FRR in charge. It still had 0 flaps and a 13ms worst gap, about the same as before.

That was still only one IPv4 session, though, and restarting bfdd took it down, so that’s what I worked on next.

Surviving a bfdd restart

Restarting bfdd is a normal thing to do, for an upgrade or a config change, so I didn’t want it taking routes down. The reason it did was pretty simple. As soon as bfdd disconnected, xdp-bfd threw the session away, because as far as it knew nobody was managing it anymore. bfdd added the session back when it reconnected, but by then the peer had already seen it go down, and any routes depending on it went down too.

So I added an option called --dp-hold. It tells xdp-bfd to hold on to its sessions while bfdd is away and keep answering the peer as if nothing happened. When bfdd reconnects and adds its sessions again, xdp-bfd lines each one up with the session it’s been holding, so the peer never notices, and anything bfdd doesn’t add back gets removed. Rafael Zalamena on the FRR list helped me work out how this should behave.

With that, restarting bfdd stopped taking the session down in my tests. Much later, on the big test mesh, I found where it falls short: one clean FRR restart took all of the roughly 1,000 sessions down at once. --dp-hold only helps when bfdd goes away without warning, like a crash. When FRR is stopped properly, bfdd first sends a delete for every session, and xdp-bfd removes them, since that’s what it was asked to do.

Refusing bad packets

The other big gap was that xdp-bfd barely checked the packets it received. In this design every packet that gets through tells xdp-bfd the peer is still alive, and it also gets bounced back to the peer. So a fake packet that got past the checks could keep a session up even after the real peer had died.

So I added checks. A packet has to arrive with a TTL of 255. Every router a packet passes through lowers its TTL by one, so 255 means it came from a directly connected box (this is called GTSM). The discriminator in the packet has to match the session. And a session can only be created by bfdd, not by someone sending packets at the box. If a packet fails any of these, XDP drops it and userspace never sees it.

The discriminator check needed one exception. When a peer restarts, it doesn’t know my discriminator yet, so it sends zero. I have to accept that, or the session could never come back up. But I only accept zero when the packet also says the peer is Down, so a random packet with zero in it still gets dropped. Here’s the check:

if (cfg && cfg->my_disc) {
        __u32 ydisc = bpf_ntohl(bfd->your_disc);
        if (ydisc != cfg->my_disc &&
            !(ydisc == 0 && rstate <= 1)) {   /* Down or AdminDown */
                count(3);
                return XDP_DROP;
        }
}

The first time I tested this, from another host, the checks counted every bad packet and the session still flapped. The reject paths were returning XDP_PASS instead of XDP_DROP, so the bad packets were counted and then handed up to userspace anyway, and a forged packet with the right discriminator overwrote the session’s state. After switching both to XDP_DROP, I sent 600 forged packets at it, all 600 were dropped in XDP, and the session never noticed.

The review

After adding those checks, I had someone else review the code. I nearly ignored two of their comments, but both were real bugs.

One of them was in the checks I had just written. They read the TTL and the discriminator from fixed spots in the packet, which only works if the IP header is the usual 20 bytes. A packet with IP options has a longer header, so everything after it starts later, and my code was reading the wrong bytes. That meant a packet with options could get past both checks and reach userspace. Single-hop BFD never uses IP options, so I made XDP drop any packet that has them.

The other one was in how xdp-bfd reads bfddp messages from bfdd. When a message came in broken, the reader cleared its buffer and kept reading from the same connection. It could then start reading halfway through the next message and treat whatever bytes were there as a new one. I changed it to close the connection instead. bfdd reconnects on its own, and with --dp-hold the sessions stay up while it does.

They also noticed that the structs shared between the XDP program and userspace were copied into three different files and kept in sync by hand, so I moved them into one header. I also kept a note of the suggestions I didn’t take and why, and that note ended up being really useful later.

After this, the next thing I needed was more than one session. That’s part 4.