IPv6, echo, and multihop

Back in part 3 I had a list of things xdp-bfd couldn’t do yet. After the 64-session work in July, the biggest ones left were IPv6, echo mode, and multihop sessions. This part goes through all three, which took the rest of July.

IPv6

Adding IPv6 was mostly careful plumbing. I changed the session key so both IPv4 and IPv6 addresses fit in it, then taught the XDP program to read IPv6 headers and build IPv6 replies. The more interesting part was something I hadn’t planned.

This whole project rests on one idea: sending the reply from the kernel is what keeps sessions up under load. Up to here, though, I’d only shown that by comparing different programs on different runs, so the load was never exactly the same between them.

While I was adding IPv6, there was a point where the XDP program could already read IPv6 packets, but IPv6 replies were still being sent from userspace. IPv4 replies were already coming from the kernel. So I ran both at the same time, on the same box, through the same stress, with 300ms timers. IPv4 had zero flaps. IPv6 had 19, with gaps of up to 1.9 seconds.

Then I turned on the kernel reply for IPv6 and ran it again. IPv6 had zero flaps too. The only thing that changed between those two runs was where the IPv6 reply came from, so that settled it for me.

With both in place, I ran 32 IPv4 and 32 IPv6 sessions together under stress, and neither had a single flap.

Echo

Echo is a BFD mode where you send a packet addressed to yourself. The neighbour’s router just forwards it back to you, without its BFD daemon ever seeing it, and you time how long the round trip took. It checks that the neighbour can still forward packets, not just that its BFD software is running. I expected it to be one of the easier features.

XDP can only do half of it, though. Bouncing a neighbour’s echo back is easy, because there’s a packet in hand, the same trick as part 2. But sending my own echoes means creating packets from nothing, and XDP can’t do that. So the bouncing happens in XDP and the sending happens from userspace. Since userspace sending can be starved, I also made sure a failing echo only gets reported and never takes a session down on its own.

Up to then, I had only tested the bouncing with echo packets I built by hand. While working on the FRR side of echo, I found out why no real peer had ever been part of those tests: xdp-bfd had been telling every peer it couldn’t receive echo. BFD has a field for how often you’re willing to receive echo packets, and xdp-bfd was sending 0 there, which means “don’t send me any”. Once I fixed that, FRR started sending echoes to it, and xdp-bfd bounced back all 433 of them in that test, the fastest in 12 microseconds.

The most confusing bug came under load. Echo sending stalled for 2.6 seconds at a time, but the loss counter stayed at zero and echo still showed as healthy. The counter only counts a lost echo when it’s time to send the next one and the last one hasn’t come back. During a stall, it’s never time to send the next one, so nothing ever got counted. I only caught it by separately measuring the time between sends.

In September I found one more echo problem, this time with IPv6. FRR’s IPv6 echo works differently from IPv4. Instead of addressing it to itself, bfdd sends it to the peer and expects the peer to send it back. On the xdp-bfd side that only worked while the local bfdd happened to have an echo socket open, which in data plane mode depends on the order its config gets applied. After an FRR restart it didn’t have one, so the host answered those echoes with an ICMP error instead, and all 28 IPv6 echo sessions on the test mesh started flapping. XDP now bounces those echoes itself, and the socket problem is an open FRR fix.

Multihop

Normal BFD sessions are between directly connected boxes, which is why part 3 required a TTL of 255. Multihop sessions go through other routers on the way, so the TTL arrives lower, and instead each session gets a minimum TTL it will accept. I kept the old strict check for single-hop sessions and only let lower TTLs through for sessions that are actually set up as multihop.

I wrote a test for it, and everything passed on the first try. When I looked at the capture afterwards, though, the test hadn’t been testing multihop at all. Multihop BFD uses port 4784, and my test was sending to 3784, the port for normal single-hop sessions. The capture showed FRR sending to 4784 and my test packets going to 3784, and none of them had touched the multihop code. After I fixed the port, the test finally ran against the multihop path.

Next: demand mode, auth, and restarts that don’t drop anything.