Checking my own work

While I was building the things in part 6, from the middle of August to the end of September, I spent just as much time checking what I already had. Most of xdp-bfd runs inside the kernel, where I can’t step through it or print anything, and nearly all the tests were ones I’d written myself while building each feature.

The review

I started by getting someone else to go through all of it. The first thing they found was that the two halves of xdp-bfd didn’t agree with each other. XDP and userspace both look at incoming BFD packets, and each had its own checks. A packet that said it was 200 bytes long but was really 24 got dropped by XDP and accepted by userspace, because userspace never looked at the length. I moved the checks into one function that both of them call.

The TTL check had a worse problem. Userspace asked the kernel to drop anything below TTL 255 using IP_MINTTL, but when I tested it in network namespaces, a TTL-64 packet went straight through. It turns out Linux only enforces that option for TCP. For UDP it does nothing, so the check had never worked. Now userspace reads the TTL off each packet itself.

Dead but still answering

The review also asked what happens when the engine dies. I killed it with kill -9, and XDP kept answering the peer anyway, using whatever config it had last. The peer had no idea anything was wrong. An XDP program normally stays attached after the process that loaded it is gone, so I switched to attaching it through a BPF link, which the kernel removes when its owner dies.

A stuck engine was harder. At the end of August I froze it for 15 seconds, and 55 of 64 sessions stayed Up the whole time. It’s the same thing that kept sessions alive under load in part 2, except this time XDP was telling the peer everything was fine for an engine that wasn’t running at all. So in September I added a dead-man gate. The engine bumps a heartbeat every time its loop goes round, and if the heartbeat gets older than a second, XDP stops answering and lets the peers time out on their own.

My first test for the gate was wrong too. It said the problem was fixed as soon as any session went down, and it said that while 55 were still up. A session going down while the engine is frozen doesn’t prove much anyway, since the frozen engine explains it just as well. So now the test runs twice: once with the gate on, where the session has to go down, and once with it off, where it has to stay up. After that I did the same for every new test, running it once with the thing it tests turned off to make sure it fails.

Fuzzing

Around the same time I pointed a fuzzer at the code that reads bfdd’s messages. On its first run, at the end of August, it found a read 42 bytes past the end of the buffer. If a reply couldn’t go out because the queue was full, the connection got dropped in the middle of the parse loop, the byte count went to zero, and the subtraction after it wrapped around. I fixed the loop and kept the input that caused it as a test.

Later I fuzzed the XDP program too. For every input, a dropped packet had to leave every session untouched, and anything sent back had to be a valid BFD packet. It got through about 10 million inputs without breaking either rule.

By the end of September the tests had grown into four layers, from a few seconds of checks that don’t need root, up to stock FRR bfdd running in containers against xdp-bfd. Every pull request also goes through about 28 CI checks.

The test that never ran

One of the checks I was counting on wasn’t running at all. Before every push, a git hook ran a script that fires bad packets at the test mesh and checks that each one gets dropped. At some point I’d moved the script to another folder, and the hook was still calling the old path. The error landed in the hook’s own “couldn’t parse, skipping” branch, which exits with success, so every push had passed for weeks without the test running once. When I fixed the path, the script turned out to have two more bugs of its own.

Answering too fast

Near the end of September I went through the RFC itself, and the biggest problem came straight out of part 2. XDP answers every packet the peer sends, so xdp-bfd was sending at the peer’s pace, not its own. If I set 300ms and the peer sent every 100ms, I was sending three times faster than the RFC allows. Now XDP only answers on its own while the peer’s pace is one I’m allowed to match. Otherwise userspace sends at my own rate, and XDP just watches for the peer going quiet.

Next, packaging it so someone else could run it.