From 1 session to 8192

Every number up to part 3 came from a single session. The maps had room for 64 and the docs said “multi-session”, but I had never actually run more than one. A real router talks to a lot of neighbours, so I wanted to know what happens when xdp-bfd has to handle more than one at a time.

Going past one session

I started small. Two sessions came up fine, and when I killed FRR on one of the neighbours, that session went down in 31ms while the other one carried on without noticing. Sixteen sessions were fine too, even with the L3 stress from part 1 running.

Going to 64 sessions found five bugs. Four of them were mine, like my per-session source ports starting at 49152, which is exactly where bfdd hands out its own, so the two collided.

The fifth was in FRR. When bfdd connected, it sent all 64 sessions to xdp-bfd in one go, overflowed an 8KB buffer, and silently lost everything after the 40th. That bug ended up with its own post.

The 64-session runs under L3 showed one more thing. At one point 19 sessions flapped at the exact same moment. It was the same cause as the single flap in part 3: a softirq that ran late. With one session, a late softirq only delays one reply. With 64, every session’s reply goes through that same softirq, so they were all late together, and they all went down together. They came back on their own in about 50ms.

Why 1024 sessions was eating a CPU core

All of that was back in July. In September I came back to scale and raised the default to 1024 sessions. XDP handled that without any trouble, but the userspace side was now using 16.8% of a CPU core just to keep those sessions going.

The reason was that userspace checked the maps on a timer, going through every session each time whether anything had changed or not. With one session that was nothing. With 1024 it added up. I changed it so userspace only wakes up when there’s actually something to handle. After that it sat at 2.1% of a core with the same 1024 sessions, and sessions were still being declared down on time.

How far it goes

After that I wanted to see how far it could be pushed, so I added a --max-sessions option that goes up to 8192 and tested it against a normal FRR bfdd on the other end, with 300ms timers:

At 8192 I got 38 flaps, so I looked into them. xdp-bfd was fine. The bfdd on the other side was using 53% of a core just to keep up and was falling behind.

An audit I did just before this scale work also found a bug. XDP could read a session’s config while userspace was halfway through updating it, and send a reply carrying another session’s discriminators, or half of each version. A test built to hit that race produced these broken replies until I fixed it by having XDP lock the config and work from its own copy, which only costs about 37 nanoseconds per reply.

The strangest problem wasn’t in xdp-bfd at all. During the 4096-session tests, the host running them lost ARP for its own management address. My test setup used a lot of network namespaces, and the kernel’s neighbour table is shared between all of them. The test had filled it past its limit.

Next up is IPv6, echo and multihop.