bfdd losing sessions to a full buffer

I was testing xdp-bfd with more and more sessions, with FRR’s bfdd handing them over. At 16 sessions everything worked. At 64, some sessions just never came up, and there was nothing about them in any log.

Losing sessions on connect

The problem was in what bfdd sends when it connects to a data plane. It goes through every configured session and puts one registration message for each into an 8KB output buffer, all in one go, before any of it gets sent. A session with default settings takes a 140-byte message, so 58 of those fit. My sessions had timers configured, which makes each message 203 bytes, so only 40 fit. With my 64 sessions, the last 24 didn’t make it.

A full buffer by itself would have been a normal error. What made it bad was what bfdd did with the sessions that didn’t fit. Before registering them, it had already switched off its own software BFD for them, since the data plane was supposed to take over. Then the registration failed, and bfdd cleared the session’s link to the data plane. So those sessions weren’t being run by the data plane or by bfdd. They still showed up in show bfd peers, but nothing was actually doing anything with them.

I wanted the FRR maintainers to be able to reproduce this without needing xdp-bfd, and it turned out a real data plane wasn’t needed at all. A small script that accepts bfdd’s connection and just counts the bytes it receives was enough. With enough sessions configured, the count always stopped at exactly one buffer’s worth.

The fix, #22645, has two parts. When the buffer fills up, bfdd now sends what’s in it before giving up. During the registration burst the socket can almost always take more, so normally this just works. And if a registration still fails, bfdd logs it and goes back to running that session in software, the same way it already did when a data plane disconnects. With 128 sessions and the byte-counting script, the old code registered 58 and silently lost the other 70. With the fix, all 128 got through.

Losing deletes on shutdown

With that fixed, I looked at what happens when bfdd shuts down. It puts a delete message for every data plane session into the same buffer, but by then its event loop has stopped, so nothing ever sends them. Then it closes the connection and throws the buffer away with the messages still in it.

What the data plane got depended on the version. Before my first fix, it got no deletes at all. After it, the new “send when full” code kicked in once halfway through, so exactly 58 deletes arrived and the rest were lost. The second case is actually more confusing for a data plane. Getting nothing looks like a crash, but getting 58 looks like a normal shutdown that happened to leave some sessions behind.

The fix for this one, #22692, makes bfdd write out whatever is left in the buffer right before it closes the connection on shutdown. It writes as much as the socket will take, and if anything is still left over, it logs how many bytes were lost instead of dropping them silently. With a real data plane holding 64 sessions, a clean bfdd restart used to leave 6 of them behind. With the fix, all 64 deletes arrived and nothing was left behind.

Both fixes are merged into FRR’s master branch. On released versions, if you have more than about 40 data plane sessions (up to 58, depending on your config), add them through vtysh after the data plane has connected, instead of putting them in frr.conf.