bfdd dropping the data plane on a counters query

With 64 sessions running through xdp-bfd, I ran show bfd peers counters in FRR and it worked. I ran it again, and bfdd dropped its connection to xdp-bfd partway through. The third time it worked again, and from then on it kept alternating, failing every second run.

Something that fails every other time usually means something is left over from the previous run. In this case it turned out to be bfdd’s input buffer.

What was going wrong

When you run the counters command, bfdd asks the data plane for each session’s counters and reads the replies into an 8KB buffer. Normally bfdd clears out the replies it has already read to make room, but the function used for this command only did that after three reads in one go. The counters command reads one reply at a time, so the clean-up never happened. After a full run over 64 sessions, 5,120 bytes of the buffer were still taken up by replies bfdd had already dealt with.

On the next run there were only 3,072 bytes free, which is enough for 38 replies. Reply 39 filled the buffer exactly, and bfdd then asked the socket for zero bytes, because there was no room for more. A zero-byte read always returns zero, and in bfdd a read that returns zero means the other side closed the connection. So bfdd decided xdp-bfd had hung up and closed a perfectly good connection. Reconnecting cleared the buffer, which is why the run after that worked again.

That works out to 64 plus 39, so 103 replies per connection, and that’s exactly what I saw every time.

From the command line, the failing run still printed the full list of peers, just with old or zero counters for every session after the point where it broke, and with no error unless debug logging was on. Each failure also made bfdd reconnect and register every session again, so anything polling this command for monitoring would keep setting it off.

The fix

The fix clears out already-read replies every time that function runs, so the buffer can’t fill up with them. It also stops bfdd from ever asking for zero bytes. If the buffer is somehow still full after clearing it, a single message must be bigger than the whole buffer, so bfdd now logs that as an error and closes the connection properly.

My first version returned an error there without closing the connection, which would have left it stuck. Greptile, the automated reviewer on the PR, pointed that out, so I changed it to close the connection the same way bfdd does for its other errors.

Writing a test for it

Donald Sharp asked for a topotest that shows the fix works. The catch was that this bug only happens when bfdd is connected to an external data plane, and FRR’s tests had nothing that could act as one.

He pointed me at fpm_listener, a small fake receiver FRR already uses to test a similar connection in zebra, so I wrote bfd_dplane_listener the same way. It accepts bfdd’s connection, keeps track of the sessions bfdd sends it, answers counter requests and reports every session as up, without running any real BFD. The test runs the counters command 20 times over 8 peers and checks that every request got an answer without a reconnect. On the old code it fails with 103 of 160 counter requests were answered, which is right where the buffer runs out.

Getting that merged needed a few extra changes I hadn’t expected: a build macro the new program needed, packaging entries for both RPM and Debian, and a noreturn attribute because FRR’s OpenBSD build treats warnings as errors. The fix itself was only a few lines, and the test setup ended up being most of the pull request.

It’s merged as #22694, and the fake data plane is now part of FRR’s tree, so this part of bfdd can be tested in CI.