How 8KB of buffer silently lost BFD sessions

While building xdp-bfd, an offload engine that runs the BFD fast path in XDP, I needed FRR’s bfdd to hand session lifecycle over to my dataplane. This turned up during the 64-session scale run. Below roughly twenty peers it worked. Above it, sessions started disappearing, and nothing in the logs said why.

What the dataplane protocol does at connect time

When bfdd connects to a distributed-BFD dataplane, it walks every configured session and enqueues one registration message per session into the dataplane client’s output buffer. That buffer is 8KB, and the whole walk happens in a single event-loop pass, before the write event ever gets a chance to run.

At roughly 140 bytes per message, 8KB holds about 58 registrations. With 64 sessions configured, the last six never made it.

Why the failure was invisible

A failed enqueue was not the end of it. For each session past the limit, bfdd had already disabled the software implementation, on the assumption that the dataplane was taking over. It then reset bs->bdc to NULL when registration failed.

That combination is the actual bug. The session was:

• never delivered to the dataplane, so the dataplane did not know it existed
• no longer handled in software, because that path had been switched off
• left with a null dataplane context, so every subsequent operation on it became a silent no-op

The session still appeared in show bfd peers. It simply never ran anywhere. No warning was logged, because from the enqueue path’s point of view a full buffer is an ordinary condition.

Reproducing it without a dataplane

The useful part of the investigation was making it reproducible for someone who does not have an offload engine sitting around. A dataplane does not have to do anything for this bug to show up: it only has to accept a TCP connection and count bytes.

A socket that accepts the connection and reads without ever replying is enough. Configure N sessions, connect, and count the bytes that arrive. The count stops at exactly one buffer’s worth, every time, and the cutoff is deterministic.

That mattered more than the fix. A bug report with a reproducer that needs a specific piece of unreleased software is a bug report nobody can act on.

The fix

Two commits, each useful on its own.

The first makes the enqueue path try to flush before giving up. During the registration burst the socket is almost always writable, and the packet capture showed zero TCP backpressure, so the common case now simply succeeds:

if (buflen > STREAM_WRITEABLE(bdc->outbuf) &&
    !(bdc->client && bdc->connecting)) {
        /*
         * On socket failure or close the context is freed by
         * the flush. The buffer is non empty here, so a zero
         * return means exactly that: don't touch `bdc` again.
         */
        if (bfd_dplane_flush(bdc) == 0)
                return -1;
}

There is a subtlety in the error path. A zero return from the flush with a non-empty buffer means the context was freed on a socket error, so it must not be touched again. Clients that are still connecting are skipped entirely, because flushing would spin on EAGAIN.

A review bot flagged this as possibly leaving queued messages with no scheduled writer, and a maintainer asked whether it needed fixing. It does not, and being able to say exactly why is the useful part: bfd_dplane_flush() loops on STREAM_READABLE() and treats EAGAIN, EWOULDBLOCK, and EINTR as retry, so it cannot return with data still buffered. Its only two exits are a fully drained buffer, at which point cancelling the write event is correct because there is nothing left to write, or freeing the context on socket error. After a successful flush the whole buffer is writeable, so the space check that follows can only fail for a single message larger than 8KB. The largest bfddp message I measured is 203 bytes.

The second commit stops a failed registration from stranding the session. On failure it logs a warning and returns the session to the software implementation, which mirrors the recovery that already existed when a dataplane detaches. Software BFD is worse than offloaded BFD, but it is dramatically better than nothing at all, which is what the session had before.

Result

With 128 sessions configured and a byte-counting sink, the unpatched build delivers exactly 58 registrations, 8120 bytes, and silently loses the other 70. The patched build delivers all 128, with the byte count exact and Output full events at zero.

End to end against the engine, the reconnect burst that previously registered 40 of 64 sessions now registers all 64.

The fix is merged to FRR master as #22645. It also changed the shape of a second bug on the shutdown path, which I wrote up separately. Packaged releases up to 10.5.1 still carry the bug, so if you are running distributed BFD with more than about twenty peers on a released build, add peers through vtysh after the dataplane connects rather than putting them in frr.conf.

What I took from it

The bug was not in the buffer size. 8KB is a reasonable buffer. The bug was that a resource limit was being treated as an unremarkable condition, in a code path where crossing it meant a session belonged to nobody.

Failure paths that quietly reduce functionality are worse than failure paths that crash, because a crash gets reported. This one produced a router that believed it was monitoring a link it was not monitoring, which is precisely the thing BFD exists to prevent.