One session to sixty-four

Part 6 took session continuity and validation off the list from part 5. The next item on it is the one I had been quietly avoiding, because it invalidates every number published so far.

Everything measured up to this point, the whole bake-off, the final matrix, the FRR integration, has been one session. The maps and the daemon were sized for 64 from the hardening work onward, and I had written “multi-session capable” in the docs on the strength of that sizing. Sizing a table for 64 and testing one entry in it are not the same claim, and the distance between them turned out to be five bugs wide.

There is also a reason to expect trouble specifically here. The whole design rests on the peer’s packet being the clock: a session’s timing comes from its own inbound stream, processed in softirq. With one session that is a clean story. With sixty-four it is sixty-four streams sharing one receive path, and the question of whether they stay independent is exactly the sort of thing that is obvious in hindsight and untested in fact.

Four things that only worked because there was one of them

The receive path matched sessions on the peer’s IP address alone, and never recovered the local destination address at all. With one session that is sufficient and looks like a design. With two sessions to the same peer from different local addresses it is a coin flip. Matching now happens on the address pair, using IP_PKTINFO with recvmsg to recover the destination, and it only matters for bootstrap packets where your_disc is still zero, since established sessions demux on the discriminator as before.

Source ports were next. RFC 5881 says they should be unique per session, which I had ignored because with one session it is a distinction without a difference. Each slot now transmits from 49152 + slot, which meant moving userspace transmission to per-slot sockets, opened lazily and kept for the life of the process so that a reused slot reuses its socket and teardown needs no file descriptor bookkeeping.

The interesting one is map ownership, and it is the same class of mistake as the timestamp race in part 4: two things touching one value from two contexts, correct in every test until they are not.

The kernel had been clearing cfg->poll in place when the peer’s Final arrived, while userspace pushed the whole tx_cfg struct whenever it updated its mirror. A push landing at the wrong moment could resurrect a Poll sequence that had already finished. What makes it worth writing about is that the repair is not a lock:

/* Poll termination (RFC 5880 s6.8.4): peer answered our P with
 * F. tx_cfg is userspace-owned, so ack via kernel-owned
 * final_seq instead of clearing cfg->poll in place (a racing
 * userspace mirror push could resurrect the finished poll). */
if (cfg && cfg->poll && (bfd->flags & BFD_F_FINAL))
        st->final_seq = cfg->poll_seq;

tx_cfg became strictly userspace-owned. The kernel acknowledges the Final by writing final_seq into session_state, a map it already owns and userspace already polls, and the transmit path from part 4 sets the Poll bit only while final_seq and poll_seq disagree. That is the st->final_seq != cfg->poll_seq condition in the payload rebuild, which I glossed over at the time.

Every map value ended up with exactly one writer. The race is not guarded against, it is unrepresentable, which is a much better place to leave a concurrency bug than behind a lock you have to remember to take.

The last one is smaller but has the same shape. The receive path sets a session’s alive flag and the sweep clears it, and those run on different CPUs, so plain byte writes could emit the same liveness event twice. It became a __u32, because BPF atomics are 32 or 64-bit only, with both transition sites using __sync_val_compare_and_swap and emitting only when the swap wins.

Two sessions, then sixteen, then sixty-four

Two concurrent sessions established, slot ports visible bound and on the wire. Poll sequences driven through the new final_seq path by changing the transmit interval to 20ms and back while Up: both negotiated, zero down events, and zero residual Poll bits on the wire afterwards.

The test I cared about most was kill isolation, because one session dying must not disturb its neighbour and until then that had been an assumption rather than a measurement. FRR was stopped dead on one neighbour, with no farewell packet. The sweep declared that session down after 31.2ms of silence, inside the same 30 to 33ms envelope as always, while the other session’s uptime ran on uninterrupted.

Under two-session L3 stress, the same condition that flaps stock bfdd 44 times, both sessions held with zero flaps: p50 8.75 / p99 10.02 / max 14.7ms on one and p50 8.74 / p99 10.02 / max 12.6ms on the other. Two RX-clocked streams do not degrade each other, which is not obvious in advance, since they share a receive path.

Sixteen sessions next, eight per peer via secondary addresses: 0 flaps under L3, and a mass-kill of 8 detected at 30.5 to 32.8ms with no batch drift.

Sixty-four, and five bugs

The full-capacity run found five things. Four were mine, and none of them could have appeared earlier.

The slot ports I had just introduced collided with bfdd’s own port allocator, which is the sort of bug that only exists when two programs are running together at scale; moving the slot range up to 65472 and above resolved it. AdminDown turned out to have no exit, so a session could enter it and stay there. Sourcing from INADDR_ANY broke the peer address demux I had also just introduced, so per-slot sockets now bind their local address explicitly. And the mark-and-sweep reconciliation added for graceful restart in the previous milestone was unmarking incorrectly, tearing down sessions on a reconnect where the discriminators had not changed at all.

The fifth was not in my code. bfdd’s dataplane client silently and permanently strands sessions when the registration burst overflows its 8KB output buffer, which became FRRouting#22638, fixed in #22645. It is the first of three bugs this scale work eventually found in that one file.

The pattern that started with the UNIX socket bug held: every time the engine pushed FRR’s dataplane path somewhere it had not been pushed before, something fell out.

The design’s own boundary

The 32-session mass-kill was detected in 30.1 to 32.9ms with no drift, the same envelope as one session. Good.

But under L3 stress there was one correlated flap of 19 sessions at once, from a single RX-softirq stall.

That is worth stating plainly rather than burying, because it is the honest limit of the design at this scale. RX-clocked TX shares fate with softirq latency, as part 5 already said for one session. At 64 sessions that fate is correlated: a single stall in the receive path delays every session’s reply at once, so they do not fail independently. Recovery was autonomous and took about 50ms, but 19 sessions flapping together is a different operational event from one session flapping.

It is the quantified false-flap boundary of the architecture, and finding it was the point of the milestone. Part 1 argued that the maximum matters more than the percentile; this is the same argument turned on my own engine, and the per-session maximum is what exposed it.

What I took from it

“The maps are sized for 64” is not a claim about behaviour, and I had let it stand in the documentation as though it were. Four of the five bugs here were mine, all latent for weeks, all invisible to every test I had run, because a single session cannot exercise demux, cannot collide with a port allocator, and cannot race another session’s map writes.

The engine did not get faster in this milestone. It got true.

Next

IPv6, which turned out to contain the cleanest controlled experiment in the project, and which found the second and third bugs in the same bfdd file as the first.