IPv6, and the bake-off argument in miniature

Part 7 got the engine to sixty-four sessions. IPv6 was the next item on the list, and on the face of it the least interesting one: parse a different header, branch where the protocols differ, done.

There was a question buried in it though, and it had been nagging since the bake-off.

The entire argument for this project rests on one claim: that moving transmission into the kernel is what buys the resilience, rather than any of the other things that changed along the way. That claim came from comparing different programs on different runs. A skeptic could reasonably say the XDP build was better for some other reason, or that the load was not identical, or that six weeks of incidental fixes did the work.

Adding a second address family creates the chance to answer that properly. If v6 transmits from userspace while v4 rides the kernel path, on the same engine, on the same box, at the same instant, under the same load, then everything is controlled except the one variable in question.

So the milestone had two jobs: make the engine dual-stack, and settle whether the kernel reply is actually load-bearing.

The structural move came first, and alone

The session key widened from two __be32 fields to two 16-byte addresses, with v4 stored v4-mapped (::ffff:a.b.c.d), so both families share one hash map, one slot-socket pool, and the same fast path with no possibility of key collision.

That step contains no v6 code at all. It exists to prove the layout change against the live v4 session before anything interesting is built on it. A 10,000-packet regression run, RX-clocked lockstep intact through the new key.

Each of the five landing steps carried a v4 regression check, so the shared code never regressed the working path.

The v6 parse path

A branch on ethertype: fixed 40-byte ipv6hdr, GTSM as hop_limit 255, and a deliberate refusal to walk extension headers. UDP hidden behind an extension chain never reaches a session, while non-UDP first headers are XDP_PASSed to the stack so ICMPv6 neighbour discovery survives an attached program.

The kernel reply is family-branched where the protocols actually differ. v4 keeps udp->check = 0 and the IP-checksum trim recompute. v6 has no IP checksum but a mandatory UDP one, so the echo recomputes it over the swapped pseudo-header as a 34-word fold.

The ordering constraint is the part worth writing down: the fold must complete before bpf_xdp_adjust_tail, which invalidates every packet pointer, and must never read past the 24 BFD bytes that survive the trim.

if (ip6) {
        __u32 csum = 0;
        __u16 *w = (__u16 *)&ip6->saddr;
        for (int i = 0; i < 16; i++)   /* saddr + daddr */
                csum += w[i];
        csum += udp->len;              /* pseudo length */
        csum += bpf_htons(IPPROTO_UDP);
        w = (__u16 *)udp;              /* UDP hdr, check == 0 */
        for (int i = 0; i < 4; i++)
                csum += w[i];
        w = (__u16 *)bfd;              /* 24-byte payload */
        for (int i = 0; i < 12; i++)
                csum += w[i];
        csum = (csum & 0xffff) + (csum >> 16);
        csum = (csum & 0xffff) + (csum >> 16);
        __u16 c = ~csum & 0xffff;
        udp->check = c ? c : 0xffff;   /* RFC 768: 0 -> 0xffff */
}

if (excess > 0 && bpf_xdp_adjust_tail(ctx, -excess))
        return XDP_DROP;

Thirty-four words, fixed count, every pointer bounds-proven above so the verifier accepts it. The correctness of the whole block lives in the order of those two statements: fold first, trim second. Reverse them and you are folding over freed pointers, which the verifier will not catch because the pointers were valid when it checked them.

The oversized-frame case, 16 trailing bytes echoed back as a valid 24-byte reply with a recomputed checksum and a patched payload_len, was verified against injected traffic rather than just reasoned about.

The experiment

Here is the answer to the question this milestone opened with.

For a window during the landing, the engine was genuinely dual-stack in parsing but only v4 had the kernel reply. That window is the experiment. Both families ran the L3+L4 ladder concurrently at 3x300ms: v4 on kernel RX-clocked TX, v6 transmitting from userspace at RFC pacing.

• v4: 0 flaps.
• v6: 19 flaps, TX gaps to 1900ms.

The RT throttle’s ~950ms per second starvation exceeds the 900ms detect budget once per cycle, each flap recovering autonomously in about 3ms.

Same run. Same instant. Same engine. Same host, same load, same code except for one thing: which side of the kernel boundary the transmit clock lives on.

With the kernel reply enabled, the rerun of the same ladder: v6 0 flaps, TX max 300.5ms, indistinguishable from v4.

That settles it. The resilience is not an accident of the rewrite, or of six weeks of incidental fixes, or of a favourable run. It is the kernel reply, and here is the same engine failing and then not failing with only that changed.

You rarely get a controlled experiment that clean outside a textbook, and I got it by accident, from the ordinary business of landing a feature in stages.

Scale, and closing an upstream loop

The v6 spoof harness repeated the m5 validation from a third host: wrong your_disc and hop_limit 64 both XDP_DROPped and counted, correct credentials passed.

The mixed-family scale run closed the milestone: 32 v4 plus 32 v6 at the 64-session cap, 3x10ms timers, through the same L3+L4 ladder. Zero flaps in either family, zero wire transitions, and per-slot maximum TX gaps sitting in a single 13 to 14.5ms band with the two families statistically indistinguishable.

The v4-only baseline had shown correlated stall flaps at this session count. The dual-stack engine beats its own earlier result.

The scale work also found the 8KB output buffer bug in bfdd that stranded sessions silently, and the validation session that followed found two more in the same file. All three were invisible without a real dataplane at scale, which is the recurring theme of this project’s relationship with FRR.

Next

Echo mode, where the feature I most wanted to put in the kernel turned out to be the one XDP fundamentally cannot do.