Echo mode, and instrumentation that cannot see its own failure

Echo was the feature I expected to be easy.

It is the one part of BFD where the packet belongs to neither endpoint’s control plane. The originator sends a packet addressed to itself, the neighbour’s forwarding plane loops it straight back without its BFD daemon ever seeing it, and the originator times the round trip. It tests the neighbour’s data path rather than its software, which is exactly the sort of thing a dataplane engine should be good at. Every other milestone had been about dragging work down into the kernel; this one looked like it belonged there already.

It turned out to be the only feature in the project that the architecture forbids outright, and working out why took longer than any of the code.

XDP cannot originate packets, and there is no way around it

XDP_TX is a verdict on a frame that has just arrived. That is precisely what makes the control path work here, the peer’s packet is the clock, but an echo has no inbound packet to clock off.

There is no timer callback that can transmit. No helper produces a second packet from one. The redirect helpers move the frame rather than copy it. The kernel-side escape is bpf_clone_redirect, which exists only for TC and lwt, and TC never sees anything at the echo cadence here because control packets are XDP_TX’d straight past it.

Putting echo transmission in the kernel would have meant moving the control bounce out of XDP into TC: skb allocation in the hot path, and a rewrite of the one mechanism this whole project rests on.

That is the trade in full. Gain a kernel-side echo originator, lose the RX-clocked control path that is the entire result. I declined it, and I want to be honest that declining took a while, because the alternative is admitting a feature does not fit and shipping something less than you intended. Writing down why took longer than the feature would have.

So the milestone split along the line the hardware draws

Answering a neighbour’s echo is a reflection, and a reflection has a packet in hand, so it lives entirely in XDP: MAC swap, TTL decrement, checksum recompute, XDP_TX, no session lookup, payload untouched. It is the same trick as the control path, applied to somebody else’s packet.

Originating one has no such luxury, so it runs from userspace over a raw socket. A raw socket rather than a normal UDP one, because a self-addressed packet through the latter is routed to loopback and never reaches the wire at all.

That split is not a compromise I am papering over. One half is a production capability and the other is a diagnostic, they have different reliability properties, and the documentation says so in those words.

The reflector’s argument is capability rather than speed. With ip_forward=0 the host stack discards a self-addressed echo as a martian, so a non-router host cannot participate in echo at all. The reflector lets it, without advertising the box as a forwarder.

The guard matters as much as the mechanism, and it is four lines:

if (iph->ttl != 255)      /* GTSM: single-hop echoes only */
        return XDP_PASS;
/* A classic echo is self-addressed to the originator. */
if (iph->saddr != iph->daddr)
        return XDP_PASS;
/* Reflect only for a peer of an echo-active session; otherwise
 * this is an arbitrary 3785 packet (amplification vector). */
struct bfd_addr esrc;
key_set_v4(&esrc, iph->saddr);
if (!bpf_map_lookup_elem(&echo_peers, &esrc))
        return XDP_PASS;

Without that last lookup this is a reflector for anyone who sends a self-addressed packet to port 3785, which is an amplification primitive with my name on it. The map is written only for peers of sessions that have echo enabled, so the box answers exactly the neighbours that asked and nobody else. XDP_PASS rather than XDP_DROP on the rejects, because an unrecognised packet on 3785 is not necessarily hostile and the stack can have it.

Two spec errors, surfaced the moment there was code

The design document proposed demuxing returned echoes by source address and port, following the unaffiliated-echo draft. That cannot work. The return is still addressed to our own local address, which names no session, and several sessions may share one.

The discriminator written into the payload is the only thing that identifies the session, and classic echo never loops Your Discriminator, so it survives the trip untouched. Which makes the return path a lookup on the payload rather than on the headers:

__u32 ed = bpf_ntohl(eb->my_disc);
struct session_key *ek = bpf_map_lookup_elem(&echo_disc, &ed);
if (!ek)
        return XDP_PASS;
struct session_state *es = bpf_map_lookup_elem(&bfd_sessions, ek);
if (!es)
        return XDP_PASS;
es->echo_last_seen_ns = bpf_ktime_get_ns();

Two lookups where the address would normally have given it in one: the discriminator names a session key, the key names the session. That indirection is the whole cost of the addresses being useless here.

The second was subtler and cost a debugging round. Every return was being dropped before the echo branch was reached, because the v4 parser applies GTSM first and returns arrive at TTL 254 by definition. The symptom was the reject counter climbing at exactly the echo rate while the echo counter stayed flat, which is the kind of thing a counter tells you and a log does not.

Detection is advisory, permanently

The sweep marks each echo-active session alive or not and reports it, but never feeds the session state machine.

With userspace transmission, a local scheduling stall is indistinguishable from a path failure: echoes stop leaving, returns stop arriving, the timestamp goes stale. Wiring that into the state machine would convert our own scheduling delay into a teardown, which is the exact failure this engine was built to avoid.

Verified from both sides: disabling forwarding on the neighbour stops the returns, loss climbs one for one, echo liveness flips, and all 64 control sessions stay up.

The sharpest lesson in the milestone

Under load, echo transmission stalled for 2.6 seconds at a time. The loss counter read zero throughout. The liveness flag read healthy.

Both were correct. Both were useless.

Loss only increments when an echo is outstanding as the next one falls due, and during a total stall nothing ever falls due. The liveness figure is printed on transmit, and transmit is what stopped.

Instrumentation driven by the thing being measured cannot observe that thing failing.

A windowed inter-send gap, sampled independently, showed it immediately: 13ms idle against a 10ms interval, already 30% over budget with nothing running.

And a lesson about my own habits

Measuring the cost of the always-on additions produced a second lesson.

Flap count had been the working metric since the bake-off. At 64 sessions it turned out to vary from 0 to 20 across runs of identical reference code, which is wider than any effect worth measuring.

A full day went into chasing that variance. The cause was a debug bfd peer line persisted in the neighbour’s config file, surviving every restart, contaminating every run including the supposed baselines. Several confident conclusions were drawn and retracted in the process, each overturned by the next run.

The metric was replaced with the per-session maximum transmit gap, which yields 64 numbers per run instead of one rare event. The answer became a bound rather than a claim: the reference build’s own median spans 22.3 to 27.1ms across two runs, the echo build sits at 23.2 inside that spread, so the additions cost less than roughly 5ms of median. Which is not the same as zero, and the docs say that too.

The upstream bug had two halves facing each other

RFC 5880 requires that echo not be transmitted faster than the neighbour advertises it can receive. For an offloaded session bfdd never performs that negotiation, which I wrote up separately.

But that fix has no input unless the dataplane reports the neighbour’s advertisement upward, and this engine was hardcoding it to zero. The engine also had the mirror of the same bug outbound: it advertised its own echo receive interval as zero, which the RFC defines as “cannot receive echo packets”, so no conforming neighbour would ever have echoed at it.

Which explains, in retrospect, why every reflector test until then had used a hand-built frame.

With both halves fixed the chain closes and can be watched end to end: the neighbour advertises 200ms, the daemon negotiates it against a locally configured 50ms, and the wire cadence moves from 50 to 200. And the reflector finally answered a real implementation rather than a scapy script: 433 echoes, 433 reflected, 12 microseconds minimum turnaround, on a host that would have dropped every one of them as a martian.

Next

Multihop, which looks like the smallest milestone and taught the same lesson the project keeps teaching.