Under attack

Up to part 9, the thing trying to break xdp-bfd was always its own box being too busy. The other way to break BFD is from the wire.

Who I was defending against

The person I planned for is anyone on the same link who can see every packet and send whatever they like, as fast as the link goes, but doesn’t have the auth key. Plain BFD can’t stop them, since everything is sent in the clear and they can just copy a real packet. What I could do was make sure authenticated sessions can’t be faked, and that a flood aimed at one session doesn’t take the others down with it.

Floods on the VMs

I started on the 64-session VM mesh in September. The first flood I tried was junk sent to the BFD port. XDP was passing it up to userspace, where it filled the socket queue and pushed real packets out, and three sessions went down. So now XDP drops anything that isn’t a valid packet for a session it knows, and with the same flood all 64 stayed up.

Signed packets were the other worry, because checking a signature means running the HMAC, and that’s expensive. So each authenticated session got a budget of failures: after eight bad signatures within one detection time, its packets get dropped without being checked until that time is up. Half a million bad packets at one session only got hashed 248 times. That session went down, and the other 63 didn’t notice.

Floods at 1024 sessions

Later in September, at 1024 sessions, three more floods got through, all using packets that looked real enough that XDP couldn’t just drop them. One named a real session from an unknown address. That has to go to userspace, because a peer’s address can legitimately change, and it kept the engine busy enough that 221 sessions went down. Another spoofed echo packets, and bouncing them all overflowed the transmit queue our real replies go out on. The last one used a real session but changed its timers in every packet, so XDP sent a reply for each one.

Each of those got its own rate limit, and now only the session being flooded can go down.

Floods on real hardware

On the real machines from part 9, I flooded the xdp-bfd box at about 1.2 million packets a second, one kind of attack at a time. Only one of them took down sessions it wasn’t aimed at, between 37 and 48 of them. That attack filled the switch port to the xdp-bfd box. The switch puts packets marked CS6 in a priority queue, and xdp-bfd and bfdd mark their BFD packets that way, except bfdd’s IPv6 echo, which wasn’t marked. So those echoes waited behind the flood and got dropped. I fixed the marking in FRR (#23495), and after that no other sessions went down.

The worst hour I could set up

Then I tried everything at once, for an hour: a flood at full line rate switching between five kinds of attack, while the stress on the box switched between a priority-99 hog, CPU and timer load, and filling memory until it swapped. That ended with 165,028 down events.

The engine was fine the whole time. It was the box’s receive path that gave out. Junk that wasn’t BFD was going up to the kernel on the same CPU that handled BFD, and on this NIC one stuck receive queue held up the other three. So I added an option to send the non-BFD traffic to other CPUs, turned off Ethernet flow control so the driver drops per queue instead, and spread incoming UDP across the queues by port. With those, the same hour came down to 4,148.

A neighbour that holds the key

The last case was a neighbour that does have the key and sends valid packets on purpose. The failure budget never kicks in for those, since every signature checks out, so each one cost a full HMAC, almost 40 times more than a packet that gets dropped early. So I added a second limit in front of the HMAC that counts every check, and allows twice as many as a real peer is ever allowed to send. A check that fails gives its slot back, so someone without the key can’t use up the real peer’s share. On normal traffic it never kicked in once.

Where it’s at

On real hardware, xdp-bfd now holds all 1022 sessions through the same load that flapped stock bfdd 18,910 times. It does everything bfdd can hand a data plane: RFC 5880, 5881 and 5883, with echo, demand mode and authentication. It does keyed SHA1 the way bfdd does it, as an HMAC, and my issue about that in FRR is still open.

Two xdp-bfd engines still can’t face each other, for the reason in part 2. S-BFD and micro-BFD are what I want to do next, once the auth work is finished on the FRR side too. On FRR, 11 of my fixes are merged but none are in a release yet, and six more are still open.

Looking back

When I went back through the bugs while writing this series, almost every one was found in a packet capture and almost none in a log. The daemon in part 1 logged zero timeouts while its session was dying. The bug in part 2 showed up as one missing packet that the logs never mentioned. The multihop test in part 5 passed while sending to the wrong port. Each time, the logs looked fine and the capture showed what was really going on.

Thanks for reading all ten parts. The code is at github.com/w453y/xdp-bfd, with the captures on its docs branch, and if you liked it, a star would mean a lot.