Abdul Wasey: Writing

Building xdp-bfd, part 10: under attack

w453y.me@gmail.com (Abdul Wasey) — Sat, 03 Oct 2026 00:00:00 GMT

Up to part 9, the thing trying to break xdp-bfd was always its own box being too busy. The other way to break BFD is from the wire.

Who I was defending against

The person I planned for is anyone on the same link who can see every packet and send whatever they like, as fast as the link goes, but doesn’t have the auth key. Plain BFD can’t stop them, since everything is sent in the clear and they can just copy a real packet. What I could do was make sure authenticated sessions can’t be faked, and that a flood aimed at one session doesn’t take the others down with it.

Floods on the VMs

I started on the 64-session VM mesh in September. The first flood I tried was junk sent to the BFD port. XDP was passing it up to userspace, where it filled the socket queue and pushed real packets out, and three sessions went down. So now XDP drops anything that isn’t a valid packet for a session it knows, and with the same flood all 64 stayed up.

Signed packets were the other worry, because checking a signature means running the HMAC, and that’s expensive. So each authenticated session got a budget of failures: after eight bad signatures within one detection time, its packets get dropped without being checked until that time is up. Half a million bad packets at one session only got hashed 248 times. That session went down, and the other 63 didn’t notice.

Floods at 1024 sessions

Later in September, at 1024 sessions, three more floods got through, all using packets that looked real enough that XDP couldn’t just drop them. One named a real session from an unknown address. That has to go to userspace, because a peer’s address can legitimately change, and it kept the engine busy enough that 221 sessions went down. Another spoofed echo packets, and bouncing them all overflowed the transmit queue our real replies go out on. The last one used a real session but changed its timers in every packet, so XDP sent a reply for each one.

Each of those got its own rate limit, and now only the session being flooded can go down.

Floods on real hardware

On the real machines from part 9, I flooded the xdp-bfd box at about 1.2 million packets a second, one kind of attack at a time. Only one of them took down sessions it wasn’t aimed at, between 37 and 48 of them. That attack filled the switch port to the xdp-bfd box. The switch puts packets marked CS6 in a priority queue, and xdp-bfd and bfdd mark their BFD packets that way, except bfdd’s IPv6 echo, which wasn’t marked. So those echoes waited behind the flood and got dropped. I fixed the marking in FRR (#23495), and after that no other sessions went down.

The worst hour I could set up

Then I tried everything at once, for an hour: a flood at full line rate switching between five kinds of attack, while the stress on the box switched between a priority-99 hog, CPU and timer load, and filling memory until it swapped. That ended with 165,028 down events.

The engine was fine the whole time. It was the box’s receive path that gave out. Junk that wasn’t BFD was going up to the kernel on the same CPU that handled BFD, and on this NIC one stuck receive queue held up the other three. So I added an option to send the non-BFD traffic to other CPUs, turned off Ethernet flow control so the driver drops per queue instead, and spread incoming UDP across the queues by port. With those, the same hour came down to 4,148.

A neighbour that holds the key

The last case was a neighbour that does have the key and sends valid packets on purpose. The failure budget never kicks in for those, since every signature checks out, so each one cost a full HMAC, almost 40 times more than a packet that gets dropped early. So I added a second limit in front of the HMAC that counts every check, and allows twice as many as a real peer is ever allowed to send. A check that fails gives its slot back, so someone without the key can’t use up the real peer’s share. On normal traffic it never kicked in once.

Where it’s at

On real hardware, xdp-bfd now holds all 1022 sessions through the same load that flapped stock bfdd 18,910 times. It does everything bfdd can hand a data plane: RFC 5880, 5881 and 5883, with echo, demand mode and authentication. It does keyed SHA1 the way bfdd does it, as an HMAC, and my issue about that in FRR is still open.

Two xdp-bfd engines still can’t face each other, for the reason in part 2. S-BFD and micro-BFD are what I want to do next, once the auth work is finished on the FRR side too. On FRR, 11 of my fixes are merged but none are in a release yet, and six more are still open.

Looking back

When I went back through the bugs while writing this series, almost every one was found in a packet capture and almost none in a log. The daemon in part 1 logged zero timeouts while its session was dying. The bug in part 2 showed up as one missing packet that the logs never mentioned. The multihop test in part 5 passed while sending to the wrong port. Each time, the logs looked fine and the capture showed what was really going on.

Thanks for reading all ten parts. The code is at github.com/w453y/xdp-bfd, with the captures on its docs branch, and if you liked it, a star would mean a lot.

Building xdp-bfd, part 9: running it on real hardware

w453y.me@gmail.com (Abdul Wasey) — Wed, 30 Sep 2026 00:00:00 GMT

Every number up to part 8 came from Proxmox VMs, with the hypervisor bridge as the thing watching the wire. I wanted to know if any of it held up on real machines, with real NICs and no hypervisor in the way.

The setup

I used four machines on their own switch: one running xdp-bfd with FRR on top, one FRR peer doing all its sessions in software, one to generate load, and one that only captures. The first three are Intel i5-12500 boxes with Intel I210 NICs, with XDP running in native mode inside the NIC driver, on Ubuntu 26.04 with kernel 7.0.

Without a hypervisor, I needed something else to watch the wire from outside. The switch copies everything the xdp-bfd box and the peer send over to the capture box, which timestamps each packet on its NIC. I checked that against a capture on the peer itself, and the two agreed to within 0.2%.

I moved the whole 1024-session test mesh over as it was, with every kind of session from the earlier parts, plus 40 at 3x10ms. The stress ladder was the one from part 1, scaled up to 12 threads.

Stock bfdd first

I started with plain FRR bfdd doing all 1022 sessions itself. The timer load (L3) didn’t hurt it at all. The real-time load (L4) did: 18,910 flaps across 957 of the 1022 sessions, including the ones with 1 second timers. bfdd stalled for up to 2.5 seconds at a time, and it kept flapping for a while after the stress had stopped.

Then I ran xdp-bfd exactly as it had run on the VMs. It did a lot better, but it wasn’t zero: 698 flaps across 213 sessions. On the VMs the ladder had come out clean, but those runs had 64 sessions, not 1022.

Three things the VMs hadn’t shown

The first one was the dead-man gate from part 7, which stops XDP answering once userspace has been silent for longer than a bound, which was 1 second. But under real-time load, Linux lets real-time tasks use 95% of each one-second period, and normal tasks only get the last sliver of it. A normal task stuck behind real-time hogs ends up running about once a second, so the gaps in the userspace loop came out between 1.05 and 2.1 seconds. The check was taking down sessions that were fine. The default bound is 3 seconds now.

The second was in demand mode. Those sessions were timing out their own Poll even after the peer had answered it. That turned out to have three separate causes, and I fixed all of them.

The third was in the package from part 8. Its systemd unit ran the engine at real-time priority 10, below the priority-50 load, so under L4 it got no CPU at all and 957 sessions went down, once each. It runs at 60 now.

With those three fixed, the ladder came out at 0 flaps on all 1022 sessions. The 3x10ms sessions had a largest gap of 20.2ms against their 30ms budget.

Upgrading under load

Part 8 said systemctl reload and package upgrades hand over without dropping anything. On the real machines I found that neither did. Reload never worked at all, because of a check on the process name, and a package upgrade stopped the engine before installing the new one.

After fixing both, I installed a new package with dpkg -i over a running engine with 1024 sessions. The handover took 10 to 31ms, and the peer saw 0 sessions go down. The packaged engine also went through the whole ladder with 0 flaps, using about 2% of one core for 1024 sessions.

Where the replies were waiting

Next I looked at how long replies took. Inside XDP, building a reply takes about 1.5 microseconds. On the wire the replies were leaving 118 microseconds after the peer’s packet arrived, and 250 at p99.

My first guesses were interrupt moderation and CPU sleep states, but turning either of them off on its own didn’t change much. It turned out to be PCIe power saving (ASPM) on the link to the NIC, which puts the link to sleep between packets. With ASPM off it dropped to 33 microseconds. On these boxes the firmware owns ASPM, so it has to be turned off in the BIOS or with setpci. BFD timers are in milliseconds, so none of this changes detection.

Priority 60 isn’t enough either

The engine was at priority 60 now, but that only works as long as nothing runs above it. That’s the same problem I had with priority 90 back in part 2. So I tried a priority-99 hog against the engine at 60, and got 959 down events.

Linux has another scheduling class, SCHED_DEADLINE, where a task reserves a slice of CPU time in every period, and the kernel runs it ahead of every real-time priority. So the engine now reserves 1ms in every 10ms. Against the same priority-99 hog, that gave 0 flaps. Even 0.2ms was enough. If the kernel refuses the reservation, for example because the service is pinned to only some of the CPUs, the engine logs why and keeps running at priority 60.

Next is what happens when the problem isn’t a busy box but someone on the wire trying to break it: part 10.

My fixes to FRR’s BFD daemon

w453y.me@gmail.com (Abdul Wasey) — Mon, 28 Sep 2026 00:00:00 GMT

While building xdp-bfd, I ran FRR’s bfdd with xdp-bfd as its data plane. Not many people use that part of bfdd, and I kept running into bugs in it. Later on, adding demand mode and authentication turned up a few more. So far eleven of my fixes have been merged, and this post goes through them in roughly the order I hit them.

All of them are on FRR’s master branch, and none are in a release yet. Where there’s a workaround for released versions, I’ve put it in. Also, if you want to use a data plane at all, you need FRR 10.1.2 or newer, because older versions keep dropping the connection to it.

Data plane plumbing

The UNIX socket transport never worked

#22621. This was the very first one. bfdd and xdp-bfd were running on the same box, so I wanted them to talk over a UNIX socket, but every connect failed with EINVAL. The same setup over TCP worked fine.

strace showed the socket path was right, but the address length bfdd passed in was 112 bytes, and a UNIX socket address is only 110. It turns out bfdd keeps addresses in a union big enough for every address type, and it was passing the size of the whole union instead of the size of the address inside it. TCP sockets don’t mind a length that’s too long, but UNIX sockets check it and refuse it. Going by the git history, as far as I can tell unixc: never worked on Linux. The fix passes the real length. On released versions, use TCP.

Sessions lost to an 8KB buffer, twice

#22645 and #22692. When I went from 16 sessions to 64, bfdd silently lost every session after the 40th when it connected to xdp-bfd. On shutdown it had the same problem the other way round and lost the last few delete messages. This one has its own post.

A zero-byte read that looked like a closed socket

#22694. Running show bfd peers counters dropped the connection to xdp-bfd every second time I ran it. This one has its own post too.

Echo

Echo interval

#22805. BFD peers tell each other how often they’re willing to receive echo packets, and you’re not supposed to send faster than that. While I was adding echo to xdp-bfd, I had a peer asking for echoes every 200ms, and bfdd was telling xdp-bfd to send them every 50ms, which was just my local setting.

bfdd does work out the right interval, but only in the code where it handles packets itself. For a session handed to a data plane that code never runs, and the one other place that could have done it skips data plane sessions on purpose. The confusing part was that bfdd had the peer’s 200ms stored, and show bfd peer displayed it, so everything looked right in the CLI. I only noticed because I was measuring how often echoes actually went out. The fix uses whichever of the two intervals is slower, and tells the data plane whenever that changes.

IPv6 echo from the wrong address

#22920. With IPv6, turning on echo mode could take a perfectly good session down.

An earlier change in bfdd made it only bounce back echo packets whose source address matches a session it knows, which is a sensible safety check. But bfdd’s own IPv6 echo socket isn’t tied to an address, so the kernel picks the source for it. If the interface has more than one address in the same network, the kernel can pick one the peer doesn’t know about. The peer drops the echo, and the session goes down. The fix sets the source address on each echo packet. It has to be per packet, because the same socket sends echoes for every session.

Writing the test took longer than the fix. I needed Linux to pick the wrong address on purpose, and adding a second address didn’t do it, because Linux kept using the older one. What worked in the end was marking the right address as deprecated, so the kernel avoids it. The test also checks that the kernel really did pick the wrong address, so it can’t quietly pass if that behaviour ever changes.

Demand mode

#23023. Demand mode lets two BFD peers stop sending regular packets once both sides are up, and only check the path when they need to. bfdd had the option defined, but it didn’t do anything. The flag was set to 0 in three places and never to 1, and there were TODO: demand mode comments sitting in the code. I talked the design through with Rafael Zalamena first, then implemented it, including passing it through to data planes.

Zero discriminator checked against the wrong state

#23284. While I was testing demand mode, I restarted one side and it never came back. The side that stayed up just kept refusing the restarted peer’s packets.

A peer that has restarted sends a zero where the other side’s discriminator should be, and the RFC says to accept that as long as the packet says the peer is Down. bfdd was checking whether its own session was Down instead. Normally nobody notices, because the detection timer runs out, the session goes Down, and the next packet gets through. In demand mode that timer doesn’t run, so the session just stayed stuck. The fix checks the state in the packet, like the RFC says.

Auth

The replay check never ran

#23281. With keyed SHA1, every packet carries a sequence number, and bfdd is supposed to reject old or repeated ones so nobody can replay captured packets. It turns out that check never ran. bfdd decided whether it had seen a sequence number yet by checking if the last one was zero, and the only code that ever updated it was inside that same check. So it stayed at zero forever, and every packet was accepted, replays included.

FRR’s existing auth test never caught this, because two well-behaved BFD implementations never send a bad sequence number. So the test I added uses a small BFD speaker that sends its sequence numbers out of order on purpose.

No usable key meant no auth at all

#23282. With a 20-byte SHA1 key, bfdd sent plain unauthenticated packets, while show bfd peer still showed the keychain.

bfdd limited every key to 16 bytes, which is the limit for simple passwords, but SHA1 keys can be up to 20. With a longer key, bfdd found no usable key and quietly fell back to no authentication. A key with no algorithm set did the same thing. Now the length limit depends on the algorithm, and if a keychain has no usable key, the session doesn’t come up at all instead of running without auth. Anyone who was running without auth by accident will see their session go down after upgrading, so I pointed that out in the PR.

Auth keys never reached the data plane

#23331. Once xdp-bfd could sign packets itself, I found that bfdd had no way to give it the keys. A session with authentication got handed to the data plane and went out with no authentication at all, while show bfd peer still said auth was on. Now bfdd sends the keys that are usable now and the ones that will be later, with the times each one is valid, so the data plane can switch keys by itself. Part 6 of the xdp-bfd series has more on this.

Still in review

These six were still open in early October. Five of them are approved and waiting to be merged; #23465 hasn’t been reviewed yet.

#23453: don’t send an echo interval to the data plane when the peer says it can’t receive echo
#23463: let a data plane tell bfdd what it supports, so authenticated sessions only go to one that can handle them
#23464: forget the peer’s discriminator after a detection time, which fixes a case where a restarted peer stays down
#23465: don’t open echo sockets when a data plane handles the packets
#23495: send IPv6 echo with the same traffic class as the rest of BFD, so a congested switch doesn’t drop it first. Found in the flood tests in part 10.
#23496: open the echo sockets for a session that’s enabled after bfdd starts. Since a change in FRR 10.7.0, echo sessions set up that way got no sockets and kept flapping.

Thanks to Rafael Zalamena, Russ White, Donald Sharp and the other FRR reviewers for reviewing the ones that are merged.

Building xdp-bfd, part 8: shipping it

w453y.me@gmail.com (Abdul Wasey) — Tue, 22 Sep 2026 00:00:00 GMT

Until September, the only place xdp-bfd had ever run was my own test setup, built from source. I wanted it to be something someone else could install next to their FRR and just run.

Packages

I made packages for Debian, Ubuntu and Fedora, and I wanted them to run without root. So I went through what the engine actually needs, and it came down to four capabilities. Two are for loading the XDP program, one is for attaching it to an interface, and the last is for the raw socket that echo packets go out on. The service runs as its own user with just those four.

Older distros couldn’t even build it, because it needed clang 20. The problem was a 32-bit atomic operation that older clang versions can’t compile for BPF. Making those atomics 64-bit fixed it, and it builds with clang 17 now.

Which kernels

For kernels, I stopped going by version numbers and just checked whether the XDP program loads. Most kernels from 6.1 up are fine. AlmaLinux 9 says it’s 5.14 and still loads it, because Red Hat backports the BPF parts of newer kernels. Debian 11 and the stock kernel on Ubuntu 22.04 don’t.

Which FRR

The FRR version mattered more, so I ran the FRR tests against 13 releases and master. Everything older than 10.1.2 connected to xdp-bfd and then dropped the connection about once a second, and no session ever came up. bfdd was going into its reconnect code right after connecting and closing the socket. That was fixed upstream in 10.1.2, so that’s the oldest version that works.

Even on newer releases, two of the bugs from my FRR post still apply, because the fixes are only on master. The UNIX socket doesn’t work, so it has to be TCP, and bfdd only gets about 58 sessions across to xdp-bfd each time it connects.

VyOS

Then I tried it on VyOS, which comes with its own FRR. Sessions came up on the fast path, except the ones configured without a local address. bfdd was handing those to xdp-bfd with the local address left blank. XDP looks sessions up by both addresses, so nothing ever matched, and the peer’s packets got dropped as unknown. Now xdp-bfd fills in the address the kernel would use to reach that peer.

Restarts and the README

The restart handover from part 6 went into the package too, so systemctl reload and package upgrades would hand over without dropping anything. Both passed CI. Neither actually worked on a real install, which I only found out in part 9.

The README had grown to 786 words by then. I cut it down to 266, just what xdp-bfd is and how to build and run it, and moved everything else to the wiki.

Next, running it on real hardware.

Interviewing with Jane Street for a Linux Engineer role

w453y.me@gmail.com (Abdul Wasey) — Wed, 16 Sep 2026 00:00:00 GMT

In August I interviewed with Jane Street for a Linux Engineer role, and I didn’t get it. They asked me not to share anything about the interview itself, so this post doesn’t: no questions, no tasks, nothing from inside it. What I can share is the ground I made sure I had covered going in, because none of that is theirs. It’s the same ground any Linux, systems or infrastructure role expects you to know.

The invite.

And the email after it.

Before writing this, I asked if I could write about the interview, and whether they’d rather I didn’t name them. This was the reply, and it’s why there’s nothing from the interview itself in here.

What I asked.

Their reply.

Most of my day-to-day work is kernel and data plane code. Before the interview I went over the fundamentals around it as well, from the kernel boundary outwards.

The basics

These are what any Linux, systems or infrastructure role expects you to know: syscalls and what strace shows, fork and exec, threads and processes, signals and process states (D state, orphans, zombies, PID 1), virtual memory, page faults and overcommit, inodes and permissions, file descriptors, pipes and redirection order, what the shell expands before a program runs, the boot path from firmware to systemd, load and iowait, and sockets and TCP states. I wanted to be able to explain each of them without looking anything up.

The deeper parts

Most of my own time goes into the areas below, and I’ve written about all of them here.

XDP and eBPF. In xdp-bfd, replies are built inside an XDP program by rewriting the incoming packet. That meant BPF timers, keeping BPF maps consistent with userspace, pinning maps and links so a new process can take over without dropping anything, and fitting SHA1 into the 512-byte BPF stack.

Scheduling. Part 1 of that series is about how the normal scheduler picks the next task and what SCHED_FIFO does to everything below it. Part 9 gets into real-time throttling, and why a SCHED_DEADLINE reservation is the only thing that outranks a priority-99 task.

The receive path. On real hardware I had to deal with NIC queues, flow control and moving work to other CPUs with a cpumap, and I found PCIe power saving showing up as reply latency. Those are in part 9 and part 10.

TCP and tunnels. My offsite backups were slow because cubic collapses on a small amount of loss over a long path, and switching to BBR with fq pacing fixed it. The same post covers why a WireGuard peer behind NAT needs a keepalive.

Routing daemons. I have eleven fixes merged in FRR’s BFD daemon, from its data plane protocol to authentication, and they’re all in the FRR post.

How it went

From my side, the interview went well. I got through everything within the hour, and it wrapped up around the 56th minute. I don’t know what Jane Street’s criteria are, so I can’t say what decided it. Better luck next time.

Building xdp-bfd, part 7: checking my own work

w453y.me@gmail.com (Abdul Wasey) — Tue, 15 Sep 2026 00:00:00 GMT

While I was building the things in part 6, from the middle of August to the end of September, I spent just as much time checking what I already had. Most of xdp-bfd runs inside the kernel, where I can’t step through it or print anything, and nearly all the tests were ones I’d written myself while building each feature.

The review

I started by getting someone else to go through all of it. The first thing they found was that the two halves of xdp-bfd didn’t agree with each other. XDP and userspace both look at incoming BFD packets, and each had its own checks. A packet that said it was 200 bytes long but was really 24 got dropped by XDP and accepted by userspace, because userspace never looked at the length. I moved the checks into one function that both of them call.

The TTL check had a worse problem. Userspace asked the kernel to drop anything below TTL 255 using IP_MINTTL, but when I tested it in network namespaces, a TTL-64 packet went straight through. It turns out Linux only enforces that option for TCP. For UDP it does nothing, so the check had never worked. Now userspace reads the TTL off each packet itself.

Dead but still answering

The review also asked what happens when the engine dies. I killed it with kill -9, and XDP kept answering the peer anyway, using whatever config it had last. The peer had no idea anything was wrong. An XDP program normally stays attached after the process that loaded it is gone, so I switched to attaching it through a BPF link, which the kernel removes when its owner dies.

A stuck engine was harder. At the end of August I froze it for 15 seconds, and 55 of 64 sessions stayed Up the whole time. It’s the same thing that kept sessions alive under load in part 2, except this time XDP was telling the peer everything was fine for an engine that wasn’t running at all. So in September I added a dead-man gate. The engine bumps a heartbeat every time its loop goes round, and if the heartbeat gets older than a second, XDP stops answering and lets the peers time out on their own.

My first test for the gate was wrong too. It said the problem was fixed as soon as any session went down, and it said that while 55 were still up. A session going down while the engine is frozen doesn’t prove much anyway, since the frozen engine explains it just as well. So now the test runs twice: once with the gate on, where the session has to go down, and once with it off, where it has to stay up. After that I did the same for every new test, running it once with the thing it tests turned off to make sure it fails.

Fuzzing

Around the same time I pointed a fuzzer at the code that reads bfdd’s messages. On its first run, at the end of August, it found a read 42 bytes past the end of the buffer. If a reply couldn’t go out because the queue was full, the connection got dropped in the middle of the parse loop, the byte count went to zero, and the subtraction after it wrapped around. I fixed the loop and kept the input that caused it as a test.

Later I fuzzed the XDP program too. For every input, a dropped packet had to leave every session untouched, and anything sent back had to be a valid BFD packet. It got through about 10 million inputs without breaking either rule.

By the end of September the tests had grown into four layers, from a few seconds of checks that don’t need root, up to stock FRR bfdd running in containers against xdp-bfd. Every pull request also goes through about 28 CI checks.

The test that never ran

One of the checks I was counting on wasn’t running at all. Before every push, a git hook ran a script that fires bad packets at the test mesh and checks that each one gets dropped. At some point I’d moved the script to another folder, and the hook was still calling the old path. The error landed in the hook’s own “couldn’t parse, skipping” branch, which exits with success, so every push had passed for weeks without the test running once. When I fixed the path, the script turned out to have two more bugs of its own.

Answering too fast

Near the end of September I went through the RFC itself, and the biggest problem came straight out of part 2. XDP answers every packet the peer sends, so xdp-bfd was sending at the peer’s pace, not its own. If I set 300ms and the peer sent every 100ms, I was sending three times faster than the RFC allows. Now XDP only answers on its own while the peer’s pace is one I’m allowed to match. Otherwise userspace sends at my own rate, and XDP just watches for the peer going quiet.

Next, packaging it so someone else could run it.

Fixing my slow offsite backups over WireGuard

w453y.me@gmail.com (Abdul Wasey) — Sat, 12 Sep 2026 00:00:00 GMT

I run two Proxmox nodes at home with 28 guests between them. Every night at 02:00 each node backs everything up to its own disk with vzdump, and a cron job rsyncs those backups to an offsite box over WireGuard. The offsite box sits behind a NAT I don’t control, so the tunnel is the only way to reach it. That’s about 150 GB a night, 91 GB from one node and 59 GB from the other, and it had been working for months.

23 days of failed syncs

On 10 September I was setting up Prometheus, Loki and Grafana across the homelab with Claude Code, and one of the things on the list was monitoring the backup script. Before writing anything, Claude read the script’s log to see what it looked like. The first node’s log was fine. The second node’s had the same error every night:

rsync: [Receiver] mkdir "/root/backups/node-b" failed: Permission denied (13)
rsync error: error in file IO (code 11) at main.c(791) [Receiver=3.2.7]

The last time it had finished was 18 August, so it had been failing every night for 23 days. Each run failed in under a second and only wrote the error to a log file I never looked at, so I had no idea. 59 GB of backups had never left the box.

I never found out what changed on the offsite box that day. Making the directory exist fixed the error. Before running it again, we made the script report on itself. It now writes its exit code, duration and bytes sent to a file Prometheus picks up, even when it fails, and every step has a timeout. One check took more than I expected. A quick TCP connect to port 22 wasn’t enough, because when the offsite box was half dead, it answered the TCP handshake and then hung in the middle of the SSH handshake. Only a real SSH login with a timeout catches that.

Then it was slow

With the permission fixed, both nodes started a real sync, and both failed after about 100 minutes with rsync’s exit code 30, a timeout while sending. The progress output showed the transfer starting at hundreds of MB/s and then sinking to a few MB/s.

Claude measured the link by piping 400 MB through SSH:

dd if=/dev/zero bs=1M count=400 | ssh peer "cat > /dev/null"

That gave 15 Mbps, and then 16 and 17 on the next two runs. From there it decided the link was 16 Mbps, worked out that shipping 150 GB at that speed would take 21.3 hours out of every 24, and told me offsite replication was impossible as I’d set it up. Its suggestions were to move to Proxmox Backup Server or drop the bigger guests from the job.

That didn’t match what I was seeing. 87 GB had already arrived on the offsite box, and the tunnel runs over a 1 Gbps link. So I ran iperf3 over UDP through the tunnel, in both directions:

$ iperf3 -c iperf.example.net -u -b 1000M -t 10
[  5]   0.00-10.00  sec  1.04 GBytes   896 Mbits/sec  0.000 ms  0/943220 (0%)

$ iperf3 -c iperf.example.net -u -b 1000M -t 10 -R
[  5]   0.00-10.00  sec  1.05 GBytes   903 Mbits/sec  0.011 ms  101653/1051648 (9.7%)

About 900 Mbps both ways. If something on the path were limiting it to 16 Mbps, sending 1000 Mbps of UDP would have lost almost all of it, not 9.7%. The dd test had only measured one TCP stream going through SSH, and that was slow for some other reason.

The tunnel was broken too

I also had a ping running from the offsite box back to the homelab, and that showed a different problem. One run lost 33% of its packets, and a few minutes later nothing was getting through at all. The internet from the same box was fine, so I wondered if it was my OPNsense firewall.

The WireGuard status on OPNsense showed the offsite peer’s last handshake was over an hour old. WireGuard renews its handshake about every two minutes while traffic is flowing, so anything past three minutes means the tunnel is dead. The other peers on the same server had handshakes from seconds ago, so the firewall was fine. The difference was that the offsite peer had no persistent keepalive set.

WireGuard doesn’t send anything when there’s nothing to send. The NAT in front of the offsite box forgets the mapping after a short time with no traffic, and after that, my side has no way to reach it. Only the offsite box sending something opens it up again. That’s also why, for months, I’d kept a ping running from the offsite box all the time. Traffic going out from behind the NAT was the only thing keeping the mapping alive, so that ping was doing the keepalive’s job by hand.

I set a 25-second keepalive on both ends, though it’s the one on the offsite box that matters, since it’s the side behind the NAT. The loss went from 33% to zero:

50 packets transmitted, 50 received, 0% packet loss
rtt min/avg/max/mdev = 52.068/52.459/52.879/0.196 ms

Still slow

The backups were still slow with the tunnel fixed, so I measured TCP with iperf3 this time. Towards the homelab it did 502 Mbps with no retransmits. Towards the offsite box, the direction the backups go, it did 41 Mbps with 164 retransmits, and four parallel streams more than doubled that. If the link itself were the limit, four streams would have shared the same 41 Mbps. So something was holding back each single TCP flow.

Next we checked everything in between. OPNsense was 99% idle during the transfer, the MSS was clamped below the MTU on both ends, the offsite box’s CPU was idle, and its tunnel interface had no drops. None of it explained anything, so I swept UDP at a few fixed rates instead, to see what the path itself was doing:

offered  50M  ->  50.0 Mbps   0.029% loss
offered 100M  -> 100   Mbps   0.038% loss
offered 200M  -> 200   Mbps   0.032% loss
offered 400M  -> 400   Mbps   0.033% loss

400 Mbps went through with 0.033% loss, about 3 packets in 10,000. The path was fine.

Why one TCP flow was slow

That much loss is nothing to UDP, but it hurts a single TCP flow over a 52ms round trip. Both of my nodes were using cubic, the default congestion control on Linux, and cubic treats every lost packet as a sign of congestion and cuts its sending rate. With a steady trickle of loss over a long path, it never gets to speed up.

There’s a rough formula for this, from Mathis: throughput is about MSS divided by RTT times the square root of the loss rate. With a 1200-byte MSS, 52ms and 0.033% loss, that comes out at about 10 Mbps. With almost no loss, the same path allows about 584 Mbps. Cubic actually got around 38, more than the formula says, probably because the formula assumes an older and more cautious algorithm, but it’s nowhere near 584.

BBR works differently. Instead of slowing down on loss, it measures how fast the path is and how long the round trip takes, and sends at that rate. So I tried both, one stream each, in the same minute:

$ echo cubic > /proc/sys/net/ipv4/tcp_congestion_control
$ iperf3 -c 10.0.0.2 -t 10
[  5]   0.00-10.00  sec  36.5 MBytes  30.6 Mbits/sec  312   sender
[  5]   0.00-10.04  sec  33.4 MBytes  27.9 Mbits/sec        receiver

$ modprobe tcp_bbr
$ echo bbr > /proc/sys/net/ipv4/tcp_congestion_control
$ iperf3 -c 10.0.0.2 -t 10
[  5]   0.00-10.00  sec   473 MBytes   396 Mbits/sec  1999  sender
[  5]   0.00-10.04  sec   472 MBytes   394 Mbits/sec        receiver

27.9 Mbps with cubic, 394 with BBR. The retransmits went up, from 312 to 1999. BBR was losing more packets, but it didn’t back off each time. Through SSH, the same 400 MB test from before now ran at 200 Mbps instead of about 20.

The fix

The fix was three lines of sysctl on both nodes:

# /etc/sysctl.d/99-homelab-tcp.conf
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
net.ipv4.tcp_wmem = 4096 16384 33554432

fq is the queueing discipline BBR is designed to work with, since it does the packet pacing BBR relies on. The bigger send buffer is a separate, smaller thing. The old 4 MB limit capped a single flow at about 645 Mbps over this path, which wasn’t the problem here, but there was no reason to keep it. I also added tcp_bbr to /etc/modules-load.d/ so it loads after a reboot.

After that, both nodes finished. The second node was in sync after 9 minutes, its first clean sync since 18 August, and the first node after 18, 69 GB image included. The earlier runs had already got 87 GB across before timing out, so these two only had to finish the job. They moved 57.7 GB between them, which is an average of 285 Mbps through rsync and SSH. At that rate, the full 150 GB that was supposed to need 21.3 hours a day takes a bit over an hour.

The other changes

A few other things I changed along the way were really workarounds for cubic, from before I knew it was cubic. I switched rsync from --inplace to --append-verify, so a resumed 69 GB file appends from where it stopped instead of comparing both copies from the start. I split the sync across four parallel rsync workers, which went from 19 Mbps to 58, because each stream gets its own congestion window. And I switched SSH to chacha20, which was about 20% faster on these CPUs. I kept all three anyway.

The monitoring

The monitoring I was setting up in the first place is running now too. The backup script reports how long each run took, how much it sent and how it exited, and Grafana shows that for both nodes:

Run time, data sent and exit code for each node since 10 September. The spikes on the 10th are the runs from before the fix.

There are seven alert rules around it. The main one fires when the last successful sync is more than 48 hours old, since the offsite box going offline for one night is normal and three nights isn’t. Two of them watch the backup peer itself, one for its WireGuard handshake going stale and one for its keepalive being turned off.

The main alert and the two for the backup peer, none of them firing. Click for all seven.

Building xdp-bfd, part 6: demand mode, auth and hitless restarts

w453y.me@gmail.com (Abdul Wasey) — Sun, 06 Sep 2026 00:00:00 GMT

After part 5, three big things were still missing, and they took up September. There was no demand mode, no authentication, and restarting xdp-bfd itself, for an upgrade or anything else, took every session down with it.

Demand mode

In demand mode, two peers stop sending regular packets once both are Up, and only check the path every now and then. bfdd had a setting for it, but the setting didn’t do anything. The flag was set to 0 in three places and never set to 1 anywhere. So I implemented it in FRR first, which has its own section in the FRR post, and then in xdp-bfd.

The trick from part 2 doesn’t work here, though. When the peer stops sending, there’s nothing for XDP to bounce. So while the peer is in demand mode, xdp-bfd stops replying from XDP and leaves the occasional Poll to userspace. While xdp-bfd is the side asking for quiet, it also holds off its own detection, or it would time out a peer that was only doing what it was asked.

At first it only worked in one direction. When xdp-bfd was the side asking for quiet, the peer just kept sending. It turned out xdp-bfd was going quiet within a microsecond of both sides being Up, before it had sent a single packet saying it wanted demand mode, so the peer had never actually been asked. Now it says so three times first.

Once both sides are quiet, something still has to check the path is there. The RFC leaves that optional and bfdd never does it, so xdp-bfd sends a Poll once a second.

Signing packets inside XDP

BFD authentication adds a signature to every packet, so the other side can tell it really came from you. In xdp-bfd the replies are built in XDP, so XDP had to be the one signing them, and it also had to check the signature on every packet coming in.

The kernel doesn’t give XDP programs a hash function to call, so I wrote SHA1 into the XDP program myself. The tricky part was that BPF programs only get 512 bytes of stack. The usual way of computing SHA1 keeps 80 values around at once, and that didn’t fit next to everything else. So my version only keeps the last 16 and works out the rest as it goes.

FRR’s keyed SHA1 also isn’t quite what the BFD RFC describes. It’s an HMAC, which is a different way of mixing the key in. xdp-bfd does it FRR’s way, since FRR is what it has to talk to, and I opened an issue in FRR about the difference.

Getting the keys from FRR

xdp-bfd could sign packets now, but bfdd had no way to hand it the keys. The message bfdd uses to pass a session to xdp-bfd had a TODO comment where the key was supposed to go. On top of that, when you set up an authenticated session in FRR and it got handed to a data plane, it went out with no authentication at all, while show bfd peer still said auth was on.

I fixed that in FRR, and it was merged in late September. bfdd now sends the keys that are usable now and the ones that will be later, along with when each is valid, not just the current key. That way xdp-bfd can switch to the next key on its own when one expires. Around the same time I also fixed two more auth bugs in bfdd itself, and those are in the FRR post.

On the test mesh, all 68 authenticated sessions came up.

Restarting without dropping anything

The other problem was restarts. Every time I restarted xdp-bfd, every session went down with it. On the test mesh with 1024 sessions, one restart caused 1022 flaps.

What I wanted was for a new copy of xdp-bfd to take over from the old one without the peers ever going unanswered. So I added a --pin option. With it, xdp-bfd pins its BPF maps and the attachment of its XDP program in the kernel, so they stay there after the process exits. When a new xdp-bfd starts, it picks up the pinned sessions, swaps its own XDP program into the pinned attachment in one step, and tells the old one to exit. The attachment and the session state never go away, so the peers keep getting answers the whole time. The old process was gone within 13ms and the new one had taken over in about 12ms, and this time the restart caused zero flaps.

With that, the test mesh was 1024 sessions on Proxmox VMs, covering everything from the earlier parts, and all of them came up apart from two I left pointing at peers that don’t exist on purpose.

Next: checking my own work.

FRR bfdd dropping the data plane on "show bfd peers counters"

w453y.me@gmail.com (Abdul Wasey) — Wed, 19 Aug 2026 00:00:00 GMT

With 64 sessions running through xdp-bfd, I ran show bfd peers counters in FRR and it worked. I ran it again, and bfdd dropped its connection to xdp-bfd partway through. The third time it worked again, and from then on it kept alternating, failing every second run.

Something that fails every other time usually means something is left over from the previous run. In this case it turned out to be bfdd’s input buffer.

What was going wrong

When you run the counters command, bfdd asks the data plane for each session’s counters and reads the replies into an 8KB buffer. Normally bfdd clears out the replies it has already read to make room, but the function used for this command only did that after three reads in one go. The counters command reads one reply at a time, so the clean-up never happened. After a full run over 64 sessions, 5,120 bytes of the buffer were still taken up by replies bfdd had already dealt with.

On the next run there were only 3,072 bytes free, which is enough for 38 replies. Reply 39 filled the buffer exactly, and bfdd then asked the socket for zero bytes, because there was no room for more. A zero-byte read always returns zero, and in bfdd a read that returns zero means the other side closed the connection. So bfdd decided xdp-bfd had hung up and closed a perfectly good connection. Reconnecting cleared the buffer, which is why the run after that worked again.

That works out to 64 plus 39, so 103 replies per connection, and that’s exactly what I saw every time.

From the command line, the failing run still printed the full list of peers, just with old or zero counters for every session after the point where it broke, and with no error unless debug logging was on. Each failure also made bfdd reconnect and register every session again, so anything polling this command for monitoring would keep setting it off.

The fix

The fix clears out already-read replies every time that function runs, so the buffer can’t fill up with them. It also stops bfdd from ever asking for zero bytes. If the buffer is somehow still full after clearing it, a single message must be bigger than the whole buffer, so bfdd now logs that as an error and closes the connection properly.

My first version returned an error there without closing the connection, which would have left it stuck. Greptile, the automated reviewer on the PR, pointed that out, so I changed it to close the connection the same way bfdd does for its other errors.

Writing a test for it

Donald Sharp asked for a topotest that shows the fix works. The catch was that this bug only happens when bfdd is connected to an external data plane, and FRR’s tests had nothing that could act as one.

He pointed me at fpm_listener, a small fake receiver FRR already uses to test a similar connection in zebra, so I wrote bfd_dplane_listener the same way. It accepts bfdd’s connection, keeps track of the sessions bfdd sends it, answers counter requests and reports every session as up, without running any real BFD. The test runs the counters command 20 times over 8 peers and checks that every request got an answer without a reconnect. On the old code it fails with 103 of 160 counter requests were answered, which is right where the buffer runs out.

Getting that merged needed a few extra changes I hadn’t expected: a build macro the new program needed, packaging entries for both RPM and Debian, and a noreturn attribute because FRR’s OpenBSD build treats warnings as errors. The fix itself was only a few lines, and the test setup ended up being most of the pull request.

It’s merged as #22694, and the fake data plane is now part of FRR’s tree, so this part of bfdd can be tested in CI.

FRR bfdd losing data plane sessions to a full buffer

w453y.me@gmail.com (Abdul Wasey) — Sat, 15 Aug 2026 00:00:00 GMT

I was testing xdp-bfd with more and more sessions, with FRR’s bfdd handing them over. At 16 sessions everything worked. At 64, some sessions just never came up, and there was nothing about them in any log.

Losing sessions on connect

The problem was in what bfdd sends when it connects to a data plane. It goes through every configured session and puts one registration message for each into an 8KB output buffer, all in one go, before any of it gets sent. A session with default settings takes a 140-byte message, so 58 of those fit. My sessions had timers configured, which makes each message 203 bytes, so only 40 fit. With my 64 sessions, the last 24 didn’t make it.

A full buffer by itself would have been a normal error. What made it bad was what bfdd did with the sessions that didn’t fit. Before registering them, it had already switched off its own software BFD for them, since the data plane was supposed to take over. Then the registration failed, and bfdd cleared the session’s link to the data plane. So those sessions weren’t being run by the data plane or by bfdd. They still showed up in show bfd peers, but nothing was actually doing anything with them.

I wanted the FRR maintainers to be able to reproduce this without needing xdp-bfd, and it turned out a real data plane wasn’t needed at all. A small script that accepts bfdd’s connection and just counts the bytes it receives was enough. With enough sessions configured, the count always stopped at exactly one buffer’s worth.

The fix, #22645, has two parts. When the buffer fills up, bfdd now sends what’s in it before giving up. During the registration burst the socket can almost always take more, so normally this just works. And if a registration still fails, bfdd logs it and goes back to running that session in software, the same way it already did when a data plane disconnects. With 128 sessions and the byte-counting script, the old code registered 58 and silently lost the other 70. With the fix, all 128 got through.

Losing deletes on shutdown

With that fixed, I looked at what happens when bfdd shuts down. It puts a delete message for every data plane session into the same buffer, but by then its event loop has stopped, so nothing ever sends them. Then it closes the connection and throws the buffer away with the messages still in it.

What the data plane got depended on the version. Before my first fix, it got no deletes at all. After it, the new “send when full” code kicked in once halfway through, so exactly 58 deletes arrived and the rest were lost. The second case is actually more confusing for a data plane. Getting nothing looks like a crash, but getting 58 looks like a normal shutdown that happened to leave some sessions behind.

The fix for this one, #22692, makes bfdd write out whatever is left in the buffer right before it closes the connection on shutdown. It writes as much as the socket will take, and if anything is still left over, it logs how many bytes were lost instead of dropping them silently. With a real data plane holding 64 sessions, a clean bfdd restart used to leave 6 of them behind. With the fix, all 64 deletes arrived and nothing was left behind.

Both fixes are merged into FRR’s master branch. On released versions, if you have more than about 40 data plane sessions (up to 58, depending on your config), add them through vtysh after the data plane has connected, instead of putting them in frr.conf.

Building xdp-bfd, part 5: IPv6, echo, and multihop

w453y.me@gmail.com (Abdul Wasey) — Mon, 03 Aug 2026 00:00:00 GMT

Back in part 3 I had a list of things xdp-bfd couldn’t do yet. After the 64-session work in July, the biggest ones left were IPv6, echo mode, and multihop sessions. This part goes through all three, which took the rest of July.

IPv6

Adding IPv6 was mostly careful plumbing. I changed the session key so both IPv4 and IPv6 addresses fit in it, then taught the XDP program to read IPv6 headers and build IPv6 replies. The more interesting part was something I hadn’t planned.

This whole project rests on one idea: sending the reply from the kernel is what keeps sessions up under load. Up to here, though, I’d only shown that by comparing different programs on different runs, so the load was never exactly the same between them.

While I was adding IPv6, there was a point where the XDP program could already read IPv6 packets, but IPv6 replies were still being sent from userspace. IPv4 replies were already coming from the kernel. So I ran both at the same time, on the same box, through the same stress, with 300ms timers. IPv4 had zero flaps. IPv6 had 19, with gaps of up to 1.9 seconds.

Then I turned on the kernel reply for IPv6 and ran it again. IPv6 had zero flaps too. The only thing that changed between those two runs was where the IPv6 reply came from, so that settled it for me.

With both in place, I ran 32 IPv4 and 32 IPv6 sessions together under stress, and neither had a single flap.

Echo

Echo is a BFD mode where you send a packet addressed to yourself. The neighbour’s router just forwards it back to you, without its BFD daemon ever seeing it, and you time how long the round trip took. It checks that the neighbour can still forward packets, not just that its BFD software is running. I expected it to be one of the easier features.

XDP can only do half of it, though. Bouncing a neighbour’s echo back is easy, because there’s a packet in hand, the same trick as part 2. But sending my own echoes means creating packets from nothing, and XDP can’t do that. So the bouncing happens in XDP and the sending happens from userspace. Since userspace sending can be starved, I also made sure a failing echo only gets reported and never takes a session down on its own.

Up to then, I had only tested the bouncing with echo packets I built by hand. While working on the FRR side of echo, I found out why no real peer had ever been part of those tests: xdp-bfd had been telling every peer it couldn’t receive echo. BFD has a field for how often you’re willing to receive echo packets, and xdp-bfd was sending 0 there, which means “don’t send me any”. Once I fixed that, FRR started sending echoes to it, and xdp-bfd bounced back all 433 of them in that test, the fastest in 12 microseconds.

The most confusing bug came under load. Echo sending stalled for 2.6 seconds at a time, but the loss counter stayed at zero and echo still showed as healthy. The counter only counts a lost echo when it’s time to send the next one and the last one hasn’t come back. During a stall, it’s never time to send the next one, so nothing ever got counted. I only caught it by separately measuring the time between sends.

In September I found one more echo problem, this time with IPv6. FRR’s IPv6 echo works differently from IPv4. Instead of addressing it to itself, bfdd sends it to the peer and expects the peer to send it back. On the xdp-bfd side that only worked while the local bfdd happened to have an echo socket open, which in data plane mode depends on the order its config gets applied. After an FRR restart it didn’t have one, so the host answered those echoes with an ICMP error instead, and all 28 IPv6 echo sessions on the test mesh started flapping. XDP now bounces those echoes itself, and the socket problem is an open FRR fix.

Multihop

Normal BFD sessions are between directly connected boxes, which is why part 3 required a TTL of 255. Multihop sessions go through other routers on the way, so the TTL arrives lower, and instead each session gets a minimum TTL it will accept. I kept the old strict check for single-hop sessions and only let lower TTLs through for sessions that are actually set up as multihop.

I wrote a test for it, and everything passed on the first try. When I looked at the capture afterwards, though, the test hadn’t been testing multihop at all. Multihop BFD uses port 4784, and my test was sending to 3784, the port for normal single-hop sessions. The capture showed FRR sending to 4784 and my test packets going to 3784, and none of them had touched the multihop code. After I fixed the port, the test finally ran against the multihop path.

Next: demand mode, auth, and restarts that don’t drop anything.

Building xdp-bfd, part 4: from 1 session to 8192

w453y.me@gmail.com (Abdul Wasey) — Sun, 26 Jul 2026 00:00:00 GMT

Every number up to part 3 came from a single session. The maps had room for 64 and the docs said “multi-session”, but I had never actually run more than one. A real router talks to a lot of neighbours, so I wanted to know what happens when xdp-bfd has to handle more than one at a time.

Going past one session

I started small. Two sessions came up fine, and when I killed FRR on one of the neighbours, that session went down in 31ms while the other one carried on without noticing. Sixteen sessions were fine too, even with the L3 stress from part 1 running.

Going to 64 sessions found five bugs. Four of them were mine, like my per-session source ports starting at 49152, which is exactly where bfdd hands out its own, so the two collided.

The fifth was in FRR. When bfdd connected, it sent all 64 sessions to xdp-bfd in one go, overflowed an 8KB buffer, and silently lost everything after the 40th. That bug ended up with its own post.

The 64-session runs under L3 showed one more thing. At one point 19 sessions flapped at the exact same moment. It was the same cause as the single flap in part 3: a softirq that ran late. With one session, a late softirq only delays one reply. With 64, every session’s reply goes through that same softirq, so they were all late together, and they all went down together. They came back on their own in about 50ms.

Why 1024 sessions was eating a CPU core

All of that was back in July. In September I came back to scale and raised the default to 1024 sessions. XDP handled that without any trouble, but the userspace side was now using 16.8% of a CPU core just to keep those sessions going.

The reason was that userspace checked the maps on a timer, going through every session each time whether anything had changed or not. With one session that was nothing. With 1024 it added up. I changed it so userspace only wakes up when there’s actually something to handle. After that it sat at 2.1% of a core with the same 1024 sessions, and sessions were still being declared down on time.

How far it goes

After that I wanted to see how far it could be pushed, so I added a --max-sessions option that goes up to 8192 and tested it against a normal FRR bfdd on the other end, with 300ms timers:

Sessions	xdp-bfd CPU	Flaps
2048	3.8%	0
4096	6.2%	0
8192	12.1%	38

At 8192 I got 38 flaps, so I looked into them. xdp-bfd was fine. The bfdd on the other side was using 53% of a core just to keep up and was falling behind.

An audit I did just before this scale work also found a bug. XDP could read a session’s config while userspace was halfway through updating it, and send a reply carrying another session’s discriminators, or half of each version. A test built to hit that race produced these broken replies until I fixed it by having XDP lock the config and work from its own copy, which only costs about 37 nanoseconds per reply.

The strangest problem wasn’t in xdp-bfd at all. During the 4096-session tests, the host running them lost ARP for its own management address. My test setup used a lot of network namespaces, and the kernel’s neighbour table is shared between all of them. The test had filled it past its limit.

Next up is IPv6, echo and multihop.

Building xdp-bfd, part 3: results, FRR, and hardening

w453y.me@gmail.com (Abdul Wasey) — Sun, 19 Jul 2026 00:00:00 GMT

With the XDP version from part 2 passing the FIFO test, I wanted to see how it did on the whole stress ladder from part 1. So I ran it through every level plus a five minute soak, at normal process priority, with stock FRR on the other end:

Level	p50	p99	max	flaps
L1 CPU hogs	8.97	10.03	13.3	0
L2 context switching	8.83	10.01	12.0	0
L3 timer storm	8.72	10.11	28.0	1
L4 FIFO hogs	8.72	10.01	11.5	0
5 min soak	8.77	10.01	10.7	0

The only flap was under L3, the same load that made bfdd flap 44 times. I wanted to know what caused it, so I went through the capture around it. One reply had been stuck in softirq for 28ms, just long enough for the peer’s 30ms timer to run out, and the peer took the session down.

It came back by itself, though. My userspace saw the peer go Down and brought the session back up, and the whole thing took 3.8ms. Across all eleven minutes of testing, the session was down for less than 4ms. The scheduler wasn’t a problem anymore, but a slow softirq still could be, and that was the one time it happened.

Hooking it up to FRR

All of those runs had xdp-bfd managing the session by itself, with its own config. On a real router, though, BFD gets set up in FRR along with BGP and the rest, so I wanted FRR to be the one in charge and xdp-bfd to only handle the packets.

bfdd already has a mode for that. It can hand the packet side of its sessions to a separate program over a protocol called bfddp, which exists so hardware vendors can plug their own forwarding hardware into FRR. I added bfddp support to xdp-bfd so bfdd could use it the same way. You set up BFD in FRR like normal, and show bfd peers looks like normal, but the packets go through XDP, and even the counters FRR shows are read from the BPF map.

Since both were running on the same box, I first tried connecting them over a UNIX socket, and bfdd failed to connect every time with EINVAL. Over TCP it connected fine. The UNIX socket failure turned out to be a bug in FRR, and it was the first of quite a few FRR bugs I ran into, so those have their own post.

Once the two were talking, I ran the FIFO test again with FRR in charge. It still had 0 flaps and a 13ms worst gap, about the same as before.

That was still only one IPv4 session, though, and restarting bfdd took it down, so that’s what I worked on next.

Surviving a bfdd restart

Restarting bfdd is a normal thing to do, for an upgrade or a config change, so I didn’t want it taking routes down. The reason it did was pretty simple. As soon as bfdd disconnected, xdp-bfd threw the session away, because as far as it knew nobody was managing it anymore. bfdd added the session back when it reconnected, but by then the peer had already seen it go down, and any routes depending on it went down too.

So I added an option called --dp-hold. It tells xdp-bfd to hold on to its sessions while bfdd is away and keep answering the peer as if nothing happened. When bfdd reconnects and adds its sessions again, xdp-bfd lines each one up with the session it’s been holding, so the peer never notices, and anything bfdd doesn’t add back gets removed. Rafael Zalamena on the FRR list helped me work out how this should behave.

With that, restarting bfdd stopped taking the session down in my tests. Much later, on the big test mesh, I found where it falls short: one clean FRR restart took all of the roughly 1,000 sessions down at once. --dp-hold only helps when bfdd goes away without warning, like a crash. When FRR is stopped properly, bfdd first sends a delete for every session, and xdp-bfd removes them, since that’s what it was asked to do.

Refusing bad packets

The other big gap was that xdp-bfd barely checked the packets it received. In this design every packet that gets through tells xdp-bfd the peer is still alive, and it also gets bounced back to the peer. So a fake packet that got past the checks could keep a session up even after the real peer had died.

So I added checks. A packet has to arrive with a TTL of 255. Every router a packet passes through lowers its TTL by one, so 255 means it came from a directly connected box (this is called GTSM). The discriminator in the packet has to match the session. And a session can only be created by bfdd, not by someone sending packets at the box. If a packet fails any of these, XDP drops it and userspace never sees it.

The discriminator check needed one exception. When a peer restarts, it doesn’t know my discriminator yet, so it sends zero. I have to accept that, or the session could never come back up. But I only accept zero when the packet also says the peer is Down, so a random packet with zero in it still gets dropped. Here’s the check:

if (cfg && cfg->my_disc) {
        __u32 ydisc = bpf_ntohl(bfd->your_disc);
        if (ydisc != cfg->my_disc &&
            !(ydisc == 0 && rstate <= 1)) {   /* Down or AdminDown */
                count(3);
                return XDP_DROP;
        }
}

The first time I tested this, from another host, the checks counted every bad packet and the session still flapped. The reject paths were returning XDP_PASS instead of XDP_DROP, so the bad packets were counted and then handed up to userspace anyway, and a forged packet with the right discriminator overwrote the session’s state. After switching both to XDP_DROP, I sent 600 forged packets at it, all 600 were dropped in XDP, and the session never noticed.

The review

After adding those checks, I had someone else review the code. I nearly ignored two of their comments, but both were real bugs.

One of them was in the checks I had just written. They read the TTL and the discriminator from fixed spots in the packet, which only works if the IP header is the usual 20 bytes. A packet with IP options has a longer header, so everything after it starts later, and my code was reading the wrong bytes. That meant a packet with options could get past both checks and reach userspace. Single-hop BFD never uses IP options, so I made XDP drop any packet that has them.

The other one was in how xdp-bfd reads bfddp messages from bfdd. When a message came in broken, the reader cleared its buffer and kept reading from the same connection. It could then start reading halfway through the next message and treat whatever bytes were there as a new one. I changed it to close the connection instead. bfdd reconnects on its own, and with --dp-hold the sessions stay up while it does.

They also noticed that the structs shared between the XDP program and userspace were copied into three different files and kept in sync by hand, so I moved them into one header. I also kept a note of the suggestions I didn’t take and why, and that note ended up being really useful later.

After this, the next thing I needed was more than one session. That’s part 4.

Building xdp-bfd, part 2: getting transmit out of userspace

w453y.me@gmail.com (Abdul Wasey) — Fri, 10 Jul 2026 00:00:00 GMT

Part 1 ended with my simple daemon breaking under SCHED_FIFO load: 25 flaps in a minute. So I started trying other ways to get the packets out on time, running each one under the same test as before (four FIFO hogs, 60 seconds, 3x10ms, captured from the hypervisor).

Raising the priority

The first thing I tried was running my daemon at real-time priority 90, above the priority-50 hogs. That fixed it completely: zero flaps and a 15ms worst gap.

It only worked because nothing else on my test box wanted a higher priority, though. On a real router the forwarding path runs at high priority too, and I didn’t want BFD fighting it for the CPU. So I kept looking.

Trying SO_TXTIME

Next I tried SO_TXTIME, which I had high hopes for. It lets you queue a packet with a launch time, and the etf qdisc sends it at exactly that time. My thinking was that even if the daemon got to run late, the packets it had already queued would still go out on time.

The first problem showed up while setting it up. I put the etf qdisc on all of the interface’s queues, and the interface quietly stopped resolving neighbours. It turns out etf drops every packet on its band that has no launch time, and that includes ARP, so I had to limit the tc filter to just the BFD flow.

Once it worked, I ran the FIFO test with one packet queued at a time, and it flapped 48 times. So I changed it to queue five packets ahead. My first try at that kept renegotiating the session once a second: packets like the Final that answers a Poll need to go out right away, and I was scheduling them a full second in the future. The logs looked normal, and I only spotted it in the capture. With that fixed, the five-packet version got the best p99 of anything I’d tried, 10.1ms, and it still flapped 48 times.

Next I checked the qdisc stats to see if it was dropping packets. It wasn’t. Every packet I’d queued had gone out exactly on time. The gaps were happening after the queue ran dry, while the daemon was waiting for CPU to queue the next ones. Five packets only lasts 50ms, and the daemon was going almost a whole second between turns.

I did think about queuing a lot more, but every packet carries the session state from when it was built. With a second of packets queued up, I’d be telling the peer what the session looked like a second ago.

Where that left me

At this point I had tried three different ways of sending from userspace. Next to bfdd from part 1, they looked like this:

How it sends	flaps	p50	p99	max gap
FRR bfdd	constantly	8.8	287	960ms
my simple daemon	25	10.0	12.0	324ms
simple daemon at `chrt -f 90`, pinned core	0	10.0	13.0	15.0ms
SO_TXTIME + etf, one packet queued	48	10.0	13.2	1517ms
SO_TXTIME + etf, 5 packets queued	48	10.0	10.1	994ms

The only one that survived was the one I’d bumped to a higher priority. So I started looking at sending from inside the kernel instead, since that keeps running when userspace doesn’t. XDP was the obvious place to try, except that XDP only runs when a packet arrives. It can’t send one on its own.

Bouncing the peer’s packet back

My first thought was to find some way for XDP to send on a timer. BPF has timers that run in the kernel, but a timer callback doesn’t have a packet, so there’s nothing for it to send. I checked whether kernel 7.0 had changed that, and it hadn’t.

Then I noticed that the peer is already sending me a packet every 10ms. XDP can’t create a packet, but it can change one that just arrived and send it back out. So that’s what I did. When the peer’s packet comes in, my XDP program swaps the addresses around so it points back at the peer, writes my side of the BFD session into it, and sends it straight back. Userspace keeps a small config map up to date with what my side should say, and XDP just copies from it. Here’s the part that fills in the reply:

bfd->vers_diag   = (1 << 5) | (cfg->diag & 0x1f);
bfd->flags       = ((cfg->state & 0x3) << 6) | send_final;
if (!send_final && cfg->poll && st->final_seq != cfg->poll_seq)
        bfd->flags |= BFD_F_POLL;
bfd->detect_mult = cfg->mult;
bfd->my_disc     = bpf_htonl(cfg->my_disc);
bfd->your_disc   = bpf_htonl(cfg->your_disc);
/* ...timers the same way... */
return XDP_TX;

The reply leaves about 30 microseconds after the peer’s packet arrives, and none of it waits on userspace. I don’t have a send timer anymore, because the peer’s packets are the timer. If the peer’s packet has the Poll bit set, the reply gets Final, so even Poll sequences get answered without userspace being involved.

Then I ran it through the same FIFO test, at normal priority. It had zero flaps and a 12.5ms worst gap. The p50 was different too: all the userspace versions sat at exactly 10.00ms, because that’s their own timer, but the XDP version was at 8.75ms. That’s the peer’s timing, jitter included, coming straight back to it.

Noticing a dead peer

That covered sending, but I also needed to notice when the peer stops. A dead peer doesn’t send anything, so my XDP program never runs and has no way to notice. So I added a BPF timer that looks at every session every 5ms and marks any that have been quiet for longer than their detect time. Userspace then picks that up, takes the session down and tells bfdd. Under the timer stress from part 1 it caught a dead peer 33 to 34ms after it stopped, with a 30ms detect time. For a long time I thought the extra few ms came from the 5ms timer, but when I tested that later, the extra actually came from how often the userspace loop checked for the mark.

Under load, it sometimes marked a perfectly healthy session as dead. The log said the session had been silent for about 18 quintillion ms.

Here’s what was happening. The timer grabs the current time and then goes through the sessions. If a packet comes in on another CPU during that, the session’s last-seen time ends up a little newer than the time the timer grabbed. Subtracting the two should give a tiny negative number, but I was using unsigned math, so it wrapped around to 18 quintillion instead.

The fix was to subtract as signed and skip anything negative.

What stayed in userspace

With sending and detection in the kernel, I left everything else in userspace: the state machine, bringing sessions up, and the slow once-a-second packets the RFC wants while a session is down. My plan was for userspace to go quiet once a session was Up and let XDP handle it from there.

That plan didn’t survive testing. A session got stuck going round in a loop every 32.6ms, and the logs said the state machine was perfectly happy. In the capture there was one packet missing each time, and it was a Final. Sessions often go Up because the peer sent a Poll, so the Final answering it was exactly the packet userspace had just decided not to send, and XDP had already used up the peer’s packet by then. Now userspace sends that one packet as the session goes Up, and XDP takes over after it.

Since every reply rides on one of the peer’s packets, the peer does need its own timer for any of this to work. I used to assume two xdp-bfd engines facing each other would just sit there waiting for each other. When I actually tried it, they did the opposite: each reply set off the next one, and they bounced about 690,000 packets back and forth in five seconds, with a 10ms interval configured. It needs something like FRR, with its own timer, on the other end, and that’s still the case today.

Next, putting FRR in charge of it and hardening it.

Building xdp-bfd, part 1: why does software BFD flap?

w453y.me@gmail.com (Abdul Wasey) — Mon, 06 Jul 2026 00:00:00 GMT

BFD is the thing that tells BGP or OSPF a neighbour is gone. Two boxes send each other tiny UDP packets every few milliseconds, and if a few in a row go missing, the link is declared dead and routes get pulled. At 3x10ms (send every 10ms, give up after 3 misses) that happens in 30ms.

The catch is that BFD is only useful if it’s on time. If the sender is late, the other side declares a perfectly fine link dead and tears down routes for nothing. So a slow BFD daemon is worse than no BFD at all.

The common advice is that software BFD can’t be trusted with fast timers when the box is busy. That’s why people run 3x300ms, why SONiC caps software BFD at 300ms, and why real routers push BFD onto line cards. I wanted to know if that’s actually true, why, and whether a normal Linux box could get line-card behaviour out of XDP.

New to BFD or XDP? Quick terms

BFD: failure detection between two routers (RFC 5880). Session is one BFD conversation, flap is a session going down and back up.
FRR: the routing suite most Linux routers run. bfdd is its BFD daemon. It can hand packet work to an external data plane over a protocol called bfddp, which is how xdp-bfd plugs in.
XDP: an eBPF hook at the very start of packet receive. A program looks at the packet and returns XDP_PASS (carry on), XDP_DROP or XDP_TX (send it straight back out). It can only react to packets. It can’t create one.
Softirq: kernel work that runs ahead of every normal process. XDP runs here, so it keeps running when userspace is starved.
SCHED_FIFO: real-time scheduling. FIFO tasks beat every normal task, and the kernel only leaves normal tasks about 50ms per second when FIFO work eats every core.
p99 / max: p99 is the value 99% of samples fall under. Max is the single worst. For BFD the max is the one that kills you.

The setup

Three Ubuntu 26.04 VMs (kernel 7.0, virtio-net) on an isolated Proxmox bridge: the box under test, an FRR peer, and a spare. I measured everything with tcpdump on the hypervisor bridge, not inside the guest. That ended up mattering a lot, more on that below.

Baseline: FRR 10.5.1 bfdd, one session at 3x10ms, and a stress-ng ladder on the box under test. Gaps are time between packets sent to the peer, in ms. Normal spacing is 7.5 to 10ms because BFD adds jitter on purpose.

Level	Load	p50	p99	max gap	Result
L1	4 CPU hogs	8.81	10.03	~10	fine
L2	8 CPU hogs + context switching	8.73	10.03	750	flapped
L3	CPU + timer pressure	8.82	10.16	970	44 flaps in 120s
L4	4 SCHED_FIFO hogs	8.82	287	960	~40% of packets never sent

Look at L3. 44 flaps in two minutes, and the p99 is 10.16ms. That’s a perfect number. The stalls that kill the session are rare, a handful per hundred seconds, so they never show up in a percentile. If you graph p99 packet spacing on a dashboard, this box looks healthy while it’s pulling routes every few seconds. Only the max shows it.

So I wrote a dumb daemon to prove the point

Before touching the kernel I wanted a second data point. I wrote the simplest BFD daemon I could: 160 lines, one thread, a recv() with a 2ms timeout as the clock, and the RFC state machine. The plan was to watch it die like bfdd did and then go build the XDP version.

It didn’t die. Under the exact L3 load that flapped bfdd 44 times, it had zero flaps and a 13ms p99. I ran it again because I didn’t believe it.

So why did my 160 lines survive when bfdd didn’t? After digging for a bit, it came down to how Linux picks what runs next. The normal scheduler keeps track of how much CPU each task has used, and when a task wakes up, the one that’s used the least gets to go first. My daemon woke up every 2ms, did a few microseconds of work and went back to sleep, so it had barely used any CPU. Every time it woke up, it got to cut in front of the stress processes almost immediately.

bfdd is a much bigger program. It’s one event loop handling zebra, config and a lot of timers, and the timer stress I was running hit exactly the part where it wakes up. So it wasn’t that userspace can’t do BFD. bfdd just wakes up in a way that this kind of load hurts.

Then I added real-time load

Next I tried the L4 load on it: four SCHED_FIFO hogs, one per vCPU. This time my daemon broke too, with 25 flaps in 60 seconds and a 324ms max gap. Real-time tasks always run before normal ones, so with all four cores busy my daemon only got a tiny slice of CPU each second, and the quick wakeups stopped mattering.

The weird part was the daemon’s log. It showed zero timeouts for the whole run, like the peer’s packets had arrived right on time. The capture on the hypervisor said the peer had seen 300ms of silence and dropped the session.

Here’s what happened. While my daemon wasn’t running, the peer’s packets were piling up in the socket buffer. When it got CPU again it read them all at once, and from its point of view everything had arrived fine. If I’d only had the daemon’s logs I would have missed this completely, which is why every timing and flap number in this series comes from a capture taken outside the box being tested.

So the next question was whether I could get packets out on time even when userspace can’t run at all. That’s part 2.

How my homelab is set up

w453y.me@gmail.com (Abdul Wasey) — Sat, 18 Apr 2026 00:00:00 GMT

Everything I self-host runs on two Proxmox nodes at home. There are 28 guests on them right now, half containers and half VMs: my mail, my Matrix server, my password manager, DNS, media, this site, a local LLM, and the lab I used for xdp-bfd. This is how it all fits together.

Here’s all of it in one picture, with each guest on the node it’s running on today:

Everything, and where it runs today. Click for the full size.

The two nodes

Both nodes are in one Proxmox cluster. Each one has its own ZFS pool for the host, but most guests live on storage that both nodes can see: one LUN over iSCSI, with a shared LVM volume group on it. That’s what lets me move a running guest from one node to the other. I do that by hand, though. The HA manager is running, but I haven’t set any guest up for HA.

Each node talks to the storage over four iSCSI paths, each on its own network. Multipath uses two of them at a time and keeps the other two as standby, and if every path drops for a moment, the guests’ disk I/O just waits instead of erroring out. I don’t bond any links at all. The storage paths are where the redundancy is.

Two nodes, one shared LUN, four paths from each node.

It’s not perfect. With only two nodes there’s no tiebreaker, so if one node goes down, the cluster loses quorum. And plain LVM on shared storage can’t do snapshots, so the guests on it don’t get one before I change something.

Containers or VMs

If something only needs a userspace, it’s a container: the reverse proxy, the whole DNS layer, Vaultwarden, Guacamole, monitoring and this site. If it needs its own kernel or a different OS, it’s a VM: the firewall, mail, Matrix, the media box, code-server, the LLM host and the eBPF lab. Two more VMs are a small test cluster of their own, on the same bridges as the real one.

Eleven of the guests run their software in Docker. Synapse and the DNS layer don’t. They run directly on their guests.

The firewall is one of the guests

My edge firewall is OPNsense, and it runs as a VM on the same cluster it routes for. The internet comes in as a tagged VLAN on the hosts’ uplink bridge, and OPNsense routes five internal networks from there: the main LAN, the cluster interconnect, a second private network, a VLAN for testing, and WireGuard, which is how my own devices and the offsite backup box get in. It also hands out DHCP, serves NTP, runs its own resolver, and keeps an OpenVPN tunnel open going out.

The firewall and the segments behind it.

Two kinds of traffic never go through it. The iSCSI paths go straight from each node to the storage, and the eBPF lab has two bridges with no physical ports at all, so its traffic only moves between VMs. The lab VMs are on the main LAN as well, though, so it’s the lab networks that are cut off, not the machines.

One proxy in front of everything

Every web service sits behind one nginx reverse proxy, which has about 70 hostnames on it now. It runs my own nginx image with a JWT module built in.

Most of my services have their own login, so the proxy just passes those through. A few admin pages don’t have one, and those need a valid token. One map over the client’s address, the hostname and the user agent decides which requests need a token, so the same page can be open from home and locked from outside. Getting that decision to happen at runtime meant changing the JWT module itself.

What happens to a request once it reaches the proxy.

The certificates are wildcards from Let’s Encrypt, and certbot gets them over DNS, so nothing has to be open to the internet for a renewal. The Proxmox web UI is behind the proxy too, as one upstream across both nodes, so if one node stops answering, I end up on the other.

DNS

I wanted filtering and full recursion for my own devices, and encrypted DNS with a different filter for anyone outside, so there are two Pi-holes. The internal one has unbound running on the same guest for recursion, and my devices just query it directly.

Encrypted DNS comes in through the proxy instead. A DNS over HTTPS request hits nginx, which picks one of two dnsdist listeners depending on where the request came from, and each listener hands it to one of the Pi-holes. DNS over TLS goes through two stunnel instances, one in front of each Pi-hole.

Plain queries go straight to the filtering resolver. Encrypted ones come in at the proxy, which decides which policy they get.

Mail, chat and the rest

Mail is Mailcow in its own VM, which is 18 containers by itself: Postfix and Dovecot, rspamd and ClamAV, a scanner for Office macros, and SOGo for webmail. Matrix is Synapse with PostgreSQL, plus coturn for calls, all installed straight on the VM, and registration is closed.

The rest is Vaultwarden, Guacamole, Stirling-PDF, code-server, an iperf3 endpoint, a local LLM on llama.cpp, and the media stack, which is Jellyfin, the *arr apps and qBittorrent. The media VM’s traffic goes out through a Cloudflare WARP tunnel with a kill switch, so if the tunnel drops, that traffic stops instead of going out directly.

Monitoring and backups

I only added monitoring on 10 September, long after most of what it watches. It’s Prometheus, Loki, Grafana and Alertmanager in one container, with agents on 19 hosts. Prometheus pulls from the agents instead of having them push, so if a host goes quiet, it shows up as down instead of just going silent. Blackbox probes check things from the outside too, and an alert fires three weeks before any certificate expires.

The alerts go to a private room on my own Matrix server. The obvious problem is that if the cluster goes down, the alerts go down with it.

The backups are a nightly vzdump on each node, then an rsync to an offsite box over WireGuard. That box isn’t always online, so the alert only fires when the last good sync is more than 48 hours old. The first thing the new monitoring found was that one node’s sync had been failing every night for 23 days, and that turned into a much longer story about TCP congestion control.

The nightly backups, and how I'd find out if they stopped.

Turning off JWT auth for my VPN in nginx

w453y.me@gmail.com (Abdul Wasey) — Tue, 16 Sep 2025 00:00:00 GMT

I run about thirty self-hosted services behind one nginx reverse proxy, and all of them sit behind a JWT auth gateway that logs you in with Google. For anything reachable from the internet that’s what I want. But most of the time I’m already on my own WireGuard VPN, and having to log in again just to open a dashboard got annoying.

So I wanted auth to stay on for everyone, except traffic coming from the VPN subnet. nginx has a geo module for exactly this kind of thing. It sets a variable based on the client’s address:

geo $jwt_enabled {
    default        on;
    10.10.10.0/20  off;   # VPN
}

The plan was to pass that to the JWT module with auth_jwt_enabled $jwt_enabled;. When I tried it, nginx refused to start, saying the value was invalid.

Why the variable didn’t work

In an nginx config it looks like variables work everywhere, but they only work in directives that were written to accept them. Each directive decides in its C code how its value gets read, and auth_jwt_enabled was set up as an on/off flag:

NGX_HTTP_MAIN_CONF | NGX_HTTP_SRV_CONF | NGX_HTTP_LOC_CONF | NGX_CONF_FLAG,
ngx_conf_set_flag_slot,

A flag like this gets read once, when nginx loads its config. It has to be exactly on or off, and it gets stored as a 1 or a 0. $jwt_enabled isn’t either of those, so nginx rejected it before any request came in.

Adding variable support

nginx already has a way for a directive to take a value that gets worked out per request, called a complex value. So I changed auth_jwt_enabled to use that instead of a flag. The config now stores the expression, and the module evaluates it for each request:

ngx_int_t enabled = 1;

if (jwtcf->enabled != NULL) {
  ngx_str_t cv;
  if (ngx_http_complex_value(r, jwtcf->enabled, &cv) == NGX_OK && cv.len > 0) {
    if (ngx_strncmp(cv.data, "off", 3) == 0) {
      enabled = 0;
    }
  }
}

Auth only turns off if the value is exactly off. If it’s empty, misspelled, or the geo block doesn’t match anything, auth stays on. Plain auth_jwt_enabled on; still works the same as before, since on is just a value that never changes.

Every site started asking for a token

After building my first version, every site on the box started asking for a JWT, including ones that had no JWT config at all.

With the old flag, “not set” had its own special value, and the module turned that into off. That’s why it did nothing on sites that never mentioned it. With my change the setting was a pointer, and “not set” was now just an empty pointer. My code started from enabled = 1 and only turned it off for an explicit off, so an empty pointer meant on.

I fixed it by adding a check at the start: if no JWT key is configured, the module steps aside. Without a key it can’t check tokens anyway.

Getting it merged

The maintainer liked the change and added some formatting fixes and tests. He also removed my key check, saying other checks further down already handled that case. I rebuilt with his changes and tried a config with no JWT settings, and nothing asked for a token, so I agreed to drop it.

For the tests, he couldn’t easily test a subnet in CI, so he set the variable from a request header instead:

map $http_test_auth_enabled $jwt_enabled {
    default  on;
    on       on;
    off      off;
}

A request with Test-Auth-Enabled: on should get a 401, and one with off should get a 200.

It was released in v2.4.0 about a month after I opened the PR, and my reverse proxy runs it now. On the VPN, dashboards open straight away, and from anywhere else I get the Google login.

Proxmox Disks page failing on a USB SSD

w453y.me@gmail.com (Abdul Wasey) — Sat, 08 Mar 2025 00:00:00 GMT

I plugged a 2.5” SSD into one of my Proxmox nodes through a USB-to-SATA adapter and opened Node → Disks in the web UI. It showed a spinner for a while and then Communication Failure.

The disk itself was working. lsblk listed it, fdisk could read it, and I mounted it and copied files onto it without any problem. Only the Disks page failed, and there was nothing about it in journalctl, syslog or the pvedaemon logs.

Finding the call that hung

Proxmox’s API is written in Perl, so I could read the code behind the Disks page directly on the node. The request goes from PVE/API2/Disks.pm into PVE/Diskmanage.pm, and for each disk it calls get_smart_data(), which runs smartctl:

if (!$nosmart) {
    eval {
        my $smartdata = get_smart_data($devpath, !is_ssdlike($type));
        $health = $smartdata->{health} if $smartdata->{health};
        # ...wearout for SSDs...
    };
}

The eval there catches errors, but it doesn’t help if the command is just slow, and smartctl was being run without any timeout. To check whether this was the problem, I emptied out that block, restarted pvedaemon and reloaded the page. The Disks page loaded instantly, so the SMART call was what was holding it up.

It wasn’t actually hanging

I filed bug #6224 with what I’d found. Fabian Grünbichler replied with two questions: does smartctl ever finish if you run it by hand, and how long does it take?

# time smartctl -a /dev/sdb
Read NVMe Identify Controller failed: scsi error unsupported field in scsi command

real    2m0.758s

It did finish, after two minutes. I’d been describing it as a hang, but smartctl was returning, just long after the web UI had given up waiting. That meant the fix was a timeout, not tracking down something stuck.

Daniel Kral worked out where the two minutes came from. smartmontools waits up to 60 seconds for a SCSI command by default, so that a disk in a big storage shelf has time to spin up. A USB-to-SATA adapter can take that whole 60 seconds to fail a command instead of failing it right away, and my two minutes was that timeout being hit twice.

The fix

The main change was adding a timeout to the smartctl call:

-	run_command($cmd, noerr => 1, outfunc => sub {
+	run_command($cmd, noerr => 1, timeout => 10, outfunc => sub {

Ten seconds, since a healthy drive answers almost immediately. Two smaller patches went in with it. One passes the real error up when the timeout hits, and the other shows the disk’s health as unknown in that case, because the web UI expected a health value and would otherwise get an error message instead.

The patches are in pve-storage now, though the bug itself is still marked PATCH AVAILABLE instead of resolved. I didn’t write them myself. My part was tracing the call, isolating it, and answering the maintainers’ questions with the timings they asked for.