Fixing the MTU

On a random morning I got a text from my friend Daryll Swer, asking how many bytes mqvpn adds to each packet. mqvpn is a multipath VPN that runs over QUIC, using xquic underneath. Like any VPN it adds its own headers, so packets inside the tunnel have to be a bit smaller than the link’s MTU. If it gets that wrong, packets get fragmented or just dropped.

I thought I’d measure it in an afternoon and send him a number. The number was quick, but it led to a few bugs, and fixing those took two weeks.

Measuring it

I set up two network namespaces joined by a 1500-byte veth, ran the mqvpn server in one and the client in the other, and sent don’t-fragment pings of different sizes through the tunnel while capturing the real packets on the veth:

for s in 72 472 972 1272 1354 1355; do
  ip netns exec mqc ping -M do -c 3 -s $s 10.8.0.1
done

Every packet on the wire was the packet inside the tunnel plus 62 bytes on IPv4, and 82 on IPv6. That’s the UDP and IP headers, a 13-byte QUIC header, the 16-byte AEAD tag, and 5 bytes of framing. So that was the answer to Daryll’s question.

The part that bugged me was the tunnel’s MTU. It was fixed at 1382, and with 82 bytes of overhead a 1500 link has room for 1418 even on IPv6. So mqvpn was leaving 36 bytes unused on every packet. In the source, every QUIC packet was hard-coded to 1400 bytes. That had been set when path MTU discovery was turned off a while back, because enabling it made upload about 10 times slower (mqvpn#7). A fixed 1400 was the safe choice, but I wanted to know if those 36 bytes could be used.

The extra 16 bytes

I raised the packet size to 1436 in mqvpn and xquic, so the TUN would be 1418 and packets on the wire exactly 1500 bytes. A ping test looked right, 1418 went through and 1419 didn’t, so I thought that was it.

Then I ran iperf3 through the tunnel to make sure it held up under load. The throughput was fine, but the capture had 15,648 fragments in it. The ones I looked at were 1508-byte packets, 8 bytes too big for the link. All of them came from the server, which is the side that sends data and acknowledges the client’s data at the same time.

I wanted to know where those 8 bytes came from. Even counting the 16-byte tag, the QUIC packets were coming out bigger than the size I’d set, so I searched xquic for anything else that added 16 bytes and found this:

size_t
xqc_get_po_remained_size_with_ack_spc(xqc_packet_out_t *po)
{
    return xqc_get_po_remained_size(po) + XQC_ACK_SPACE;   /* 16 */
}

When xquic is about to send a packet, it can add an ACK to it at the last moment, even if the packet is already full of data. To make room for that, every packet keeps 16 bytes spare that only the ACK can use. So a packet could be the size you set, plus 16 bytes of ACK, plus the 16-byte tag. With mqvpn’s old 1400 there was enough room left over that it never mattered, but at 1436 it went over.

I fixed it in xquic#86 by not letting that ACK room go past the real maximum. A full 1436 packet just puts its ACK in the next packet instead. Then I added a PathMTU option to mqvpn (#391, #409) that works out the packet size from your link’s MTU. A 1500 link now gets a 1418 TUN, up from 1382, with no fragments.

That still meant you had to know your link’s MTU and type it in. Path MTU discovery is supposed to figure that out by itself, but it was off because of #7. So that was next.

Why PMTUD made upload slow

PMTUD in xquic works by sending probe packets bigger than the current size. If a probe gets acknowledged, the path can take that size.

I set up a test path with netem at 100 Mbit/s and a 40 ms round trip and turned PMTUD on, expecting upload to crash like #7 said. It didn’t. Upload ran at 85-88 Mbit/s, the same as with PMTUD off, and I got the same thing with every MTU I tried.

It turns out that by default, Linux just fragments a packet that’s too big for the link. So the probes went out as fragments, got put back together on the other side, and got acked. No probe was ever lost, every size looked fine, and the probing wasn’t finding anything.

A lot of real networks drop fragments, so I made my test path do that too, with an nftables rule. With a 1450 link and only the probes too big for it:

With 24 lost probes, CUBIC dropped to 27.5 Mbit/s while BBR2 didn’t care. Since only CUBIC slowed down, it had to be something about congestion control, and sure enough, xquic’s loss detection counted a lost probe exactly like a lost data packet. So CUBIC took every lost probe as congestion and cut its speed, every 500 ms, even though a lost probe only means it was too big.

That got fixed in xquic#93, which carries my tests. Lost probes are now just dropped without counting as loss, and CUBIC is back to 89.8 Mbit/s.

The ICMP errors that never arrived

Next I wanted to see what happens when two clients on different links use the same server: one behind a 1400 link, one on 1500. For this I used a change that lets the server know each client’s size (now mqvpn#440). When a packet from the internet is too big for the 1400 client, the server should send back an ICMP “fragmentation needed” error, so the sender shrinks its packets.

The 1400 client got nothing at all, 0 Mbit/s. A ping from the sender didn’t get a reply or an error. But the server’s log said it was sending the error, once a second:

[DBG] [svr] sent ICMP Fragmentation Needed (mtu=1302) to TUN

So the error was being made and then disappearing somewhere on the server.

Here’s what was happening. The server writes these ICMP errors into its TUN with its own TUN address as the source. Linux sees a packet arrive on an interface claiming to come from one of its own addresses, decides it must be spoofed, and drops it. Linux calls those martian packets. The only way to allow it is accept_local on that interface, so I turned it on and ran the test again:

sysctl -w net.ipv4.conf.mqvpn0.accept_local=1

The ping got its “message too long” error back, and the client went from 0 to 299.9 Mbit/s. It was the same for the other errors the server sends, so traceroute had never shown the server hop either. IPv6 never had the problem. mqvpn now sets that sysctl itself (#427).

PMTUD without ICMP

That fix only helps if ICMP makes it back, and a lot of networks block it. The nice thing about xquic’s PMTUD is that it doesn’t need ICMP at all, since it only trusts probes that get acknowledged. So in mqvpn#430 I turned it back on as an option, starting from a safe 1280 TUN and probing up from there. I also set DF on mqvpn’s sockets, so Linux can’t fragment a probe and hide the result like it did in my first test.

I ran it over links of different sizes, with ICMP and fragments blocked in different combinations:

And #7 stayed fixed: 86-90 Mbit/s with PMTUD on or off.

When the link shrinks mid-session

All of that finds the right size when a connection starts, but I was more worried about the size changing later. Say a session is running fine at 1500, then something on the path drops to 1400 and ICMP is blocked. When I tried that, the tunnel went to 0 Mbit/s and stayed there. The big packets were being dropped, and nothing ever told mqvpn to send smaller ones.

My first try at detecting this was simple. If 3 big packets got lost with no big packet acked in between, I assumed the path had got smaller. To make sure it didn’t fire by mistake, I also ran the same test without shrinking anything, and it fired twice.

I tried three tweaks to the counting, and every one of them still fired when nothing was wrong. When I went through the logs, the round-trip time was about 206 ms in all of them, on a path I’d set to 40. The queue in my test path was filling up while CUBIC ramped up, and when it overflowed it dropped a burst of packets, mostly big ones. So going by losses alone, a full queue looked just like a black hole.

So I stopped trying to guess from losses. In the version that went into xquic#100, lost packets only make xquic suspicious. It then checks by sending 3 probes at the current size, each followed by a small packet and a round trip apart. A full queue lets at least one of them through, and a black hole doesn’t. That fixed the false alarms, 0 in 15 runs, and it caught every real shrink.

That worked with CUBIC, but with BBR2, which is mqvpn’s default, the tunnel still stalled after the shrink. This was a 1450 to 1400 shrink:

bbr2   per-sec Mbit: 104,94,81,80,0,0,0,0,0,0,0,0,0,0
cubic  per-sec Mbit: 103,98,91,80,35,90,90,91,90,94,91,80,90,91

To see why, I added a log for every old-size packet sent after the shrink. They were going out 37 at a time, seconds apart. BBR2 paces its sending instead of sending in bursts, so it had over a thousand packets queued up at the old size, all of them going to be lost, and new packets were stuck behind them.

So now, when xquic confirms a black hole, it throws out every old-size packet, including the ones already sent. Stream data gets split again into packets of the new size, and datagrams are dropped, since QUIC never resends those anyway. In a test that started dropping everything over 1300 bytes in the middle of a 20 MB transfer, it re-split about 2350 packets and the data came out fine.

With all of that, a 1500 to 1400 drop with ICMP blocked costs an upload about 1.5 seconds with CUBIC and 3 with BBR2, and then the tunnel carries on at about 91 Mbit/s.

So the answer to Daryll’s question is still 62 bytes on IPv4 and 82 on IPv6. What changed is that mqvpn now uses the rest of the space. On a 1500 link the TUN is 1418 instead of 1382, with no fragments. With the PMTUD option on, it finds the right size by itself, even with ICMP blocked, and if the path shrinks later it notices and comes back down.

That took two weeks and nine merged PRs, the last one being xquic#110, which catches a path that shrinks by only a few bytes. mqvpn#440, which lets clients on different links share one server with the right size for each, is still in review.