A nightly backup filled my Proxmox node and froze the cluster filesystem
I woke up around 4:30 AM (yes, I do that sometimes, even after a sleepless night) and found my homelab down. None of my public services were reachable, and OPNsense wasn’t even answering pings. My first guess was a power cut, so I went to the room where the servers are and logged in on the console. Everything was powered on and running, and the uptime looked normal. Nothing had rebooted, so it wasn’t power. Then what was wrong? Okay, time to start debugging.
OPNsense runs as a VM on pve1, one of the two Proxmox nodes in my homelab, so pve1 was where I started. The first thing I checked was disk space, at 4:41:
root@pve1:~# df -h
Filesystem Size Used Avail Use% Mounted on
rpool/ROOT/pve-1 16G 16G 0 100% /
rpool 128K 128K 0 100% /rpool
rpool/var-lib-vz 265G 265G 0 100% /var/lib/vz
rpool/ROOT 128K 128K 0 100% /rpool/ROOT
rpool/data 128K 128K 0 100% /rpool/data
/dev/fuse 128M 56K 128M 1% /etc/pve
[...]That was bad. On pve1 the root filesystem, the local storage at /var/lib/vz
and the local-zfs VM disks are all datasets on one ZFS pool, rpool, and
every one of them showed 0 bytes available. So the whole pool was full, and I
had no idea what had filled it.
This post goes through how I got it back, in the order it happened and with the actual output, and then what caused it.
Where the space went
To find out what was using the space, I looked at the pool, the datasets, and
what was under /var/lib/vz:
root@pve1:~# zpool list -o name,size,alloc,free,cap,health
NAME SIZE ALLOC FREE CAP HEALTH
rpool 556G 539G 16.8G 96% ONLINE
root@pve1:~# zfs list -o name,used,avail,refer,mountpoint
NAME USED AVAIL REFER MOUNTPOINT
rpool 542G 0B 104K /rpool
rpool/ROOT/pve-1 15.1G 0B 15.1G /
rpool/backup/ccwork 13.4G 0B 13.4G none
rpool/data 250G 0B 96K /rpool/data
rpool/data/vm-100-disk-0 49.9G 0B 49.9G -
rpool/data/vm-109-disk-0 30.2G 0B 30.2G -
rpool/data/vm-112-disk-0 59.8G 0B 59.8G -
[...]
root@pve1:~# du -xhd1 /var/lib/vz 2>/dev/null | sort -h
22G /var/lib/vz/template
243G /var/lib/vz/dump
265G /var/lib/vzzpool list says 16.8G free while every dataset says 0B. That’s not a
contradiction: ZFS holds back about 1/32 of the pool as slop space so it can
still do its own bookkeeping when it’s full, and 556G / 32 is about 17G. As far
as anything writing files was concerned, the pool was full.
The VM disks under rpool/data were the size I expected. The odd one was
/var/lib/vz/dump at 243G. That’s where my nightly vzdump job writes its
backups, and it’s normally a fraction of that. So I listed it:
root@pve1:~# ls -lhS /var/lib/vz/dump | head
total 243G
-rw-r--r-- 1 root root 111G Oct 20 03:18 vzdump-qemu-110-2026_10_20-02_08_31.vma.zst
-rw-r--r-- 1 root root 103G Oct 21 03:18 vzdump-qemu-110-2026_10_21-02_08_39.vma.dat
-rw-r--r-- 1 root root 14G Oct 21 02:08 vzdump-qemu-109-2026_10_21-02_00_01.vma.zst
-rw-r--r-- 1 root root 12G Oct 20 03:29 vzdump-qemu-115-2026_10_20-03_19_08.vma.zst
[...]
drwxr-xr-x 2 root root 3 Oct 21 02:08 vzdump-qemu-110-2026_10_21-02_08_39.tmpAlmost all of it was two backups of VM 110: yesterday’s finished 111G archive,
and tonight’s, still an unfinished .vma.dat at 103G. Tonight’s file was last
modified at 03:18, and it wasn’t growing any more.
That surprised me. VM 110 was my main dev VM, and the day before I’d split its workloads out onto their own machines and deleted about 140G from it. Inside the guest the disk had gone from 88% used to 18%, so I expected a much smaller backup. I’ll come back to why it wasn’t.
So a backup had filled the pool, and it looked stuck. That still didn’t explain why OPNsense was down, but the stuck backup was the obvious next thing to look at.
Everything I touched hung
I tried to list the vzdump task for VM 110, and the command just hung:
root@pve1:~# pvenode task list --vmid 110 --typefilter vzdump --limit 10 ^C^C^C^C^C^C^C^C^C^C^C^C^C^CCtrl-C did nothing. A command that can’t be interrupted usually means it’s
stuck inside the kernel, so I looked at what the backup task’s process was
waiting on, its wchan:
root@pve1:~# ps -o pid,ppid,stat,wchan:32,etime,cmd -p 1922728
PID PPID STAT WCHAN ELAPSED CMD
1922728 1 Ss request_wait_answer 02:48:22 task UPID:pve1:001D56A8:1ADEF823:6AD7CF49:vzdump::root@pam:request_wait_answer is the FUSE wait: a process has sent a request to a FUSE
filesystem and is waiting for the daemon to answer. On a Proxmox node the FUSE
filesystem is /etc/pve, served by pmxcfs. If pmxcfs wasn’t answering, other
Proxmox processes should be stuck too, so I checked all of them:
root@pve1:~# ps -eo pid,ppid,stat,wchan:28,etime,comm,args \
| grep -E 'pmxcfs|pvedaemon|pvestatd|vzdump|task UPID'
5506 1 Ssl futex_do_wait 52-07:07:02 pmxcfs /usr/bin/pmxcfs
5859 1 Ds path_openat 52-07:06:58 pvestatd pvestatd
5888 1 Ss hrtimer_nanosleep 52-07:06:58 pvedaemon pvedaemon
1873863 5888 D filename_create 03:38:10 pvedaemon worke pvedaemon worker
1917466 5888 D filename_create 02:57:10 pvedaemon worke pvedaemon worker
1922491 5888 D filename_create 02:51:10 pvedaemon worke pvedaemon worker
1922728 1 Ss request_wait_answer 02:50:54 task UPID:pve1: task UPID:pve1:001D56A8:1ADEF823:6AD7CF49:vzdump::root@pam:
2015252 2013969 D+ filename_create 05:50 pvenode /usr/bin/perl -T /usr/bin/pvenode task list --vmid 110 --typefilter vzdump --limit 10They were. pvestatd, three pvedaemon workers and my own pvenode were all
in D state, uninterruptible sleep, which is why Ctrl-C couldn’t kill them. The
oldest worker had been stuck for over three and a half hours. Their kernel
stacks showed all the workers stuck in mkdir():
root@pve1:~# cat /proc/1873863/stack
[<0>] filename_create+0xb4/0x170
[<0>] filename_mkdirat+0x61/0x210
[<0>] __x64_sys_mkdir+0x30/0x50
[<0>] x64_sys_call+0x958/0x2390
[<0>] do_syscall_64+0x10b/0x14e0
[<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7eand pvestatd stuck opening a file. Anything that went through the Proxmox API
or touched /etc/pve joined the queue. From here on I wrapped every Proxmox
command in timeout, and pvesh get /nodes/pve1/status came back Killed
every time. Even strace on a worker got killed by its timeout without printing
anything.
Freeing space
I didn’t know yet why everything was waiting, but the full pool was the one thing I knew was wrong, so I wanted space back before digging further. Yesterday’s VM 110 archive was the easy candidate. First I made sure nothing had it open:
root@pve1:~# lsof /var/lib/vz/dump/vzdump-qemu-110-2026_10_20-02_08_31.vma.zst
root@pve1:~#Nothing did, so I deleted it. The data that matters on VM 110 is backed up elsewhere anyway (the nightly backups also go offsite), and tonight’s copy was clearly not going to finish. Then I checked the space and tried the stuck commands again:
root@pve1:~# rm -- /var/lib/vz/dump/vzdump-qemu-110-2026_10_20-02_08_31.vma.zst
root@pve1:~# zfs list -o name,used,available rpool rpool/ROOT/pve-1 rpool/var-lib-vz rpool/data
NAME USED AVAIL
rpool 464G 78.1G
rpool/ROOT/pve-1 15.1G 78.1G
rpool/data 250G 78.1G
rpool/var-lib-vz 152G 78.1G
root@pve1:~# df -h / /var/lib/vz
Filesystem Size Used Avail Use% Mounted on
rpool/ROOT/pve-1 94G 16G 79G 17% /
rpool/var-lib-vz 230G 152G 79G 67% /var/lib/vz
root@pve1:~# timeout -k 2 10 pvesh get /nodes/pve1/status
timeout -k 2 10 pvenode task list --vmid 110 --typefilter vzdump --limit 10
Killed
Killed78G free, and nothing got unstuck. So the full pool might have started this, but whatever the processes were waiting on wasn’t going to finish just because there was room again. I had to find out what they were actually waiting for.
It was pmxcfs
The stacks I had so far only showed where each process was parked, not what it
was waiting for. To get the full picture, I had the kernel dump every blocked
task with echo w > /proc/sysrq-trigger and read it back from the journal. The
dump was huge. The part that mattered was that pvestatd and the workers were
all queued behind a lock on a directory, not on any I/O:
root@pve1:~# echo w > /proc/sysrq-trigger
root@pve1:~# journalctl -k --since "10 minutes ago" --no-pager | grep -E -B 8 -A 35 'task:(pvedaemon|pvestatd|pvenode|pvesh|pvescheduler|pmxcfs)'
[...]
Oct 21 04:57:57 pve1 kernel: task:pvestatd state:D stack:0 pid:5859 tgid:5859 ppid:1 task_flags:0x400140 flags:0x00080000
Oct 21 04:57:57 pve1 kernel: Call Trace:
Oct 21 04:57:57 pve1 kernel: <TASK>
Oct 21 04:57:57 pve1 kernel: __schedule+0x49a/0x1780
Oct 21 04:57:57 pve1 kernel: schedule+0x27/0xb0
Oct 21 04:57:57 pve1 kernel: schedule_preempt_disabled+0x15/0x30
Oct 21 04:57:57 pve1 kernel: rwsem_down_read_slowpath+0x271/0x5a0
Oct 21 04:57:57 pve1 kernel: down_read+0x48/0xc0
Oct 21 04:57:57 pve1 kernel: path_openat+0x5fa/0x13f0
[...]The one process that had actually got into /etc/pve was a pvescheduler
doing a mkdir, and it was waiting for an answer from the FUSE daemon:
Oct 21 04:57:58 pve1 kernel: task:pvescheduler state:D stack:0 pid:1979912 tgid:1979912 ppid:6647 task_flags:0x400040 flags:0x00080002
Oct 21 04:57:58 pve1 kernel: Call Trace:
Oct 21 04:57:58 pve1 kernel: <TASK>
Oct 21 04:57:58 pve1 kernel: __schedule+0x49a/0x1780
Oct 21 04:57:58 pve1 kernel: schedule+0x27/0xb0
Oct 21 04:57:58 pve1 kernel: request_wait_answer+0xd8/0x270
Oct 21 04:57:58 pve1 kernel: __fuse_simple_request+0x11f/0x310
Oct 21 04:57:58 pve1 kernel: create_new_entry+0xfc/0x400
Oct 21 04:57:58 pve1 kernel: fuse_mkdir+0xd1/0x110
Oct 21 04:57:58 pve1 kernel: vfs_mkdir+0xc4/0x240
Oct 21 04:57:58 pve1 kernel: filename_mkdirat+0x1bf/0x210
Oct 21 04:57:58 pve1 kernel: __x64_sys_mkdir+0x30/0x50
[...]It held the directory lock while it waited, so every other mkdir and open
on that path queued behind it. FUSE also tells you how many requests are sitting
unanswered:
root@pve1:~# find /sys/fs/fuse/connections -maxdepth 2 -type f -name waiting -print -exec cat {} \;
/sys/fs/fuse/connections/48/waiting
3
/sys/fs/fuse/connections/50/waiting
0Three requests that pmxcfs had never answered. So pmxcfs was alive, it just
wasn’t replying. Its own log said why:
root@pve1:~# journalctl -u pve-cluster --since "4 hours ago" --no-pager | tail -100
[...]
Oct 21 02:49:21 pve1 pmxcfs[5506]: [dcdb] notice: data verification successful
Oct 21 03:18:56 pve1 pmxcfs[5506]: [database] crit: commit transaction failed: database or disk is full#010
Oct 21 03:18:56 pve1 pmxcfs[5506]: [database] crit: rollback transaction failed: cannot rollback - no transaction is active#010
Oct 21 03:18:56 pve1 pmxcfs[5506]: [dcdb] crit: serious internal error - stop cluster connection
Oct 21 03:18:57 pve1 pmxcfs[5506]: [dcdb] crit: can't initialize servicepmxcfs keeps the cluster configuration in an SQLite database at
/var/lib/pve-cluster/config.db, on the root filesystem. At 03:18:56, the same
minute the backup stopped growing, a commit failed with database or disk is full. pmxcfs dropped its cluster connection and stopped answering, while
staying mounted, so every request into /etc/pve after that waited forever.
The database files hadn’t been touched since:
root@pve1:~# ls -lah /var/lib/pve-cluster/
[...]
-rw------- 1 root root 92K Oct 21 03:14 config.db
-rw------- 1 root root 32K Oct 21 03:18 config.db-shm
-rw------- 1 root root 4.0M Oct 21 03:18 config.db-walFreeing space didn’t help because pmxcfs had already given up. It needed a restart.
Restarting pve-cluster
Restarting the cluster filesystem on a node with a stuck backup and a VM with
lock: backup sounded like a good way to make things worse, so first I checked
that the rest of the cluster was fine. By now the space I’d freed had grown to
112G as ZFS finished releasing it, and corosync on pve1 still had quorum:
root@pve1:~# corosync-quorumtool -s
Quorum information
------------------
Date: Wed Oct 21 05:02:45 2026
Quorum provider: corosync_votequorum
Nodes: 2
Node ID: 1
Ring ID: 1.1d1
Quorate: Yes
[...]
Membership information
----------------------
Nodeid Votes Name
1 1 pve1 (local)
2 1 pve2pve2 agreed when I asked it over SSH, with pve-cluster and corosync both
active. I also wanted to be sure the stuck task wasn’t half of a migration
that pve2 was waiting on. It wasn’t:
root@pve1:~# ssh root@<pve2> 'qm status 110'
Configuration file 'nodes/pve2/qemu-server/110.conf' does not existSo the restart would just make pve1’s pmxcfs throw away its stuck state and
sync from pve2. I used --no-block so my shell wouldn’t end up in the queue
too:
root@pve1:~# systemctl restart --no-block pve-cluster
root@pve1:~# journalctl -u pve-cluster --since "2 minutes ago" --no-pager
Oct 21 05:08:37 pve1 systemd[1]: Stopping pve-cluster.service - The Proxmox VE cluster filesystem...
Oct 21 05:08:48 pve1 systemd[1]: pve-cluster.service: State 'stop-sigterm' timed out. Killing.
[...]
Oct 21 05:08:48 pve1 pmxcfs[2028894]: [dcdb] notice: leader is 2/5348
Oct 21 05:08:48 pve1 pmxcfs[2028894]: [dcdb] notice: synced members: 2/5348
Oct 21 05:08:48 pve1 pmxcfs[2028894]: [dcdb] notice: waiting for updates from leader
Oct 21 05:08:48 pve1 pmxcfs[2028894]: [status] notice: all data is up to date
Oct 21 05:08:48 pve1 pmxcfs[2028894]: [dcdb] notice: update complete - trying to commit (got 2 inode updates)
Oct 21 05:08:48 pve1 pmxcfs[2028894]: [dcdb] notice: all data is up to date
Oct 21 05:08:49 pve1 systemd[1]: Started pve-cluster.service - The Proxmox VE cluster filesystem.The old pmxcfs didn’t stop on SIGTERM and got killed after 11 seconds. The new one took pve2 as leader and was up to date in under a second. Everything that had been hanging came back:
root@pve1:~# systemctl is-active pve-cluster pvedaemon pvestatd pveproxy corosync
active
active
active
active
active
root@pve1:~# qm status 110
status: running
root@pve1:~# for f in /sys/fs/fuse/connections/*/waiting; do printf '%s: ' "$f"; cat "$f"; done
/sys/fs/fuse/connections/48/waiting: 0
/sys/fs/fuse/connections/50/waiting: 0The backup job kept going
The stuck vzdump task was still around, so I read its log to see what it had
done. With /etc/pve answering again, it had finally got its error back, after
almost three hours:
root@pve1:~# pvenode task log 'UPID:pve1:001D56A8:1ADEF823:6AD7CF49:vzdump::root@pam:' | tail -60
[...]
INFO: 93% (238.3 GiB of 256.0 GiB) in 1h 9m 57s, read: 108.9 MiB/s, write: 101.2 MiB/s
ERROR: vma_queue_write: write error - Broken pipe
INFO: aborting backup job
INFO: resuming VM again
unable to open file '/etc/pve/nodes/pve1/qemu-server/110.conf.tmp.1922728' - Transport endpoint is not connected
ERROR: Backup of VM 110 failed - vma_queue_write: write error - Broken pipe
INFO: Failed at 2026-10-21 05:08:48
INFO: Starting Backup of VM 115 (qemu)
INFO: Backup started at 2026-10-21 05:08:51Transport endpoint is not connected is what you get from a FUSE mount whose
daemon has gone away, which is exactly what the old pmxcfs did when it was
killed. vzdump cleaned up the 103G partial file, which is why free space jumped
to 178G a minute later. Then it moved on to the next VM in the job, mailcow
(115), and started writing a fresh archive to the same disk at 5 AM. I stopped
the job from the GUI:
root@pve1:~# pvenode task log 'UPID:pve1:001D56A8:1ADEF823:6AD7CF49:vzdump::root@pam:' | tail -25
[...]
INFO: 18% (11.5 GiB of 64.0 GiB) in 3m 22s, read: 74.4 MiB/s, write: 73.9 MiB/s
ERROR: interrupted by signal
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 115 failed - interrupted by signal
INFO: Failed at 2026-10-21 05:12:21VM 110 still had the lock: backup the failed backup left behind, so the last
cleanup was qm unlock 110:
root@pve1:~# qm unlock 110
root@pve1:~# grep -n '^lock:' /etc/pve/qemu-server/110.conf || echo 'No lock'
No lock
root@pve1:~# qm status 110
status: runningFive VMs were paused
At this point the Proxmox side looked healthy, but my services were still down. Something was still wrong with five VMs on pve1: 100, 109, 112, 124 and 300. So I checked their status:
root@pve1:~# for id in 100 109 112 124 300; do printf '%s ' "$id"; qm status "$id"; done
100 status: io-error
109 status: io-error
112 status: io-error
124 status: io-error
300 status: io-errorThis was the answer to how the morning started. VM 100 is OPNsense, my edge firewall. It had been paused since the pool filled, which is why nothing was reachable from outside and why it didn’t answer my pings, while the node itself looked fine from the console.
To see why only these five, I looked at where their disks live:
root@pve1:~# grep '^scsi0' /etc/pve/qemu-server/{100,109,112,124,300}.conf
/etc/pve/qemu-server/100.conf:scsi0: local-zfs:vm-100-disk-0,iothread=1,size=64G
/etc/pve/qemu-server/109.conf:scsi0: local-zfs:vm-109-disk-0,iothread=1,size=64G
/etc/pve/qemu-server/112.conf:scsi0: local-zfs:vm-112-disk-0,iothread=1,size=8T
/etc/pve/qemu-server/124.conf:scsi0: local-zfs:vm-124-disk-0,size=30G
/etc/pve/qemu-server/300.conf:scsi0: local-zfs:vm-300-disk-0,size=120GAll on local-zfs, which is rpool/data on the same pool, set up with
sparse 1. A sparse zvol only takes space as the guest writes to it, so when
the pool filled, the guests’ writes had nowhere to go. QEMU’s default for a
write error is to pause the VM instead of passing the error to the guest, and
qm status --verbose shows that as the QMP status:
root@pve1:~# qm status 100 --verbose
[...]
pid: 6008
[...]
qmpstatus: io-error
running-machine: pc-i440fx-11.0+pve0
running-qemu: 11.0.3
status: running
vmid: 100The important part was status: running. The VMs were paused, not dead. Their
QEMU processes were all still alive, some with uptimes of weeks:
root@pve1:~# ps -p 6008,6091,13131,306324,3350489 -o pid,stat,etime,cmd
PID STAT ELAPSED CMD
6008 Sl 52-07:34:22 /usr/bin/kvm -id 100 -name OPNsense-11-1
6091 Sl 52-07:34:20 /usr/bin/kvm -id 109 -name matrix-11-18
13131 Sl 52-07:23:14 /usr/bin/kvm -id 112 -name pve1 -no-sh
306324 Sl 1-06:15:50 /usr/bin/kvm -id 124 -name xdpfw-11-60
3350489 Sl 15-04:03:19 /usr/bin/kvm -id 300 [...]Everything on the shared iSCSI storage, including all the new machines I’d just
moved VM 110’s workloads to, was fine, since none of it touches rpool.
A paused VM can be resumed, so there was no need to restart anything. Before un-pausing anything I checked that the pool itself had taken no damage and that there was plenty of room:
root@pve1:~# df -h /
Filesystem Size Used Avail Use% Mounted on
rpool/ROOT/pve-1 233G 16G 218G 7% /
root@pve1:~# zpool status rpool
pool: rpool
state: ONLINE
[...]
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
scsi-35000c500f01f4fbf-part3 ONLINE 0 0 0
scsi-35000c500f01fe763-part3 ONLINE 0 0 0
errors: No known data errors(zpool status also suggested a zpool upgrade for some feature flags. That’s
unrelated, and not something to do at 5 AM on a pool that just filled up, so I
left it.)
Then qm resume, one VM at a time, starting with the two I cared about least:
root@pve1:~# qm resume 124
qm status 124 --verbose | grep -E '^(status|qmpstatus|pid):'
pid: 306324
qmpstatus: running
status: running
root@pve1:~# qm resume 300
qm status 300 --verbose | grep -E '^(status|qmpstatus|pid):'
pid: 3350489
qmpstatus: running
status: runningSame PIDs as before, so these were the same processes carrying on, not restarts. Then the rest:
root@pve1:~# qm resume 109
pid: 6091
qmpstatus: running
status: running
root@pve1:~# qm resume 112
pid: 13131
qmpstatus: running
status: running
root@pve1:~# qm resume 100
pid: 6008
qmpstatus: running
status: runningAt 5:21, 40 minutes after the first df -h, everything was running again.
Nothing had been rebooted: not the node, not the VMs, not the containers.
Why it happened
That fixed the morning, but I still didn’t know why a VM I’d just emptied had produced a 110G backup, so later that day I went back through the logs and the config.
The nightly job on pve1 runs at 02:00, writes to local (rpool/var-lib-vz),
compresses with zstd and keeps one backup per guest:
root@pve1:~# pvenode task log 'UPID:pve1:001D56A8:1ADEF823:6AD7CF49:vzdump::root@pam:' | head -1
INFO: starting new backup job: vzdump 101 103 126 106 107 108 109 114 115 121 105 110 [...] --mode snapshot --prune-backups 'keep-last=1' --notes-template '{{guestname}}' --storage local --quiet 1 --compress zstdThree things added up.
VM 110’s backup didn’t shrink when I emptied it. Its 256G disk sits on
thick LVM on the shared storage, and no discard reaches it, so the blocks I’d
“deleted” inside the guest still held their old data. vzdump reads the whole
block device. It doesn’t know what the guest’s filesystem considers free, so
140G of dead build trees compressed into the archive just like live data. The
previous night’s archive was archive file size: 110.12GB, and tonight’s was
heading for the same.
keep-last=1 deletes the old backup after the new one is done. You can see
the order in the log for VM 109 earlier that night:
root@pve1:~# pvenode task log 'UPID:pve1:001D56A8:1ADEF823:6AD7CF49:vzdump::root@pam:'
[...]
INFO: archive file size: 13.31GB
INFO: adding notes to backup
INFO: prune older backups with retention: keep-last=1
INFO: removing backup 'local:backup/vzdump-qemu-109-2026_10_20-02_00_03.vma.zst'
INFO: pruned 1 backup(s) not covered by keep-retention policySo for VM 110 the peak was two 110G archives side by side, about 220G, on top of everything else on the pool. That’s the 243G I found.
I’d shared that pool with even more. The 13.4G rpool/backup/ccwork in the
first zfs list was new. It’s a replica of the container that backs up those new
machines, and I’d started replicating it to pve1 at 16:51 the afternoon
before, into the same pool as the node’s root filesystem, its VM disks and the
backup dump. It wasn’t the trigger, but it was 13.4G less headroom. Its own
replication run at 03:30 failed because the pool was already full:
cannot receive incremental stream: out of space
cannot send 'rpool/ccwork': I/O errorZFS rejects a failed receive as a whole, so that run added nothing. But the status file for that job still showed the previous day’s success, so nothing warned anyone.
Once the pool hit zero, everything that shared it failed in its own way:
pmxcfs couldn’t commit and wedged /etc/pve, which took every Proxmox tool
with it, and the five VMs on sparse zvols paused.
What I changed
- VM 110 is out of the nightly vzdump job. Everything that matters on it is covered by the hourly and nightly backups of that backup container, so a 110G image of mostly dead blocks isn’t worth it. Without it, the rest of the job peaks at about 70G on pve1.
- The root filesystem has a reservation.
rpool/ROOT/pve-1now hasreservation=25Gon pve1 and 22G on pve2, so even if the pool fills, pmxcfs still has room to commit. That one setting would have avoided the whole frozen-cluster part of the night. - The replication script records failures. A failed run now writes
FAILED at line Nto its status instead of leaving yesterday’s success there. - Discard on the VM disks. None of the
local-zfsdisks haddiscard=on, so space freed inside those guests never went back to the pool either. OPNsense’s 64G disk had 50G “used”. Turning it on needs a stop and start per VM, so that’s still pending. - Space. VM 300 was an old lab VM I didn’t need any more. Before deleting
it, I copied off about 27MB of work that only existed there and checked the
copy. That freed 70G. I also moved mailcow and matrix to the shared
storage. pve1’s
rpoolwent from 0 bytes to 382G free, and pve2 has 418G.
The main thing I took from it: “the guest says 18% used” tells you nothing about
backup size on thick storage without TRIM, and keep-last=1 means you need room
for two copies of your biggest backup, on whatever storage the job writes to.
Mine was the same pool as the hypervisor.