Proxmox Disks page failing on a USB SSD

· proxmox, linux, debugging, storage

I plugged a 2.5” SSD into one of my Proxmox nodes through a USB-to-SATA adapter and opened Node → Disks in the web UI. It showed a spinner for a while and then Communication Failure.

The disk itself was working. lsblk listed it, fdisk could read it, and I mounted it and copied files onto it without any problem. Only the Disks page failed, and there was nothing about it in journalctl, syslog or the pvedaemon logs.

Finding the call that hung

Proxmox’s API is written in Perl, so I could read the code behind the Disks page directly on the node. The request goes from PVE/API2/Disks.pm into PVE/Diskmanage.pm, and for each disk it calls get_smart_data(), which runs smartctl:

if (!$nosmart) {
    eval {
        my $smartdata = get_smart_data($devpath, !is_ssdlike($type));
        $health = $smartdata->{health} if $smartdata->{health};
        # ...wearout for SSDs...
    };
}

The eval there catches errors, but it doesn’t help if the command is just slow, and smartctl was being run without any timeout. To check whether this was the problem, I emptied out that block, restarted pvedaemon and reloaded the page. The Disks page loaded instantly, so the SMART call was what was holding it up.

It wasn’t actually hanging

I filed bug #6224 with what I’d found. Fabian Grünbichler replied with two questions: does smartctl ever finish if you run it by hand, and how long does it take?

# time smartctl -a /dev/sdb
Read NVMe Identify Controller failed: scsi error unsupported field in scsi command

real    2m0.758s

It did finish, after two minutes. I’d been describing it as a hang, but smartctl was returning, just long after the web UI had given up waiting. That meant the fix was a timeout, not tracking down something stuck.

Daniel Kral worked out where the two minutes came from. smartmontools waits up to 60 seconds for a SCSI command by default, so that a disk in a big storage shelf has time to spin up. A USB-to-SATA adapter can take that whole 60 seconds to fail a command instead of failing it right away, and my two minutes was that timeout being hit twice.

The fix

The main change was adding a timeout to the smartctl call:

-	run_command($cmd, noerr => 1, outfunc => sub {
+	run_command($cmd, noerr => 1, timeout => 10, outfunc => sub {

Ten seconds, since a healthy drive answers almost immediately. Two smaller patches went in with it. One passes the real error up when the timeout hits, and the other shows the disk’s health as unknown in that case, because the web UI expected a health value and would otherwise get an error message instead.

The patches are in pve-storage now, though the bug itself is still marked PATCH AVAILABLE instead of resolved. I didn’t write them myself. My part was tracing the call, isolating it, and answering the maintainers’ questions with the timings they asked for.

Back to writing