- Fixing a Degraded ZFS Mirror: Reseat, Resilver, and Scrub
- Part 3 of 3 in my "First ZFS Degradation" series. See also Part 1: The Setup and Part 2: Diagnosing the Problem.
- The Game Plan
- By now I was pretty confident about what was wrong: not a dying drive, but a flaky SATA connection. The fix should be straightforward. Just take the drive offline, shut down, reseat the cables, bring it back up, and let ZFS heal itself.
- But I wanted to do this methodically. ZFS is forgiving, but I didn't want to make things worse by rushing.
- Here was my plan:
-
- - Take the faulty drive offline in ZFS (tell ZFS "stop trying to use this drive")
- - Power down the server
- - Open the case, inspect and reseat cables
- - Boot up, verify the drive is detected
- - Bring the drive back online in ZFS
- - Let the resilver complete
- - Run a scrub to verify data integrity
- - Check SMART one more time
-
- Let's walk through each step.
- Step 1: Taking the Drive Offline
- Before touching hardware, I wanted ZFS to stop trying to use the problematic drive.
- First, I set up some variables to avoid typos with that long disk ID:
- DISKID="ata-ST4000NT001-3M2101_WX11TN0Z"
-DISKPATH="/dev/disk/by-id/$DISKID"
- Then I took it offline:
- zpool offline proxmox-tank-1 "$DISKID"
- Checking the status afterward:
- zpool status -v proxmox-tank-1
- NAME STATE READ WRITE CKSUM
- proxmox-tank-1 DEGRADED 0 0 0
- mirror-0 DEGRADED 0 0 0
- ata-ST4000NT001-3M2101_WX11TN0Z OFFLINE 108 639 129
- ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0
- The state changed from FAULTED to OFFLINE. ZFS knows I intentionally took it offline rather than it failing on its own. The error counts are still there as a historical record, but ZFS isn't actively trying to use the drive anymore.
- Time to shut down and get my hands dirty.
- Step 2: Opening the Case
- I powered down the server and opened up the Fractal Node 804. This case has a lovely design with drive bays accessible from the side, which I love. No reaching out into weird corners in the case, just unscrew a couple screws, slide the drive bay out and there they are, handy and reachable.
- I located AGAPITO1 (I had handwritten labels on the drives, lesson learned after many sessions of playing "which drive is which") and inspected the connections.
- Here's the honest truth: everything looked fine. The SATA data cable was plugged in. The power connector was plugged in. Nothing was obviously loose or damaged. There was a bit of tension in the cable as it moved from one area of the case (where the motherboard is) to the drives area, but I really didn't think that was affecting the connection to either the drive or the motherboard itself.
- But "looks fine" doesn't mean "is fine". So I did a full reseat:
-
- - Unplugged and firmly replugged the SATA data cable at both ends (drive and motherboard).
- - Unplugged and firmly replugged the power connector.
- - While I was in there, checked the connections on the other disk of the mirror as well.
-
- I made sure each connector clicked in solidly. Then I closed up the case and hit the power button.
- Step 3: Verifying Detection
- The server booted up. Would Linux see the drive?
- ls -l /dev/disk/by-id/ | grep WX11TN0Z
- lrwxrwxrwx 1 root root 9 Jan 2 23:15 ata-ST4000NT001-3M2101_WX11TN0Z -> ../../sdb
- The drive was there, mapped to /dev/sdb.
- I opened a second terminal and started watching the kernel log in real time:
- dmesg -Tw
- This would show me immediately if the connection started acting flaky again. For now, it was quiet, showing just normal boot messages, the drive being detected successfully, etc. Nothing alarming.
- Step 4: Bringing It Back Online
- Moment of truth. I told ZFS to start using the drive again:
- zpool online proxmox-tank-1 "$DISKID"
- Immediately checked the status:
- zpool status -v proxmox-tank-1
- pool: proxmox-tank-1
- state: DEGRADED
-status: One or more devices is currently being resilvered.
-action: Wait for the resilver to complete.
- scan: resilver in progress since Fri Jan 2 23:17:35 2026
- 0B resilvered, 0.00% done, no estimated completion time
-
- NAME STATE READ WRITE CKSUM
- proxmox-tank-1 DEGRADED 0 0 0
- mirror-0 DEGRADED 0 0 0
- ata-ST4000NT001-3M2101_WX11TN0Z DEGRADED 0 0 0 too many errors
- ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0
- Two things to notice: the drive's error counters are now at zero (we're starting fresh), and ZFS immediately started resilvering. It shows "too many errors" as the reason for the degraded state, which is historical, it remembers why the drive was marked bad before.
- I kept watching both the status and the kernel log. No errors, no link resets.
- Step 5: The Resilver
- Resilvering is ZFS's term for rebuilding redundancy. Copying data from the healthy drive to the one that fell behind. In my case, the drive had been desynchronized for who knows how long (the pool had drifted 524GB out of sync before I noticed), so there was a lot to copy.
- I shut down my VMs to reduce I/O contention and let the resilver have the disk bandwidth. Progress:
- scan: resilver in progress since Fri Jan 2 23:17:35 2026
- 495G / 618G scanned, 320G / 618G issued at 100M/s
- 320G resilvered, 51.78% done, 00:50:12 to go
- The kernel log stayed quiet the whole time. Everything was indicating the cable reseat had worked.
- I went to bed and let it run overnight. The next morning:
- scan: resilvered 495G in 01:07:58 with 0 errors on Sat Jan 3 00:25:33 2026
- 495 gigabytes resilvered in about an hour, zero errors. But the pool still showed DEGRADED with a warning about "unrecoverable error." I was very confused about this, but I solved that with some research. Apparently, ZFS is cautious and wants human acknowledgement before declaring everything healthy again.
- zpool clear proxmox-tank-1 ata-ST4000NT001-3M2101_WX11TN0Z
- This command clears the error flags. Immediately:
- pool: proxmox-tank-1
- state: ONLINE
- scan: resilvered 495G in 01:07:58 with 0 errors on Sat Jan 3 00:25:33 2026
-
- NAME STATE READ WRITE CKSUM
- proxmox-tank-1 ONLINE 0 0 0
- mirror-0 ONLINE 0 0 0
- ata-ST4000NT001-3M2101_WX11TN0Z ONLINE 0 0 0
- ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0
- Damn, seeing this felt nice.
- Step 6: The Scrub
- A resilver copies data to bring the drives back in sync, but it doesn't verify that all the existing data is still good. For that, you run a scrub. ZFS reads every block on the pool, verifies checksums, and repairs anything that doesn't match.
- zpool scrub proxmox-tank-1
- I let this run while I brought my VMs back up (scrubs can run in the background without blocking normal operations, though performance takes a hit). A few hours later:
- scan: scrub repaired 13.0M in 02:14:22 with 0 errors on Sat Jan 3 11:03:54 2026
-
- NAME STATE READ WRITE CKSUM
- proxmox-tank-1 ONLINE 0 0 0
- mirror-0 ONLINE 0 0 0
- ata-ST4000NT001-3M2101_WX11TN0Z ONLINE 0 0 992
- ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0
- Interesting. The scrub repaired 13MB of data and found 992 checksum mismatches on AGAPITO1. From what I read, checksum errors are typically a sign of the disk being in terrible shape and needing a replacement ASAP. That sounds scary, but I took the risk and assumed those were blocks that had been written incorrectly (or not at all) during the period when the connection was flaky, and not an issue with the disk itself. ZFS detected the bad checksums and healed them using the good copies from AGAPITO2.
- I cleared the errors again and the pool was clean:
- zpool clear proxmox-tank-1 ata-ST4000NT001-3M2101_WX11TN0Z
- Step 7: Final Validation with SMART
- One more check. I wanted to see if SMART had anything new to say about the drive after all that activity:
- smartctl -x /dev/sdb | egrep -i 'overall|Reallocated|Pending|CRC|Hardware Resets'
- SMART overall-health self-assessment test result: PASSED
- 5 Reallocated_Sector_Ct PO--CK 100 100 010 - 0
-197 Current_Pending_Sector -O--C- 100 100 000 - 0
-199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
-0x06 0x008 4 41 --- Number of Hardware Resets
- Still passing. The hardware reset count went from 39 to 41 — just the reboots I did during this process.
- For completeness, I ran the long self-test. The short test only takes a minute and does basic checks, the long test actually reads every sector on the disk, which for a 4TB drive takes... a while.
- smartctl -t long /dev/sdb
- The estimated time was about 6 hours. In practice, it took closer to 12. Running VMs in parallel probably didn't help.
- But eventually:
- SMART Self-test log structure revision number 1
-Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
-# 1 Extended offline Completed without error 00% 1563 -
-# 2 Short offline Completed without error 00% 1551 -
-# 3 Short offline Completed without error 00% 1462 -
- The extended test passed. Every sector on the disk is readable. The drive is genuinely healthy — it was just the connection that was bad.
- Lessons Learned
-
- - ZFS did exactly what it's supposed to do: Despite 524+ gigabytes of desync and nearly a thousand checksum errors, I lost zero data and was back on action while keeping my VMs running. The healthy drive kept serving everything while the flaky drive was acting up, and once the connection was fixed, ZFS healed itself automatically. Also, I was operating for an unknown amount of time with only one drive. In this case it seems it was due to stupid me messing up cable management, but I'm very happy knowing if the disk had been genuinely faulty, services would have continued just fine.
- - Physical connections matter: It's easy to not pay that much attention when building a new box. Well, it bites back.
- - Monitor your pools. I only found this issue by accident, clicking around in the Proxmox UI. The pool had been degraded for who knows how long before I noticed. I'm already working in setting up a monitor to my Uptime Kuma instance so that next time the pool status stops being ONLINE I get notified immediately.
-
- I'm happy I was able to test out recoverying from a faulty disk with such a tiny issue. I learned a lot fixing it, and now I'm even more happy than before having decided to go for this ZFS pool setup.
- Quick Reference: The Commands
- For future me (and anyone else who ends up here with a degraded pool):
- # Check pool status
-zpool status -v
-
-# Watch kernel logs in real time
-dmesg -Tw
-
-# Check SMART health
-smartctl -H /dev/sdX
-smartctl -x /dev/sdX
-
-# Take a drive offline before physical work
-zpool offline
-
-# Bring a drive back online
-zpool online
-
-# Clear error flags after recovery
-zpool clear
-
-# Run a scrub to verify all data
-zpool scrub
-
-# Run SMART self-tests
-smartctl -t short /dev/sdX # Quick test (~1 min)
-smartctl -t long /dev/sdX # Full surface scan (hours)
-smartctl -l selftest /dev/sdX # Check test results
- Thanks for reading! This was Part 3: The Fix. You might also enjoy Part 1: The Setup and Part 2: Diagnosing the Problem.
- back to home
-
- 
- 
- 
- 
- 
- 
-
-