new zfs article
This commit is contained in:
parent
650f833128
commit
e9c607e63e
GPG key ID:
883EDBAA726BD96C
3 changed files with 612 additions and 0 deletions
364
20260220_agapito1_replacement_runbook.md
Normal file
364
20260220_agapito1_replacement_runbook.md
Normal file
|
|
@ -0,0 +1,364 @@
|
|||
# AGAPITO1 Replacement Runbook
|
||||
|
||||
Continuation of [20260208_second_zfs_degradation.md](20260208_second_zfs_degradation.md).
|
||||
|
||||
AGAPITO1 (`ata-ST4000NT001-3M2101_WX11TN0Z`) had a failing SATA PHY and was RMA'd. The ZFS mirror `proxmox-tank-1` has been running degraded on AGAPITO2 alone since Feb 8. The replacement drive (same model, serial `WX120LHQ`) needs to be physically installed and added to the mirror.
|
||||
|
||||
**Current state:**
|
||||
- Pool: `proxmox-tank-1` (mirror-0), DEGRADED
|
||||
- AGAPITO2 (`WX11TN2P`): ONLINE, on ata4
|
||||
- Old AGAPITO1 (`WX11TN0Z`): shows REMOVED in pool config
|
||||
- Physical: drive bay empty, SATA data + power cables still connected to mobo/PSU (should be ata3 port after the cable swap from incident 2)
|
||||
- New drive: ST4000NT001-3M2101, serial `WX120LHQ`
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Pre-shutdown state capture
|
||||
|
||||
While server is still running, log current state for reference.
|
||||
|
||||
- [x] **1.1** Record pool status
|
||||
```
|
||||
zpool status -v proxmox-tank-1
|
||||
```
|
||||
Expected: DEGRADED, WX11TN0Z shows REMOVED, WX11TN2P ONLINE.
|
||||
```
|
||||
pool: proxmox-tank-1
|
||||
state: DEGRADED
|
||||
status: One or more devices have been removed.
|
||||
Sufficient replicas exist for the pool to continue functioning in a
|
||||
degraded state.
|
||||
action: Online the device using zpool online' or replace the device with
|
||||
'zpool replace'.
|
||||
scan: scrub repaired 0B in 06:55:06 with 0 errors on Tue Feb 17 20:40:50 2026
|
||||
config:
|
||||
|
||||
NAME STATE READ WRITE CKSUM
|
||||
proxmox-tank-1 DEGRADED 0 0 0
|
||||
mirror-0 DEGRADED 0 0 0
|
||||
ata-ST4000NT001-3M2101_WX11TN0Z REMOVED 0 0 0
|
||||
ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0
|
||||
|
||||
errors: No known data errors
|
||||
```
|
||||
|
||||
- [x] **1.2** Record current SATA layout
|
||||
```
|
||||
dmesg -T | grep -E 'ata[0-9]+\.[0-9]+: ATA-|ata[0-9]+: SATA link up' | tail -20
|
||||
```
|
||||
Expected: AGAPITO2 visible on ata4. ata3 should show nothing (empty slot).
|
||||
```
|
||||
[Tue Feb 17 15:37:28 2026] ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
|
||||
[Tue Feb 17 15:37:28 2026] ata4.00: ATA-11: ST4000NT001-3M2101, EN01, max UDMA/133
|
||||
```
|
||||
|
||||
- [x] **1.3** Confirm AGAPITO2 is healthy before we start
|
||||
```
|
||||
smartctl -H /dev/disk/by-id/ata-ST4000NT001-3M2101_WX11TN2P
|
||||
```
|
||||
Expected: PASSED. If not, stop and investigate before proceeding.
|
||||
```
|
||||
SMART overall-health self-assessment test result: PASSED
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Graceful shutdown
|
||||
|
||||
- [x] **2.1** Shut down all VMs gracefully from Proxmox UI or CLI
|
||||
```
|
||||
qm list
|
||||
# For each running VM:
|
||||
qm shutdown <VMID>
|
||||
```
|
||||
|
||||
- [x] **2.2** Verify all VMs are stopped
|
||||
```
|
||||
qm list
|
||||
```
|
||||
Expected: all show "stopped".
|
||||
|
||||
- [x] **2.3** Power down the server
|
||||
```
|
||||
shutdown -h now
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Physical installation
|
||||
|
||||
- [x] **3.1** Open the case
|
||||
- [x] **3.2** Locate the dangling SATA data + power cables (from the old AGAPITO1 slot)
|
||||
- [x] **3.3** Visually inspect cables for damage — especially the SATA data connector pins
|
||||
- [x] **3.4** Label the new drive as TOMMY with a marker/sticker. Write serial `WX120LHQ` on the label too.
|
||||
- [x] **3.5** Seat the new drive in the bay
|
||||
- [x] **3.6** Connect SATA data cable to the drive — push firmly until it clicks
|
||||
- [x] **3.7** Connect SATA power cable to the drive — push firmly
|
||||
- [x] **3.8** Double-check both connectors are fully seated (wiggle test — they shouldn't move)
|
||||
- [x] **3.9** Close the case
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Boot and verify detection
|
||||
|
||||
- [x] **4.1** Power on the server, let it boot into Proxmox
|
||||
|
||||
- [x] **4.2** Verify the new drive is detected by the kernel
|
||||
```
|
||||
dmesg -T | grep -E 'ata[0-9]+\.[0-9]+: ATA-|ata[0-9]+: SATA link up'
|
||||
```
|
||||
Expected: new drive detected on ata3 (or whichever port the cable is on), at 6.0 Gbps.
|
||||
```
|
||||
[Fri Feb 20 22:57:06 2026] ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
|
||||
[Fri Feb 20 22:57:06 2026] ata3.00: ATA-11: ST4000NT001-3M2101, EN01, max UDMA/133
|
||||
[Fri Feb 20 22:57:07 2026] ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
|
||||
[Fri Feb 20 22:57:07 2026] ata4.00: ATA-11: ST4000NT001-3M2101, EN01, max UDMA/133
|
||||
```
|
||||
TOMMY on ata3, AGAPITO2 on ata4. Both at 6.0 Gbps, firmware EN01.
|
||||
|
||||
- [x] **4.3** Verify the drive appears in `/dev/disk/by-id/`
|
||||
```
|
||||
ls -l /dev/disk/by-id/ | grep WX120LHQ
|
||||
```
|
||||
Expected: `ata-ST4000NT001-3M2101_WX120LHQ` pointing to some `/dev/sdX`.
|
||||
```
|
||||
ata-ST4000NT001-3M2101_WX120LHQ -> ../../sda
|
||||
```
|
||||
|
||||
- [ ] **4.4** Set variables for convenience
|
||||
```
|
||||
NEW_DISKID="ata-ST4000NT001-3M2101_WX120LHQ"
|
||||
NEW_DISKPATH="/dev/disk/by-id/$NEW_DISKID"
|
||||
OLD_DISKID="ata-ST4000NT001-3M2101_WX11TN0Z"
|
||||
echo "New: $NEW_DISKID -> $(readlink -f $NEW_DISKPATH)"
|
||||
```
|
||||
|
||||
- [x] **4.5** Confirm drive identity and firmware version with smartctl
|
||||
```
|
||||
smartctl -i "$NEW_DISKPATH"
|
||||
```
|
||||
Expected: Model ST4000NT001-3M2101, Serial WX120LHQ, Firmware EN01, 4TB capacity.
|
||||
```
|
||||
Device Model: ST4000NT001-3M2101
|
||||
Serial Number: WX120LHQ
|
||||
Firmware Version: EN01
|
||||
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
|
||||
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
|
||||
```
|
||||
|
||||
- [x] **4.6** Check kernel logs are clean — no SATA errors, link drops, or speed downgrades
|
||||
```
|
||||
dmesg -T | grep -E 'ata[0-9]' | grep -iE 'error|fatal|reset|link down|slow|limiting'
|
||||
```
|
||||
Expected: nothing. If there are errors here on a brand new drive + known-good cable, **stop and investigate**.
|
||||
```
|
||||
[Fri Feb 20 22:57:06 2026] ata1: SATA link down (SStatus 0 SControl 300)
|
||||
[Fri Feb 20 22:57:06 2026] ata2: SATA link down (SStatus 0 SControl 300)
|
||||
```
|
||||
Clean — ata1/ata2 are unused ports. No errors on ata3 or ata4.
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: Health-check the new drive before trusting data to it
|
||||
|
||||
Don't resilver onto a DOA drive.
|
||||
|
||||
- [x] **5.1** SMART overall health
|
||||
```
|
||||
smartctl -H "$NEW_DISKPATH"
|
||||
```
|
||||
Expected: PASSED.
|
||||
```
|
||||
SMART overall-health self-assessment test result: PASSED
|
||||
```
|
||||
|
||||
- [x] **5.2** Check SMART attributes baseline
|
||||
```
|
||||
smartctl -A "$NEW_DISKPATH" | grep -E 'Reallocated|Pending|Offline_Uncorrect|CRC|Error_Rate'
|
||||
```
|
||||
Expected: all counters at 0 (it's a new/refurb drive).
|
||||
```
|
||||
1 Raw_Read_Error_Rate ... - 6072
|
||||
5 Reallocated_Sector_Ct ... - 0
|
||||
7 Seek_Error_Rate ... - 476
|
||||
197 Current_Pending_Sector ... - 0
|
||||
198 Offline_Uncorrectable ... - 0
|
||||
199 UDMA_CRC_Error_Count ... - 0
|
||||
```
|
||||
All critical counters at 0. Read/Seek error rate raw values are normal Seagate encoding.
|
||||
|
||||
- [x] **5.3** Run short self-test
|
||||
```
|
||||
smartctl -t short "$NEW_DISKPATH"
|
||||
```
|
||||
Wait ~2 minutes, then check:
|
||||
```
|
||||
smartctl -l selftest "$NEW_DISKPATH"
|
||||
```
|
||||
Expected: "Completed without error".
|
||||
```
|
||||
# 1 Short offline Completed without error 00% 0 -
|
||||
```
|
||||
Passed. 0 power-on hours — fresh drive.
|
||||
|
||||
- [x] **5.4** (Decision point) Short test passed. Proceeding.
|
||||
|
||||
---
|
||||
|
||||
## Phase 6: Add new drive to ZFS mirror
|
||||
|
||||
- [x] **6.1** Open a dedicated terminal for kernel log monitoring
|
||||
```
|
||||
dmesg -Tw
|
||||
```
|
||||
Leave this running throughout the resilver. Watch for ANY `ata` errors.
|
||||
|
||||
- [x] **6.2** Replace the old drive with the new one in the pool
|
||||
```
|
||||
zpool replace proxmox-tank-1 "$OLD_DISKID" "$NEW_DISKID"
|
||||
```
|
||||
This tells ZFS: "the REMOVED drive WX11TN0Z is being replaced by WX120LHQ". Resilvering starts automatically.
|
||||
|
||||
- [x] **6.3** Verify resilvering has started
|
||||
```
|
||||
zpool status -v proxmox-tank-1
|
||||
```
|
||||
Expected: state DEGRADED, new drive shows as part of a `replacing` vdev, resilver in progress.
|
||||
```
|
||||
resilver in progress since Fri Feb 20 23:10:58 2026
|
||||
5.71G / 1.33T scanned at 344M/s, 0B / 1.33T issued
|
||||
0B resilvered, 0.00% done
|
||||
|
||||
replacing-0 DEGRADED 0 0 0
|
||||
ata-ST4000NT001-3M2101_WX11TN0Z REMOVED 0 0 0
|
||||
ata-ST4000NT001-3M2101_WX120LHQ ONLINE 0 0 7.73K
|
||||
ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0
|
||||
```
|
||||
Resilver running. Cksum count on new drive is expected during resilver (unwritten blocks).
|
||||
|
||||
- [x] **6.4** Monitor resilver progress periodically
|
||||
```
|
||||
watch -n 30 "zpool status -v proxmox-tank-1"
|
||||
```
|
||||
Expected: steady progress, no read/write/cksum errors on either drive. Based on previous experience (~500GB at ~100MB/s with VMs down), expect roughly 1-2 hours.
|
||||
VMs were auto-started on boot. Resilver completed: 1.34T in 03:32:55 with 0 errors.
|
||||
|
||||
- [x] **6.5** VMs were already running (auto-start on boot).
|
||||
|
||||
---
|
||||
|
||||
## Phase 7: Post-resilver verification
|
||||
|
||||
Wait for resilver to complete (status will say "resilvered XXG in HH:MM:SS with 0 errors").
|
||||
|
||||
- [x] **7.1** Check final pool status
|
||||
```
|
||||
zpool status -v proxmox-tank-1
|
||||
```
|
||||
Expected: ONLINE (or DEGRADED with "too many errors" message requiring a clear — same as last time).
|
||||
```
|
||||
state: ONLINE
|
||||
scan: resilvered 1.34T in 03:32:55 with 0 errors on Sat Feb 21 02:43:53 2026
|
||||
|
||||
ata-ST4000NT001-3M2101_WX120LHQ ONLINE 0 0 7.73K
|
||||
ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0
|
||||
```
|
||||
ONLINE. 7.73K cksum on TOMMY is expected resilver artifact — clearing next.
|
||||
|
||||
- [x] **7.2** Clear residual cksum counters
|
||||
```
|
||||
zpool clear proxmox-tank-1
|
||||
```
|
||||
Counters cleared (status message and cksum count gone on re-check).
|
||||
```
|
||||
state: ONLINE
|
||||
scan: resilvered 1.34T in 03:32:55 with 0 errors on Sat Feb 21 02:43:53 2026
|
||||
|
||||
ata-ST4000NT001-3M2101_WX120LHQ ONLINE 0 0 0
|
||||
ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0
|
||||
|
||||
errors: No known data errors
|
||||
```
|
||||
|
||||
- [x] **7.3** Run a full scrub to verify data integrity
|
||||
```
|
||||
zpool scrub proxmox-tank-1
|
||||
```
|
||||
Expected: **0 errors on both drives**.
|
||||
```
|
||||
scrub repaired 0B in 03:27:50 with 0 errors on Sat Feb 21 11:38:02 2026
|
||||
|
||||
ata-ST4000NT001-3M2101_WX120LHQ ONLINE 0 0 0
|
||||
ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0
|
||||
|
||||
errors: No known data errors
|
||||
```
|
||||
|
||||
- [x] **7.4** Clean status confirmed — 0B repaired, 0 errors, both drives 0/0/0.
|
||||
|
||||
- [x] **7.5** Baseline SMART snapshot of the new drive after heavy I/O
|
||||
```
|
||||
smartctl -x "$NEW_DISKPATH" | grep -E 'Reallocated|Pending|Offline_Uncorrect|CRC|Hardware Resets|COMRESET|Interface'
|
||||
```
|
||||
Expected: 0 reallocated, 0 CRC errors, low hardware reset count.
|
||||
```
|
||||
Reallocated_Sector_Ct ... 0
|
||||
Current_Pending_Sector ... 0
|
||||
Offline_Uncorrectable ... 0
|
||||
UDMA_CRC_Error_Count ... 0
|
||||
Number of Hardware Resets ... 2
|
||||
Number of Interface CRC Errors ... 0
|
||||
COMRESET ... 2
|
||||
```
|
||||
All clean. 2 hardware resets / COMRESETs from boot — normal.
|
||||
|
||||
- ~~**7.6**~~ Skipped — extended SMART self-test is redundant after a clean resilver + scrub. ZFS checksums already verified every data block; the only thing the long test would cover is empty space that ZFS hasn't written to, which ZFS will verify on future use anyway.
|
||||
|
||||
---
|
||||
|
||||
## Phase 8: Final state — done
|
||||
|
||||
- [x] **8.1** Final pool status — already captured in 7.4. Mirror is healthy:
|
||||
```
|
||||
pool: proxmox-tank-1
|
||||
state: ONLINE
|
||||
scan: scrub repaired 0B in 03:27:50 with 0 errors on Sat Feb 21 11:38:02 2026
|
||||
config:
|
||||
NAME STATE READ WRITE CKSUM
|
||||
proxmox-tank-1 ONLINE 0 0 0
|
||||
mirror-0 ONLINE 0 0 0
|
||||
ata-ST4000NT001-3M2101_WX120LHQ ONLINE 0 0 0
|
||||
ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0
|
||||
|
||||
errors: No known data errors
|
||||
```
|
||||
|
||||
- [x] **8.2** All VMs running normally — verified from Proxmox UI
|
||||
|
||||
- [x] **8.3** Celebrate. Mirror is whole again.
|
||||
|
||||
---
|
||||
|
||||
## Abort conditions
|
||||
|
||||
Stop and investigate if any of these happen:
|
||||
|
||||
- New drive not detected after boot (bad seating or DOA)
|
||||
- SATA errors in `dmesg` during or after boot (bad cable? bad drive?)
|
||||
- SMART short test fails on new drive (DOA — contact seller)
|
||||
- Resilver stalls or produces errors on the new drive
|
||||
- Scrub finds checksum errors on the new drive
|
||||
|
||||
---
|
||||
|
||||
## Execution summary
|
||||
|
||||
Executed 2026-02-20 evening through 2026-02-21 morning. No abort conditions hit — completely clean run.
|
||||
|
||||
- TOMMY (`WX120LHQ`) installed on ata3 at 6.0 Gbps, detected first boot
|
||||
- SMART short test passed, all critical attributes at zero
|
||||
- Resilver: 1.34T in 03:32:55, 0 errors (VMs were running — auto-start on boot)
|
||||
- Scrub: repaired 0B in 03:27:50, 0 errors, both drives 0/0/0
|
||||
- Post-I/O SMART baseline clean: 0 reallocated, 0 CRC errors
|
||||
- Extended SMART test skipped — redundant after clean resilver + scrub (ZFS checksums already verified all data blocks)
|
||||
- Pool `proxmox-tank-1` fully healthy. Mirror degradation that started 2026-02-08 is resolved.
|
||||
|
|
@ -147,6 +147,10 @@
|
|||
<h2 id="writings-header">Writings</h2>
|
||||
<p>Sometimes I like to jot down ideas and drop them here.</p>
|
||||
<ul>
|
||||
<li>
|
||||
<a href="writings/replacing-a-failed-disk-in-a-zfs-mirror.html" target="_blank"
|
||||
rel="noopener noreferrer">Replacing a Failed Disk in a ZFS Mirror</a>
|
||||
</li>
|
||||
<li>
|
||||
<a href="writings/busy-mans-guide-to-optimizing-dbt-models-performance.html" target="_blank"
|
||||
rel="noopener noreferrer">Busy man's guide to optimizing dbt models performance</a>
|
||||
|
|
|
|||
244
public/writings/replacing-a-failed-disk-in-a-zfs-mirror.html
Normal file
244
public/writings/replacing-a-failed-disk-in-a-zfs-mirror.html
Normal file
|
|
@ -0,0 +1,244 @@
|
|||
<!DOCTYPE HTML>
|
||||
<html>
|
||||
|
||||
<head>
|
||||
<title>Pablo here</title>
|
||||
<meta charset="utf-8">
|
||||
<meta viewport="width=device-width, initial-scale=1">
|
||||
<link rel="stylesheet" href="../styles.css">
|
||||
</head>
|
||||
|
||||
|
||||
<body>
|
||||
<main>
|
||||
<h1>
|
||||
Hi, Pablo here
|
||||
</h1>
|
||||
<p><a href="../index.html">back to home</a></p>
|
||||
<section>
|
||||
<h2>Replacing a Failed Disk in a ZFS Mirror</h2>
|
||||
<p>If you've been following along, you know the story: I set up a <a href="why-i-put-my-vms-on-a-zfs-mirror.html">ZFS mirror for my Proxmox VMs</a>, then one of the drives <a href="a-degraded-pool-with-a-healthy-disk.html">started acting flaky</a>, and I <a href="fixing-a-degraded-zfs-mirror.html">diagnosed and fixed what turned out to be a bad SATA connection</a>.</p>
|
||||
<p>Well, the connection wasn't the whole story. A few weeks after that fix, the same drive — AGAPITO1 — started dropping off again. Same symptoms: link resets, speed downgrades, kernel giving up on the connection. I went through the cable swap dance again, tried different SATA ports on the motherboard, tried different cables. Nothing helped. The SATA PHY on the drive itself was failing.</p>
|
||||
<p>I contacted Seagate, RMA'd the drive, and ran degraded on AGAPITO2 alone for about two weeks. Then the replacement arrived. This article covers the process of physically installing a new drive and getting it into the ZFS mirror — from "box on the desk" to "pool healthy, mirror whole."</p>
|
||||
|
||||
<h3>The starting point</h3>
|
||||
<p>Before doing anything, this is what the pool looked like:</p>
|
||||
<pre><code> pool: proxmox-tank-1
|
||||
state: DEGRADED
|
||||
status: One or more devices have been removed.
|
||||
Sufficient replicas exist for the pool to continue functioning in a
|
||||
degraded state.
|
||||
action: Online the device using zpool online' or replace the device with
|
||||
'zpool replace'.
|
||||
scan: scrub repaired 0B in 06:55:06 with 0 errors on Tue Feb 17 20:40:50 2026
|
||||
config:
|
||||
|
||||
NAME STATE READ WRITE CKSUM
|
||||
proxmox-tank-1 DEGRADED 0 0 0
|
||||
mirror-0 DEGRADED 0 0 0
|
||||
ata-ST4000NT001-3M2101_WX11TN0Z REMOVED 0 0 0
|
||||
ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0
|
||||
|
||||
errors: No known data errors</code></pre>
|
||||
<p>DEGRADED with one drive REMOVED. The old drive (WX11TN0Z) was physically gone — shipped back to Seagate. AGAPITO2 (WX11TN2P) was holding down the fort alone.</p>
|
||||
<p>This is the beauty and the terror of a degraded mirror: everything works fine. Your VMs keep running, your data is intact, reads and writes happen normally. But you have zero redundancy. If that surviving drive has a bad day, you lose everything. Two weeks of running like this was two weeks of hoping AGAPITO2 stayed healthy.</p>
|
||||
|
||||
<h3>Before you touch hardware</h3>
|
||||
<p>Before doing anything physical, I wanted to capture the current state. When things go wrong during maintenance, you want to be able to compare "before" and "after."</p>
|
||||
<p>Three things to record while the server is still running:</p>
|
||||
<p><strong>Pool status</strong> — the <code>zpool status</code> output above. You want to know exactly what ZFS thinks the world looks like right now.</p>
|
||||
<p><strong>SATA layout</strong> — which drive is on which port:</p>
|
||||
<pre><code>dmesg -T | grep -E 'ata[0-9]+\.[0-9]+: ATA-|ata[0-9]+: SATA link up'</code></pre>
|
||||
<p>In my case, AGAPITO2 was on ata4 and ata3 was empty (the old drive's port). This matters because after you install the new drive, you want to confirm it shows up on the expected port.</p>
|
||||
<p><strong>Surviving drive health</strong> — make sure the drive you're depending on is actually healthy before you start:</p>
|
||||
<pre><code>smartctl -H /dev/disk/by-id/ata-ST4000NT001-3M2101_WX11TN2P</code></pre>
|
||||
<pre><code>SMART overall-health self-assessment test result: PASSED</code></pre>
|
||||
<p>If this says anything other than PASSED, stop and deal with that first. You don't want to discover your only remaining copy of data is on a failing drive while you're in the middle of hardware work.</p>
|
||||
<p>Once you've got your reference snapshots, shut down gracefully. Stop your VMs first, then power off the server. You want a clean shutdown, not a yank-the-plug situation.</p>
|
||||
<pre><code>qm shutdown <VMID> # for each running VM
|
||||
shutdown -h now</code></pre>
|
||||
|
||||
<h3>Physical installation</h3>
|
||||
<p>I won't write a hardware installation tutorial — every case and drive bay is different. But a few practical tips for homelabbers doing this for the first time:</p>
|
||||
<ul>
|
||||
<li><strong>Inspect your cables before connecting them.</strong> If the SATA data cable has been sitting disconnected in the case, check the connector pins. Bent pins or dust can cause exactly the kind of intermittent issues that started this whole saga.</li>
|
||||
<li><strong>Label the new drive.</strong> I labeled mine "TOMMY" with its serial number (WX120LHQ) written on a sticker. Yes, I name my drives. It makes debugging much easier than squinting at serial numbers.</li>
|
||||
<li><strong>Push connectors until they click.</strong> Both SATA data and power. Then do the wiggle test: grab the connector gently and try to move it. If it shifts at all, it's not fully seated.</li>
|
||||
</ul>
|
||||
<p>Seat the drive, connect both cables, close the case, and power on.</p>
|
||||
|
||||
<h3>Boot and verify detection</h3>
|
||||
<p>First thing after boot: did the kernel see the new drive?</p>
|
||||
<pre><code>dmesg -T | grep -E 'ata[0-9]+\.[0-9]+: ATA-|ata[0-9]+: SATA link up'</code></pre>
|
||||
<pre><code>[Fri Feb 20 22:57:06 2026] ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
|
||||
[Fri Feb 20 22:57:06 2026] ata3.00: ATA-11: ST4000NT001-3M2101, EN01, max UDMA/133
|
||||
[Fri Feb 20 22:57:07 2026] ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
|
||||
[Fri Feb 20 22:57:07 2026] ata4.00: ATA-11: ST4000NT001-3M2101, EN01, max UDMA/133</code></pre>
|
||||
<p>Two drives detected. TOMMY on ata3, AGAPITO2 on ata4. Both at full 6.0 Gbps. Good.</p>
|
||||
<p>Next, verify it shows up with its expected serial in <code>/dev/disk/by-id/</code>:</p>
|
||||
<pre><code>ls -l /dev/disk/by-id/ | grep WX120LHQ</code></pre>
|
||||
<pre><code>ata-ST4000NT001-3M2101_WX120LHQ -> ../../sda</code></pre>
|
||||
<p>And confirm identity with SMART:</p>
|
||||
<pre><code>smartctl -i /dev/disk/by-id/ata-ST4000NT001-3M2101_WX120LHQ</code></pre>
|
||||
<pre><code>Device Model: ST4000NT001-3M2101
|
||||
Serial Number: WX120LHQ
|
||||
Firmware Version: EN01
|
||||
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
|
||||
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)</code></pre>
|
||||
<p>Right model, right serial, right firmware, full speed. That's our drive.</p>
|
||||
<p>One more critical check: look for SATA errors in the kernel log.</p>
|
||||
<pre><code>dmesg -T | grep -E 'ata[0-9]' | grep -iE 'error|fatal|reset|link down|slow|limiting'</code></pre>
|
||||
<p>I saw <code>ata1: SATA link down</code> and <code>ata2: SATA link down</code> — those are unused ports, perfectly normal. Nothing on ata3 or ata4. If you see errors on the port your new drive is on, <strong>stop</strong>. A brand new drive throwing SATA errors on a known-good cable means the drive is likely DOA.</p>
|
||||
|
||||
<h3>Health-check before trusting it</h3>
|
||||
<p>A drive can be detected and still be dead on arrival. Before resilvering 1.3 terabytes of data onto it, I wanted to know it was actually healthy.</p>
|
||||
<p><strong>SMART overall health:</strong></p>
|
||||
<pre><code>smartctl -H /dev/disk/by-id/ata-ST4000NT001-3M2101_WX120LHQ</code></pre>
|
||||
<pre><code>SMART overall-health self-assessment test result: PASSED</code></pre>
|
||||
<p><strong>Baseline SMART attributes</strong> — the important ones to check on a new drive:</p>
|
||||
<pre><code>smartctl -A /dev/disk/by-id/ata-ST4000NT001-3M2101_WX120LHQ | grep -E 'Reallocated|Pending|Offline_Uncorrect|CRC'</code></pre>
|
||||
<pre><code> 5 Reallocated_Sector_Ct ... - 0
|
||||
197 Current_Pending_Sector ... - 0
|
||||
198 Offline_Uncorrectable ... - 0
|
||||
199 UDMA_CRC_Error_Count ... - 0</code></pre>
|
||||
<p>All zeros. Reallocated sectors would mean the drive has already had to remap bad spots. Pending sectors are blocks the drive suspects are bad but hasn't confirmed yet. CRC errors indicate data corruption during transfer. On a new or refurbished drive, all of these should be zero.</p>
|
||||
<p><strong>Short self-test:</strong></p>
|
||||
<pre><code>smartctl -t short /dev/disk/by-id/ata-ST4000NT001-3M2101_WX120LHQ
|
||||
# Wait ~2 minutes...
|
||||
smartctl -l selftest /dev/disk/by-id/ata-ST4000NT001-3M2101_WX120LHQ</code></pre>
|
||||
<pre><code># 1 Short offline Completed without error 00% 0 -</code></pre>
|
||||
<p>Passed with 0 power-on hours — a fresh drive. If any of these checks fail, don't proceed. Contact the seller and get another replacement.</p>
|
||||
|
||||
<h3>The replacement — <code>zpool replace</code></h3>
|
||||
<p>This is the moment. One command:</p>
|
||||
<pre><code>zpool replace proxmox-tank-1 ata-ST4000NT001-3M2101_WX11TN0Z ata-ST4000NT001-3M2101_WX120LHQ</code></pre>
|
||||
<p>What this does: it tells ZFS "the drive identified as WX11TN0Z (currently REMOVED) is being replaced by WX120LHQ." ZFS starts resilvering immediately — copying all data from the surviving drive (AGAPITO2) onto the new one (TOMMY).</p>
|
||||
<p>Checking status right after:</p>
|
||||
<pre><code> pool: proxmox-tank-1
|
||||
state: DEGRADED
|
||||
scan: resilver in progress since Fri Feb 20 23:10:58 2026
|
||||
|
||||
config:
|
||||
|
||||
NAME STATE READ WRITE CKSUM
|
||||
proxmox-tank-1 DEGRADED 0 0 0
|
||||
mirror-0 DEGRADED 0 0 0
|
||||
replacing-0 DEGRADED 0 0 0
|
||||
ata-ST4000NT001-3M2101_WX11TN0Z REMOVED 0 0 0
|
||||
ata-ST4000NT001-3M2101_WX120LHQ ONLINE 0 0 7.73K
|
||||
ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0</code></pre>
|
||||
<p>Notice the <code>replacing-0</code> vdev — that's a temporary structure ZFS creates during the replacement. It shows both the old (REMOVED) and new (ONLINE) drive while the resilver is in progress.</p>
|
||||
<p>The 7.73K cksum count on the new drive might look alarming, but it's expected during a resilver. Those are blocks that haven't been written yet — ZFS is aware of them and they'll clear up as the resilver progresses.</p>
|
||||
<p>I monitored progress with:</p>
|
||||
<pre><code>watch -n 30 "zpool status -v proxmox-tank-1"</code></pre>
|
||||
<p>I also kept <code>dmesg -Tw</code> running in another terminal, watching for any SATA errors. The kernel log stayed quiet the entire time.</p>
|
||||
<p>In my case, the VMs had auto-started on boot, so the resilver was competing with production I/O. It completed in about 3.5 hours: 1.34 terabytes resilvered with 0 errors. Not bad for a pair of 4TB IronWolf drives running alongside active workloads.</p>
|
||||
|
||||
<h3>Post-resilver verification</h3>
|
||||
<p>The resilver finished. Time to verify everything is actually good.</p>
|
||||
<p><strong>Pool status:</strong></p>
|
||||
<pre><code> pool: proxmox-tank-1
|
||||
state: ONLINE
|
||||
scan: resilvered 1.34T in 03:32:55 with 0 errors on Sat Feb 21 02:43:53 2026
|
||||
config:
|
||||
|
||||
NAME STATE READ WRITE CKSUM
|
||||
proxmox-tank-1 ONLINE 0 0 0
|
||||
mirror-0 ONLINE 0 0 0
|
||||
ata-ST4000NT001-3M2101_WX120LHQ ONLINE 0 0 7.73K
|
||||
ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0
|
||||
|
||||
errors: No known data errors</code></pre>
|
||||
<p>ONLINE. The <code>replacing-0</code> vdev is gone, replaced by the normal mirror with the new drive. The 7.73K cksum on TOMMY is a residual counter from the resilver — let's clear it:</p>
|
||||
<pre><code>zpool clear proxmox-tank-1</code></pre>
|
||||
<p>Now for the real test. A resilver copies data to rebuild the mirror, but a <strong>scrub</strong> reads every block on the pool, verifies all checksums, and repairs any mismatches. This is the definitive integrity check:</p>
|
||||
<pre><code>zpool scrub proxmox-tank-1</code></pre>
|
||||
<p>This ran for about 3.5 hours across 1.34T of data:</p>
|
||||
<pre><code> scan: scrub repaired 0B in 03:27:50 with 0 errors on Sat Feb 21 11:38:02 2026
|
||||
|
||||
NAME STATE READ WRITE CKSUM
|
||||
proxmox-tank-1 ONLINE 0 0 0
|
||||
mirror-0 ONLINE 0 0 0
|
||||
ata-ST4000NT001-3M2101_WX120LHQ ONLINE 0 0 0
|
||||
ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0
|
||||
|
||||
errors: No known data errors</code></pre>
|
||||
<p>Zero bytes repaired. Zero errors. Both drives at 0/0/0. That's a clean bill of health.</p>
|
||||
<p>One last thing: a post-I/O SMART check on the new drive. After hours of heavy writes (resilver) and reads (scrub), any hardware weakness should have surfaced:</p>
|
||||
<pre><code>smartctl -x /dev/disk/by-id/ata-ST4000NT001-3M2101_WX120LHQ | grep -E 'Reallocated|Pending|Offline_Uncorrect|CRC|Hardware Resets|COMRESET|Interface'</code></pre>
|
||||
<pre><code>Reallocated_Sector_Ct ... 0
|
||||
Current_Pending_Sector ... 0
|
||||
Offline_Uncorrectable ... 0
|
||||
UDMA_CRC_Error_Count ... 0
|
||||
Number of Hardware Resets ... 2
|
||||
Number of Interface CRC Errors ... 0
|
||||
COMRESET ... 2</code></pre>
|
||||
<p>All clean. The 2 hardware resets and 2 COMRESETs are just from the server booting — perfectly normal. No reallocated sectors, no CRC errors. The drive is healthy.</p>
|
||||
|
||||
<h3>The commands, all in one place</h3>
|
||||
<p>For future me and anyone else replacing a disk in a ZFS mirror:</p>
|
||||
<pre><code># --- Before shutdown ---
|
||||
|
||||
# Record pool status
|
||||
zpool status -v <pool>
|
||||
|
||||
# Record SATA layout
|
||||
dmesg -T | grep -E 'ata[0-9]+\.[0-9]+: ATA-|ata[0-9]+: SATA link up'
|
||||
|
||||
# Check surviving drive health
|
||||
smartctl -H /dev/disk/by-id/<surviving-disk-id>
|
||||
|
||||
# Stop VMs and shut down
|
||||
qm shutdown <VMID>
|
||||
shutdown -h now
|
||||
|
||||
# --- After boot with new drive ---
|
||||
|
||||
# Verify detection
|
||||
dmesg -T | grep -E 'ata[0-9]+\.[0-9]+: ATA-|ata[0-9]+: SATA link up'
|
||||
ls -l /dev/disk/by-id/ | grep <new-serial>
|
||||
smartctl -i /dev/disk/by-id/<new-disk-id>
|
||||
|
||||
# Check for SATA errors
|
||||
dmesg -T | grep -E 'ata[0-9]' | grep -iE 'error|fatal|reset|link down'
|
||||
|
||||
# Health-check the new drive
|
||||
smartctl -H /dev/disk/by-id/<new-disk-id>
|
||||
smartctl -A /dev/disk/by-id/<new-disk-id> | grep -E 'Reallocated|Pending|Offline_Uncorrect|CRC'
|
||||
smartctl -t short /dev/disk/by-id/<new-disk-id>
|
||||
smartctl -l selftest /dev/disk/by-id/<new-disk-id>
|
||||
|
||||
# --- Replace and resilver ---
|
||||
|
||||
# Replace old drive with new
|
||||
zpool replace <pool> <old-disk-id> <new-disk-id>
|
||||
|
||||
# Monitor resilver progress
|
||||
watch -n 30 "zpool status -v <pool>"
|
||||
|
||||
# Watch kernel log for SATA errors during resilver
|
||||
dmesg -Tw
|
||||
|
||||
# --- Post-resilver verification ---
|
||||
|
||||
# Check final status
|
||||
zpool status -v <pool>
|
||||
|
||||
# Clear residual cksum counters
|
||||
zpool clear <pool>
|
||||
|
||||
# Run a full scrub
|
||||
zpool scrub <pool>
|
||||
|
||||
# Post-I/O SMART check
|
||||
smartctl -x /dev/disk/by-id/<new-disk-id> | grep -E 'Reallocated|Pending|Offline_Uncorrect|CRC'</code></pre>
|
||||
|
||||
<p>The mirror degradation that started on February 8th is resolved. Two weeks of running on a single drive, an RMA, and one evening of work later, the pool is whole again. All VMs running, full redundancy restored, zero data lost throughout the entire saga. ZFS did exactly what it was designed to do.</p>
|
||||
<p><em>This is the fourth and final article in this series. If you're just arriving, start with <a href="why-i-put-my-vms-on-a-zfs-mirror.html">Part 1: Why I Put My VMs on a ZFS Mirror</a>, then <a href="a-degraded-pool-with-a-healthy-disk.html">Part 2: A Degraded Pool with a Healthy Disk</a>, and <a href="fixing-a-degraded-zfs-mirror.html">Part 3: Fixing a Degraded ZFS Mirror</a>.</em></p>
|
||||
<p><a href="../index.html">back to home</a></p>
|
||||
</section>
|
||||
</main>
|
||||
|
||||
</body>
|
||||
|
||||
</html>
|
||||
Loading…
Add table
Add a link
Reference in a new issue