diff --git a/20251230_first_zfs_degradation.md b/zfs-mirror-incident/20251230_first_zfs_degradation.md similarity index 100% rename from 20251230_first_zfs_degradation.md rename to zfs-mirror-incident/20251230_first_zfs_degradation.md diff --git a/20260208_second_zfs_degradation.md b/zfs-mirror-incident/20260208_second_zfs_degradation.md similarity index 100% rename from 20260208_second_zfs_degradation.md rename to zfs-mirror-incident/20260208_second_zfs_degradation.md diff --git a/zfs-mirror-incident/20260220_agapito1_replacement_runbook.md b/zfs-mirror-incident/20260220_agapito1_replacement_runbook.md new file mode 100644 index 0000000..a06d2a8 --- /dev/null +++ b/zfs-mirror-incident/20260220_agapito1_replacement_runbook.md @@ -0,0 +1,364 @@ +# AGAPITO1 Replacement Runbook + +Continuation of [20260208_second_zfs_degradation.md](20260208_second_zfs_degradation.md). + +AGAPITO1 (`ata-ST4000NT001-3M2101_WX11TN0Z`) had a failing SATA PHY and was RMA'd. The ZFS mirror `proxmox-tank-1` has been running degraded on AGAPITO2 alone since Feb 8. The replacement drive (same model, serial `WX120LHQ`) needs to be physically installed and added to the mirror. + +**Current state:** +- Pool: `proxmox-tank-1` (mirror-0), DEGRADED +- AGAPITO2 (`WX11TN2P`): ONLINE, on ata4 +- Old AGAPITO1 (`WX11TN0Z`): shows REMOVED in pool config +- Physical: drive bay empty, SATA data + power cables still connected to mobo/PSU (should be ata3 port after the cable swap from incident 2) +- New drive: ST4000NT001-3M2101, serial `WX120LHQ` + +--- + +## Phase 1: Pre-shutdown state capture + +While server is still running, log current state for reference. + +- [x] **1.1** Record pool status + ``` + zpool status -v proxmox-tank-1 + ``` + Expected: DEGRADED, WX11TN0Z shows REMOVED, WX11TN2P ONLINE. + ``` + pool: proxmox-tank-1 + state: DEGRADED + status: One or more devices have been removed. + Sufficient replicas exist for the pool to continue functioning in a + degraded state. + action: Online the device using zpool online' or replace the device with + 'zpool replace'. + scan: scrub repaired 0B in 06:55:06 with 0 errors on Tue Feb 17 20:40:50 2026 + config: + + NAME STATE READ WRITE CKSUM + proxmox-tank-1 DEGRADED 0 0 0 + mirror-0 DEGRADED 0 0 0 + ata-ST4000NT001-3M2101_WX11TN0Z REMOVED 0 0 0 + ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0 + + errors: No known data errors + ``` + +- [x] **1.2** Record current SATA layout + ``` + dmesg -T | grep -E 'ata[0-9]+\.[0-9]+: ATA-|ata[0-9]+: SATA link up' | tail -20 + ``` + Expected: AGAPITO2 visible on ata4. ata3 should show nothing (empty slot). + ``` + [Tue Feb 17 15:37:28 2026] ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300) + [Tue Feb 17 15:37:28 2026] ata4.00: ATA-11: ST4000NT001-3M2101, EN01, max UDMA/133 + ``` + +- [x] **1.3** Confirm AGAPITO2 is healthy before we start + ``` + smartctl -H /dev/disk/by-id/ata-ST4000NT001-3M2101_WX11TN2P + ``` + Expected: PASSED. If not, stop and investigate before proceeding. + ``` + SMART overall-health self-assessment test result: PASSED + ``` + +--- + +## Phase 2: Graceful shutdown + +- [x] **2.1** Shut down all VMs gracefully from Proxmox UI or CLI + ``` + qm list + # For each running VM: + qm shutdown + ``` + +- [x] **2.2** Verify all VMs are stopped + ``` + qm list + ``` + Expected: all show "stopped". + +- [x] **2.3** Power down the server + ``` + shutdown -h now + ``` + +--- + +## Phase 3: Physical installation + +- [x] **3.1** Open the case +- [x] **3.2** Locate the dangling SATA data + power cables (from the old AGAPITO1 slot) +- [x] **3.3** Visually inspect cables for damage — especially the SATA data connector pins +- [x] **3.4** Label the new drive as TOMMY with a marker/sticker. Write serial `WX120LHQ` on the label too. +- [x] **3.5** Seat the new drive in the bay +- [x] **3.6** Connect SATA data cable to the drive — push firmly until it clicks +- [x] **3.7** Connect SATA power cable to the drive — push firmly +- [x] **3.8** Double-check both connectors are fully seated (wiggle test — they shouldn't move) +- [x] **3.9** Close the case + +--- + +## Phase 4: Boot and verify detection + +- [x] **4.1** Power on the server, let it boot into Proxmox + +- [x] **4.2** Verify the new drive is detected by the kernel + ``` + dmesg -T | grep -E 'ata[0-9]+\.[0-9]+: ATA-|ata[0-9]+: SATA link up' + ``` + Expected: new drive detected on ata3 (or whichever port the cable is on), at 6.0 Gbps. + ``` + [Fri Feb 20 22:57:06 2026] ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300) + [Fri Feb 20 22:57:06 2026] ata3.00: ATA-11: ST4000NT001-3M2101, EN01, max UDMA/133 + [Fri Feb 20 22:57:07 2026] ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300) + [Fri Feb 20 22:57:07 2026] ata4.00: ATA-11: ST4000NT001-3M2101, EN01, max UDMA/133 + ``` + TOMMY on ata3, AGAPITO2 on ata4. Both at 6.0 Gbps, firmware EN01. + +- [x] **4.3** Verify the drive appears in `/dev/disk/by-id/` + ``` + ls -l /dev/disk/by-id/ | grep WX120LHQ + ``` + Expected: `ata-ST4000NT001-3M2101_WX120LHQ` pointing to some `/dev/sdX`. + ``` + ata-ST4000NT001-3M2101_WX120LHQ -> ../../sda + ``` + +- [ ] **4.4** Set variables for convenience + ``` + NEW_DISKID="ata-ST4000NT001-3M2101_WX120LHQ" + NEW_DISKPATH="/dev/disk/by-id/$NEW_DISKID" + OLD_DISKID="ata-ST4000NT001-3M2101_WX11TN0Z" + echo "New: $NEW_DISKID -> $(readlink -f $NEW_DISKPATH)" + ``` + +- [x] **4.5** Confirm drive identity and firmware version with smartctl + ``` + smartctl -i "$NEW_DISKPATH" + ``` + Expected: Model ST4000NT001-3M2101, Serial WX120LHQ, Firmware EN01, 4TB capacity. + ``` + Device Model: ST4000NT001-3M2101 + Serial Number: WX120LHQ + Firmware Version: EN01 + User Capacity: 4,000,787,030,016 bytes [4.00 TB] + SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s) + ``` + +- [x] **4.6** Check kernel logs are clean — no SATA errors, link drops, or speed downgrades + ``` + dmesg -T | grep -E 'ata[0-9]' | grep -iE 'error|fatal|reset|link down|slow|limiting' + ``` + Expected: nothing. If there are errors here on a brand new drive + known-good cable, **stop and investigate**. + ``` + [Fri Feb 20 22:57:06 2026] ata1: SATA link down (SStatus 0 SControl 300) + [Fri Feb 20 22:57:06 2026] ata2: SATA link down (SStatus 0 SControl 300) + ``` + Clean — ata1/ata2 are unused ports. No errors on ata3 or ata4. + +--- + +## Phase 5: Health-check the new drive before trusting data to it + +Don't resilver onto a DOA drive. + +- [x] **5.1** SMART overall health + ``` + smartctl -H "$NEW_DISKPATH" + ``` + Expected: PASSED. + ``` + SMART overall-health self-assessment test result: PASSED + ``` + +- [x] **5.2** Check SMART attributes baseline + ``` + smartctl -A "$NEW_DISKPATH" | grep -E 'Reallocated|Pending|Offline_Uncorrect|CRC|Error_Rate' + ``` + Expected: all counters at 0 (it's a new/refurb drive). + ``` + 1 Raw_Read_Error_Rate ... - 6072 + 5 Reallocated_Sector_Ct ... - 0 + 7 Seek_Error_Rate ... - 476 + 197 Current_Pending_Sector ... - 0 + 198 Offline_Uncorrectable ... - 0 + 199 UDMA_CRC_Error_Count ... - 0 + ``` + All critical counters at 0. Read/Seek error rate raw values are normal Seagate encoding. + +- [x] **5.3** Run short self-test + ``` + smartctl -t short "$NEW_DISKPATH" + ``` + Wait ~2 minutes, then check: + ``` + smartctl -l selftest "$NEW_DISKPATH" + ``` + Expected: "Completed without error". + ``` + # 1 Short offline Completed without error 00% 0 - + ``` + Passed. 0 power-on hours — fresh drive. + +- [x] **5.4** (Decision point) Short test passed. Proceeding. + +--- + +## Phase 6: Add new drive to ZFS mirror + +- [x] **6.1** Open a dedicated terminal for kernel log monitoring + ``` + dmesg -Tw + ``` + Leave this running throughout the resilver. Watch for ANY `ata` errors. + +- [x] **6.2** Replace the old drive with the new one in the pool + ``` + zpool replace proxmox-tank-1 "$OLD_DISKID" "$NEW_DISKID" + ``` + This tells ZFS: "the REMOVED drive WX11TN0Z is being replaced by WX120LHQ". Resilvering starts automatically. + +- [x] **6.3** Verify resilvering has started + ``` + zpool status -v proxmox-tank-1 + ``` + Expected: state DEGRADED, new drive shows as part of a `replacing` vdev, resilver in progress. + ``` + resilver in progress since Fri Feb 20 23:10:58 2026 + 5.71G / 1.33T scanned at 344M/s, 0B / 1.33T issued + 0B resilvered, 0.00% done + + replacing-0 DEGRADED 0 0 0 + ata-ST4000NT001-3M2101_WX11TN0Z REMOVED 0 0 0 + ata-ST4000NT001-3M2101_WX120LHQ ONLINE 0 0 7.73K + ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0 + ``` + Resilver running. Cksum count on new drive is expected during resilver (unwritten blocks). + +- [x] **6.4** Monitor resilver progress periodically + ``` + watch -n 30 "zpool status -v proxmox-tank-1" + ``` + Expected: steady progress, no read/write/cksum errors on either drive. Based on previous experience (~500GB at ~100MB/s with VMs down), expect roughly 1-2 hours. + VMs were auto-started on boot. Resilver completed: 1.34T in 03:32:55 with 0 errors. + +- [x] **6.5** VMs were already running (auto-start on boot). + +--- + +## Phase 7: Post-resilver verification + +Wait for resilver to complete (status will say "resilvered XXG in HH:MM:SS with 0 errors"). + +- [x] **7.1** Check final pool status + ``` + zpool status -v proxmox-tank-1 + ``` + Expected: ONLINE (or DEGRADED with "too many errors" message requiring a clear — same as last time). + ``` + state: ONLINE + scan: resilvered 1.34T in 03:32:55 with 0 errors on Sat Feb 21 02:43:53 2026 + + ata-ST4000NT001-3M2101_WX120LHQ ONLINE 0 0 7.73K + ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0 + ``` + ONLINE. 7.73K cksum on TOMMY is expected resilver artifact — clearing next. + +- [x] **7.2** Clear residual cksum counters + ``` + zpool clear proxmox-tank-1 + ``` + Counters cleared (status message and cksum count gone on re-check). + ``` + state: ONLINE + scan: resilvered 1.34T in 03:32:55 with 0 errors on Sat Feb 21 02:43:53 2026 + + ata-ST4000NT001-3M2101_WX120LHQ ONLINE 0 0 0 + ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0 + + errors: No known data errors + ``` + +- [x] **7.3** Run a full scrub to verify data integrity + ``` + zpool scrub proxmox-tank-1 + ``` + Expected: **0 errors on both drives**. + ``` + scrub repaired 0B in 03:27:50 with 0 errors on Sat Feb 21 11:38:02 2026 + + ata-ST4000NT001-3M2101_WX120LHQ ONLINE 0 0 0 + ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0 + + errors: No known data errors + ``` + +- [x] **7.4** Clean status confirmed — 0B repaired, 0 errors, both drives 0/0/0. + +- [x] **7.5** Baseline SMART snapshot of the new drive after heavy I/O + ``` + smartctl -x "$NEW_DISKPATH" | grep -E 'Reallocated|Pending|Offline_Uncorrect|CRC|Hardware Resets|COMRESET|Interface' + ``` + Expected: 0 reallocated, 0 CRC errors, low hardware reset count. + ``` + Reallocated_Sector_Ct ... 0 + Current_Pending_Sector ... 0 + Offline_Uncorrectable ... 0 + UDMA_CRC_Error_Count ... 0 + Number of Hardware Resets ... 2 + Number of Interface CRC Errors ... 0 + COMRESET ... 2 + ``` + All clean. 2 hardware resets / COMRESETs from boot — normal. + +- ~~**7.6**~~ Skipped — extended SMART self-test is redundant after a clean resilver + scrub. ZFS checksums already verified every data block; the only thing the long test would cover is empty space that ZFS hasn't written to, which ZFS will verify on future use anyway. + +--- + +## Phase 8: Final state — done + +- [x] **8.1** Final pool status — already captured in 7.4. Mirror is healthy: + ``` + pool: proxmox-tank-1 + state: ONLINE + scan: scrub repaired 0B in 03:27:50 with 0 errors on Sat Feb 21 11:38:02 2026 + config: + NAME STATE READ WRITE CKSUM + proxmox-tank-1 ONLINE 0 0 0 + mirror-0 ONLINE 0 0 0 + ata-ST4000NT001-3M2101_WX120LHQ ONLINE 0 0 0 + ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0 + + errors: No known data errors + ``` + +- [x] **8.2** All VMs running normally — verified from Proxmox UI + +- [x] **8.3** Celebrate. Mirror is whole again. + +--- + +## Abort conditions + +Stop and investigate if any of these happen: + +- New drive not detected after boot (bad seating or DOA) +- SATA errors in `dmesg` during or after boot (bad cable? bad drive?) +- SMART short test fails on new drive (DOA — contact seller) +- Resilver stalls or produces errors on the new drive +- Scrub finds checksum errors on the new drive + +--- + +## Execution summary + +Executed 2026-02-20 evening through 2026-02-21 morning. No abort conditions hit — completely clean run. + +- TOMMY (`WX120LHQ`) installed on ata3 at 6.0 Gbps, detected first boot +- SMART short test passed, all critical attributes at zero +- Resilver: 1.34T in 03:32:55, 0 errors (VMs were running — auto-start on boot) +- Scrub: repaired 0B in 03:27:50, 0 errors, both drives 0/0/0 +- Post-I/O SMART baseline clean: 0 reallocated, 0 CRC errors +- Extended SMART test skipped — redundant after clean resilver + scrub (ZFS checksums already verified all data blocks) +- Pool `proxmox-tank-1` fully healthy. Mirror degradation that started 2026-02-08 is resolved.