counterweight/pablohere
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Compare commits

base: counterweight:master
Branches Tags
counterweight:master
counterweight:add-ntfy-emergency-app
counterweight:2025-taxes-recap
counterweight:superhog-departure-notes
counterweight:postgres-as-a-dwh
counterweight:the-roi-of-toilets
counterweight:bathrooms-of-your-product
counterweight:valuatng-data-teams-output
..
compare: counterweight:add-ntfy-emergency-app
Branches Tags
counterweight:master
counterweight:add-ntfy-emergency-app
counterweight:2025-taxes-recap
counterweight:superhog-departure-notes
counterweight:postgres-as-a-dwh
counterweight:the-roi-of-toilets
counterweight:bathrooms-of-your-product
counterweight:valuatng-data-teams-output

No commits in common. "master" and "add-ntfy-emergency-app" have entirely different histories.
master ... add-ntfy-e

15 changed files with 2 additions and 2118 deletions
Download patch file Download diff file Expand all files Collapse all files

14
.gitignore vendored
View file

@ -1,14 +0,0 @@
# Deployment configuration (contains sensitive server details)
deploy.config
# OS files
.DS_Store
Thumbs.db
# Editor files
.vscode/
.idea/
*.swp
*.swo
*~

364
20260220_agapito1_replacement_runbook.md
View file

@ -1,364 +0,0 @@
# AGAPITO1 Replacement Runbook
Continuation of [20260208_second_zfs_degradation.md](20260208_second_zfs_degradation.md).
AGAPITO1 (`ata-ST4000NT001-3M2101_WX11TN0Z`) had a failing SATA PHY and was RMA'd. The ZFS mirror `proxmox-tank-1` has been running degraded on AGAPITO2 alone since Feb 8. The replacement drive (same model, serial `WX120LHQ`) needs to be physically installed and added to the mirror.
**Current state:**
- Pool: `proxmox-tank-1` (mirror-0), DEGRADED
- AGAPITO2 (`WX11TN2P`): ONLINE, on ata4
- Old AGAPITO1 (`WX11TN0Z`): shows REMOVED in pool config
- Physical: drive bay empty, SATA data + power cables still connected to mobo/PSU (should be ata3 port after the cable swap from incident 2)
- New drive: ST4000NT001-3M2101, serial `WX120LHQ`
---
## Phase 1: Pre-shutdown state capture
While server is still running, log current state for reference.
- [x] **1.1** Record pool status
```
zpool status -v proxmox-tank-1
```
Expected: DEGRADED, WX11TN0Z shows REMOVED, WX11TN2P ONLINE.
```
pool: proxmox-tank-1
state: DEGRADED
status: One or more devices have been removed.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using zpool online' or replace the device with
'zpool replace'.
scan: scrub repaired 0B in 06:55:06 with 0 errors on Tue Feb 17 20:40:50 2026
config:
NAME STATE READ WRITE CKSUM
proxmox-tank-1 DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
ata-ST4000NT001-3M2101_WX11TN0Z REMOVED 0 0 0
ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0
errors: No known data errors
```
- [x] **1.2** Record current SATA layout
```
dmesg -T | grep -E 'ata[0-9]+\.[0-9]+: ATA-|ata[0-9]+: SATA link up' | tail -20
```
Expected: AGAPITO2 visible on ata4. ata3 should show nothing (empty slot).
```
[Tue Feb 17 15:37:28 2026] ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[Tue Feb 17 15:37:28 2026] ata4.00: ATA-11: ST4000NT001-3M2101, EN01, max UDMA/133
```
- [x] **1.3** Confirm AGAPITO2 is healthy before we start
```
smartctl -H /dev/disk/by-id/ata-ST4000NT001-3M2101_WX11TN2P
```
Expected: PASSED. If not, stop and investigate before proceeding.
```
SMART overall-health self-assessment test result: PASSED
```
---
## Phase 2: Graceful shutdown
- [x] **2.1** Shut down all VMs gracefully from Proxmox UI or CLI
```
qm list
# For each running VM:
qm shutdown <VMID>
```
- [x] **2.2** Verify all VMs are stopped
```
qm list
```
Expected: all show "stopped".
- [x] **2.3** Power down the server
```
shutdown -h now
```
---
## Phase 3: Physical installation
- [x] **3.1** Open the case
- [x] **3.2** Locate the dangling SATA data + power cables (from the old AGAPITO1 slot)
- [x] **3.3** Visually inspect cables for damage — especially the SATA data connector pins
- [x] **3.4** Label the new drive as TOMMY with a marker/sticker. Write serial `WX120LHQ` on the label too.
- [x] **3.5** Seat the new drive in the bay
- [x] **3.6** Connect SATA data cable to the drive — push firmly until it clicks
- [x] **3.7** Connect SATA power cable to the drive — push firmly
- [x] **3.8** Double-check both connectors are fully seated (wiggle test — they shouldn't move)
- [x] **3.9** Close the case
---
## Phase 4: Boot and verify detection
- [x] **4.1** Power on the server, let it boot into Proxmox
- [x] **4.2** Verify the new drive is detected by the kernel
```
dmesg -T | grep -E 'ata[0-9]+\.[0-9]+: ATA-|ata[0-9]+: SATA link up'
```
Expected: new drive detected on ata3 (or whichever port the cable is on), at 6.0 Gbps.
```
[Fri Feb 20 22:57:06 2026] ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[Fri Feb 20 22:57:06 2026] ata3.00: ATA-11: ST4000NT001-3M2101, EN01, max UDMA/133
[Fri Feb 20 22:57:07 2026] ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[Fri Feb 20 22:57:07 2026] ata4.00: ATA-11: ST4000NT001-3M2101, EN01, max UDMA/133
```
TOMMY on ata3, AGAPITO2 on ata4. Both at 6.0 Gbps, firmware EN01.
- [x] **4.3** Verify the drive appears in `/dev/disk/by-id/`
```
ls -l /dev/disk/by-id/ | grep WX120LHQ
```
Expected: `ata-ST4000NT001-3M2101_WX120LHQ` pointing to some `/dev/sdX`.
```
ata-ST4000NT001-3M2101_WX120LHQ -> ../../sda
```
- [ ] **4.4** Set variables for convenience
```
NEW_DISKID="ata-ST4000NT001-3M2101_WX120LHQ"
NEW_DISKPATH="/dev/disk/by-id/$NEW_DISKID"
OLD_DISKID="ata-ST4000NT001-3M2101_WX11TN0Z"
echo "New: $NEW_DISKID -> $(readlink -f $NEW_DISKPATH)"
```
- [x] **4.5** Confirm drive identity and firmware version with smartctl
```
smartctl -i "$NEW_DISKPATH"
```
Expected: Model ST4000NT001-3M2101, Serial WX120LHQ, Firmware EN01, 4TB capacity.
```
Device Model: ST4000NT001-3M2101
Serial Number: WX120LHQ
Firmware Version: EN01
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
```
- [x] **4.6** Check kernel logs are clean — no SATA errors, link drops, or speed downgrades
```
dmesg -T | grep -E 'ata[0-9]' | grep -iE 'error|fatal|reset|link down|slow|limiting'
```
Expected: nothing. If there are errors here on a brand new drive + known-good cable, **stop and investigate**.
```
[Fri Feb 20 22:57:06 2026] ata1: SATA link down (SStatus 0 SControl 300)
[Fri Feb 20 22:57:06 2026] ata2: SATA link down (SStatus 0 SControl 300)
```
Clean — ata1/ata2 are unused ports. No errors on ata3 or ata4.
---
## Phase 5: Health-check the new drive before trusting data to it
Don't resilver onto a DOA drive.
- [x] **5.1** SMART overall health
```
smartctl -H "$NEW_DISKPATH"
```
Expected: PASSED.
```
SMART overall-health self-assessment test result: PASSED
```
- [x] **5.2** Check SMART attributes baseline
```
smartctl -A "$NEW_DISKPATH" | grep -E 'Reallocated|Pending|Offline_Uncorrect|CRC|Error_Rate'
```
Expected: all counters at 0 (it's a new/refurb drive).
```
1 Raw_Read_Error_Rate ... - 6072
5 Reallocated_Sector_Ct ... - 0
7 Seek_Error_Rate ... - 476
197 Current_Pending_Sector ... - 0
198 Offline_Uncorrectable ... - 0
199 UDMA_CRC_Error_Count ... - 0
```
All critical counters at 0. Read/Seek error rate raw values are normal Seagate encoding.
- [x] **5.3** Run short self-test
```
smartctl -t short "$NEW_DISKPATH"
```
Wait ~2 minutes, then check:
```
smartctl -l selftest "$NEW_DISKPATH"
```
Expected: "Completed without error".
```
# 1 Short offline Completed without error 00% 0 -
```
Passed. 0 power-on hours — fresh drive.
- [x] **5.4** (Decision point) Short test passed. Proceeding.
---
## Phase 6: Add new drive to ZFS mirror
- [x] **6.1** Open a dedicated terminal for kernel log monitoring
```
dmesg -Tw
```
Leave this running throughout the resilver. Watch for ANY `ata` errors.
- [x] **6.2** Replace the old drive with the new one in the pool
```
zpool replace proxmox-tank-1 "$OLD_DISKID" "$NEW_DISKID"
```
This tells ZFS: "the REMOVED drive WX11TN0Z is being replaced by WX120LHQ". Resilvering starts automatically.
- [x] **6.3** Verify resilvering has started
```
zpool status -v proxmox-tank-1
```
Expected: state DEGRADED, new drive shows as part of a `replacing` vdev, resilver in progress.
```
resilver in progress since Fri Feb 20 23:10:58 2026
5.71G / 1.33T scanned at 344M/s, 0B / 1.33T issued
0B resilvered, 0.00% done
replacing-0 DEGRADED 0 0 0
ata-ST4000NT001-3M2101_WX11TN0Z REMOVED 0 0 0
ata-ST4000NT001-3M2101_WX120LHQ ONLINE 0 0 7.73K
ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0
```
Resilver running. Cksum count on new drive is expected during resilver (unwritten blocks).
- [x] **6.4** Monitor resilver progress periodically
```
watch -n 30 "zpool status -v proxmox-tank-1"
```
Expected: steady progress, no read/write/cksum errors on either drive. Based on previous experience (~500GB at ~100MB/s with VMs down), expect roughly 1-2 hours.
VMs were auto-started on boot. Resilver completed: 1.34T in 03:32:55 with 0 errors.
- [x] **6.5** VMs were already running (auto-start on boot).
---
## Phase 7: Post-resilver verification
Wait for resilver to complete (status will say "resilvered XXG in HH:MM:SS with 0 errors").
- [x] **7.1** Check final pool status
```
zpool status -v proxmox-tank-1
```
Expected: ONLINE (or DEGRADED with "too many errors" message requiring a clear — same as last time).
```
state: ONLINE
scan: resilvered 1.34T in 03:32:55 with 0 errors on Sat Feb 21 02:43:53 2026
ata-ST4000NT001-3M2101_WX120LHQ ONLINE 0 0 7.73K
ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0
```
ONLINE. 7.73K cksum on TOMMY is expected resilver artifact — clearing next.
- [x] **7.2** Clear residual cksum counters
```
zpool clear proxmox-tank-1
```
Counters cleared (status message and cksum count gone on re-check).
```
state: ONLINE
scan: resilvered 1.34T in 03:32:55 with 0 errors on Sat Feb 21 02:43:53 2026
ata-ST4000NT001-3M2101_WX120LHQ ONLINE 0 0 0
ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0
errors: No known data errors
```
- [x] **7.3** Run a full scrub to verify data integrity
```
zpool scrub proxmox-tank-1
```
Expected: **0 errors on both drives**.
```
scrub repaired 0B in 03:27:50 with 0 errors on Sat Feb 21 11:38:02 2026
ata-ST4000NT001-3M2101_WX120LHQ ONLINE 0 0 0
ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0
errors: No known data errors
```
- [x] **7.4** Clean status confirmed — 0B repaired, 0 errors, both drives 0/0/0.
- [x] **7.5** Baseline SMART snapshot of the new drive after heavy I/O
```
smartctl -x "$NEW_DISKPATH" | grep -E 'Reallocated|Pending|Offline_Uncorrect|CRC|Hardware Resets|COMRESET|Interface'
```
Expected: 0 reallocated, 0 CRC errors, low hardware reset count.
```
Reallocated_Sector_Ct ... 0
Current_Pending_Sector ... 0
Offline_Uncorrectable ... 0
UDMA_CRC_Error_Count ... 0
Number of Hardware Resets ... 2
Number of Interface CRC Errors ... 0
COMRESET ... 2
```
All clean. 2 hardware resets / COMRESETs from boot — normal.
- ~~**7.6**~~ Skipped — extended SMART self-test is redundant after a clean resilver + scrub. ZFS checksums already verified every data block; the only thing the long test would cover is empty space that ZFS hasn't written to, which ZFS will verify on future use anyway.
---
## Phase 8: Final state — done
- [x] **8.1** Final pool status — already captured in 7.4. Mirror is healthy:
```
pool: proxmox-tank-1
state: ONLINE
scan: scrub repaired 0B in 03:27:50 with 0 errors on Sat Feb 21 11:38:02 2026
config:
NAME STATE READ WRITE CKSUM
proxmox-tank-1 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-ST4000NT001-3M2101_WX120LHQ ONLINE 0 0 0
ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0
errors: No known data errors
```
- [x] **8.2** All VMs running normally — verified from Proxmox UI
- [x] **8.3** Celebrate. Mirror is whole again.
---
## Abort conditions
Stop and investigate if any of these happen:
- New drive not detected after boot (bad seating or DOA)
- SATA errors in `dmesg` during or after boot (bad cable? bad drive?)
- SMART short test fails on new drive (DOA — contact seller)
- Resilver stalls or produces errors on the new drive
- Scrub finds checksum errors on the new drive
---
## Execution summary
Executed 2026-02-20 evening through 2026-02-21 morning. No abort conditions hit — completely clean run.
- TOMMY (`WX120LHQ`) installed on ata3 at 6.0 Gbps, detected first boot
- SMART short test passed, all critical attributes at zero
- Resilver: 1.34T in 03:32:55, 0 errors (VMs were running — auto-start on boot)
- Scrub: repaired 0B in 03:27:50, 0 errors, both drives 0/0/0
- Post-I/O SMART baseline clean: 0 reallocated, 0 CRC errors
- Extended SMART test skipped — redundant after clean resilver + scrub (ZFS checksums already verified all data blocks)
- Pool `proxmox-tank-1` fully healthy. Mirror degradation that started 2026-02-08 is resolved.

4
README.md
View file

@ -8,6 +8,4 @@ The `index.html` is ready in the `public` folder.
## How to deploy ## How to deploy
1. Copy `deploy.config.example` to `deploy.config` Somehow get the `public` folder behind a webserver manually and sort out DNS.
2. Fill in your server details in `deploy.config` (host, user, remote path)
3. Run `./deploy.sh` to sync the `public` folder to your remote webserver

292
dbt_model_optimization.md
View file

@ -1,292 +0,0 @@
# Busy man’s guide to optimizing dbt models performance
You have a `dbt` model that takes ages to run in production. For some very valid reason, this is a problem.
This is a small reference guide on things you can try. I suggest you try them from start to end, since they are sorted in a descending way by value/complexity ratio.
Before you start working on a model, you might want to check [the bonus guide at the bottom](Busy%20man%E2%80%99s%20guide%20to%20optimizing%20dbt%20models%20performa%20b0540bf8fa0a4ca5a6220b9d8132800d.md) to learn how to make sure you don’t change the outputs of a model while refactoring it.
If you’ve tried everything you could here and things still don’t work, don’t hesitate to call Pablo.
## 1. Is your model *really* taking too long?
> Before you optimize a model that is taking too long, make sure it actually takes too long.
>
The very first step is to really assess if you do have a problem.
We run our DWH in a Postgres server, and Postgres is a complex system. Postgres is doing many things at all times and it’s very stateful, which means you will pretty much never see *exactly* the same performance twice for some given query.
Before going crazy optimizing, I would advice running the model or the entire project a few times and observing the behaviour. It might be that *some day* it took very long for some reason, but usually, it runs just fine.
You also might want to do this in a moment where there’s little activity in the DWH, like very early or late in the day, so that other users’ activity in the DWH don’t pollute your observations.
If this is a model that is already being run regularly already, we can also leverage the statistics collected by the `pg_stat_statements` Postgres extension to check what are the min, avg, and max run times for it. Ask Pablo to get this.
## 2. Reducing the amount of data
> Make your query only bring in the data it needs, and not more. Reduce the amount of data as early as possible.
>
This option is a simple optimization trick that can be used in many areas and it’s easy to pull off.
The two holy devils of slow queries are large amounts of data and monster lookups/sorts. Both can be drastically reduced by simply reducing the amount of data that goes into the query, typically by applying some smart `WHERE` or creative conditions on a `JOIN` clause. This can be either done in your basic CTEs where you read from other models, or in the main `SELECT` of your model.
Typically, try to make this as *early* as possible in the model. Early here refers to the steps of your query. In your queries, you will typically:
- read a few tables,
- do some `SELECTs`
- then do more crazy logic downstream with more `SELECTs`
- and the party goes on for as long and complex your case is
Reducing the amount of data at the end is pointless. You will still need to read a lot of stuff early and have monster `JOIN`s , window functions, `DISTINCTs`, etc. Ideally, you want to do it when your first access an upstream table. If not there, then as early as possible within the logic.
The specifics of how to apply this are absolutely query dependent, so I can’t give you magic instructions for the query you have at hand. But let me illustrate the concept with an example:
### Only hosts? Then only hosts
You have a table `stg_my_table` with a lot of data, let’s say 100 million records, and each record has the id of a host. In your model, you need to join these records with the host user data to get some columns from there. So right now your query looks something like this (tables fictional, this is not how things look in DWH):
```sql
with
stg_my_table as (select * from {{ ref("stg_my_table") }}),
stg_users as (select * from {{ ref("stg_users")}})
select
...
from stg_my_table t
left join
stg_users u
on t.id_host_user = id_user
```
At the time I’m writing this, the real user table in our DWH has like 600,000 records. This means that:
- The CTE `stg_users` will need to fetch 600,000 records, with all their data, and store them.
- Then the left join will have to join 100 million records from `my_table` with the 600,000 user records.
Now, this is not working for you because it takes ages. We can easily improve the situation by applying the principle of this section: reducing the amount of data.
Our user table in the DWH has both hosts and guests. Actually, it has a ~1,000 hosts and everything else is just guests. This means that:
- We’re fetching around 599,000 guest details that we don’t care about at all.
- Every time we join a record from `my_table`, we do so against 600,000 user records when we only truly care about 1,000 of them.
Stupid, isn’t it?
Well, imagining that our fictional `stg_users` tables had a field called `is_host`, we can rewrite the query this way to get exactly the same result in only a fraction of the time:
```sql
with
stg_my_table as (select * from {{ ref("stg_my_table") }}),
**stg_users as (
select *
from {{ ref("stg_users")}}
where is_host = true
)**
select
...
from stg_my_table t
left join
stg_users u
on t.id_host_user = id_user
```
It’s simple to understand: the CTE will now only get the 1,000 records related to hosts, which means we save performance in both fetching that data and having a much smaller join operation downstream against `stg_my_table`.
## 3. Controlling CTE materialization
> Tell Postgres when to cache intermediate results and when to optimize through them.
>
This one requires a tiny bit of understanding of what happens under the hood, but the payoff is big and the fix is easy to apply.
### What Postgres does with your CTEs
When Postgres runs a CTE, it has two strategies:
- **Materialized**: Postgres runs the CTE query, stores the full result in a temporary buffer, and every downstream reference reads from that buffer. Think of it as Postgres creating a temporary, index-less table with the CTE's output.
- **Not materialized**: Postgres treats the CTE as if it were a view. It doesn't store anything — instead, it folds the CTE's logic into the rest of the query and optimizes everything together. This means it can push filters down, use indexes from the original tables, and skip reading rows it doesn't need.
By default, Postgres decides for you: if a CTE is referenced once, it inlines it. If it's referenced more than once, it materializes it.
The problem is that this default isn't always ideal, especially with how we write dbt models.
### Why this matters for our dbt models
Following our conventions, we always import upstream refs as CTEs at the top of the file:
```sql
with
stg_users as (select * from {{ ref("stg_users") }}),
stg_bookings as (select * from {{ ref("stg_bookings") }}),
some_intermediate_logic as (
select ...
from stg_users
join stg_bookings on ...
where ...
),
some_other_logic as (
select ...
from stg_users
where ...
)
select ...
from some_intermediate_logic
join some_other_logic on ...
```
Notice that `stg_users` is referenced twice — once in `some_intermediate_logic` and once in `some_other_logic`. This means Postgres will materialize it by default. What happens then is:
1. Postgres scans the entire `stg_users` table and copies all 600,000 rows into a temporary buffer.
2. If the buffer exceeds available memory, it spills to disk.
3. Every downstream CTE that reads from `stg_users` does a sequential scan of that buffer. Note this means indices can't be used, even if the original table had them.
4. Any filters that downstream CTEs apply to `stg_users` (like `where is_host = true`) can't be pushed down to the original table scan. Postgres reads all 600,000 rows first, stores them, and only then filters.
All of that, for a `select *` that does absolutely no computation worth caching.
### The fix
You can explicitly control this behaviour by adding `MATERIALIZED` or `NOT MATERIALIZED` to any CTE:
```sql
with
stg_users as not materialized (select * from {{ ref("stg_users") }}),
stg_bookings as not materialized (select * from {{ ref("stg_bookings") }}),
some_intermediate_logic as (
...
),
some_other_logic as (
...
)
select ...
```
With `NOT MATERIALIZED`, Postgres treats those import CTEs as transparent aliases. It can see straight through to the original table, use its indexes, and push filters down.
### When to use which
The rule of thumb is simple:
- **Cheap CTE, referenced multiple times** → `NOT MATERIALIZED`. This is the typical case for our import CTEs at the top of the file. There's no computation to cache, so materializing just wastes resources.
- **Expensive CTE, referenced multiple times** → leave it alone (or explicit `MATERIALIZED`). If a CTE does heavy aggregations, complex joins, or window functions, materializing means that work happens once. Without it, Postgres would repeat the expensive query every time the CTE is referenced.
- **Any CTE referenced only once** → doesn't matter. Postgres inlines it automatically.
If you're unsure whether a CTE is "expensive enough" to warrant materialization, just try both and measure. There's no shame in that.
## 4. Change upstream materializations
> Materialize upstream models as tables instead of views to reduce computation on the model at hand.
>
Going back to basics, dbt offers [multiple materializations strategies for our models](https://docs.getdbt.com/docs/build/materializations).
Typically, for reasons that we won’t cover here, the preferred starting point is to use views. We only go for tables or incremental materializations if there are good reasons for this.
If you have a model that is having terrible performance, it’s possible that the fault doesn’t sit at the model itself, but rather at an upstream model. Let me make an example.
Imagine we have a situation with three models:
- `stg_my_simple_model`: a model with super simple logic and small data
- `stg_my_crazy_model`: a model with a crazy complex query and lots of data
- `int_my_dependant_model`: an int model that reads from both previous models.
- Where the staging models are set to materialize as views and the int model is set to materialize as a table.
Because the two staging models are set to materialize as views, this means that every time you run `int_my_dependant_model`, you will also have to execute the queries of `stg_my_simple_model` and `stg_my_crazy_model`. If the upstream views model are fast, this is not an issue of any kind. But if a model is a heavy query, this could be an issue.
The point is, you might notice that `int_my_dependant_model` takes 600 seconds to run and think there’s something wrong with it, when actually the fault sits at `stg_my_crazy_model`, which perhaps is taking 590 seconds out of the 600.
How can materializations solve this? Well, if `stg_my_crazy_model` was materialized as a table instead of as view, whenever you ran `int_my_dependant_model` you would simply read from a table with pre-populated results, instead of having to run the `stg_my_crazy_model` query each time. Typically, reading the results will be much faster than running the whole query. So, in summary, by making `stg_my_crazy_model` materialize as a table, you can fix your performance issue in `int_my_dependant_model`.
## 5. Switch the model to materialization to `incremental`
> Make the processing of the table happen in small batches instead of on all data to make it more manageable.
>
Imagine we want to count how many bookings where created each month.
As time passes, more and more months and more and more bookings appear in our history, making the size of this problem ever increasing. But then again, once a month has finished, we shouldn’t need to go back and revisit history: what’s done is done, and only the ongoing month is relevant, right?
[dbt offers a materialization strategy named](https://docs.getdbt.com/docs/build/incremental-models) `incremental`, which allows you to only work on a subset of data. this means that every time you run `dbt run` , your model only works on a certain part of the data, and not all of it. If the nature of your data and your needs allows isolating each run to a small part of all upstream data, this strategy can help wildly improve the performance.
Explaining the inner details of `incremental` goes beyond the scope of this page. You can check the official docs from `dbt` ([here](https://docs.getdbt.com/docs/build/incremental-models)), ask the team for support or check some of the incremental models that we already have in our project and use them as references.
Note that using `incremental` strategies makes life way harder than simple `view` or `table` ones, so only pick this up if it’s truly necessary. Don’t make models incremental without trying other optimizations first, or simply because you realise that you *could* use it. in a specific model.
![dbt’s official docs (wisely) warning you of the dangers of incremental.](image%2039.png)
dbt’s official docs (wisely) warning you of the dangers of incremental.
## 6. End of the line: general optimization
The final tip is not really a tip. The above five things are the easy-peasy, low hanging fruit stuff that you can try. This doesn’t mean that there isn’t more than you can do, just that I don’t know of more simple stuff that you can try without deep knowledge of how Postgres works beneath and a willingness to get your hands *real* dirty.
If you’ve reached this point and your model still performing poorly, you either need to put your Data Engineer hat on and really deepen your knowledge… or call Pablo.
## Bonus: how to make sure you didn’t screw up and change the output of the model
The topic we are discussing in this guide is making refactors purely for the sake of performance, without changing the output of the given model. We simply want to make the model faster, not change what data it generates.
That being the case, and considering the complexity of the strategies we’ve presented here, being afraid that you messed up and accidentally changed the output of the model is a very reasonable fear to have. That’s a kind of mistake that we definitely want to avoid.
Doing this manually can be a PITA and very time consuming, which doesn’t help at all.
To make your life easier, I’m going to show you a new little trick.
### Hashing tables and comparing them
I’ll post a snippet of code here that you can run to compare if any pair of tables has *exactly* the same contents. Emphasis on exactly. Changing the slightest bit of content will be detected.
```sql
SELECT md5(array_agg(md5((t1.*)::varchar))::varchar)
FROM (
SELECT *
FROM my_first_table
ORDER BY <whatever field is unique>
) AS t1
SELECT md5(array_agg(md5((t2.*)::varchar))::varchar)
FROM (
SELECT *
FROM my_second_table
ORDER BY <whatever field is unique>
) AS t2
```
How this works is: you execute the two queries, which will return a single value each. Some hexadecimal gibberish.
If the output of the two queries is identical, it means their contents are identical. If they are different, it means there’s something different across both.
If you don’t understand how this works, and you don’t care, that’s fine. Just use it.
If not knowing does bother, you should go down the rabbit holes of hash functions and deterministic serialization.
### Including this in your refactoring workflow
Right, now you know how to make sure that two tables are identical.
This is dramatically useful for your optimization workflow. You can know simply:
- Keep the original model
- Create a copy of it, which is the one you will be working on (the working copy)
- Prepare the magic query to check their contents are identical
- From this point on, you can enter in this loop for as long as you want/need:
- Run the magic query to ensure you start from same-output-state
- Modify the working copy model to attempt whatever optimization thingie you wanna try
- Once you are done, run the magic query again.
- If the output is not the same anymore, you screwed up. Start again and avoid whatever mistake you made.
- If the output is still the same, you didn’t cause a change in the model output. Either keep on optimizing or call it day.
- Finally, just copy over the working copy model code into the old one and remove the working copy.
I hope that helps. I also recommend doing the loop as frequently as possible. The less things you change between executions of the magic query, the easier is to realize what caused errors if they appear.
![ Donald Knuth - "[StructuredProgrammingWithGoToStatements](http://web.archive.org/web/20130731202547/http://pplab.snu.ac.kr/courses/adv_pl05/papers/p261-knuth.pdf)”](image%2040.png)
Donald Knuth - "[StructuredProgrammingWithGoToStatements](http://web.archive.org/web/20130731202547/http://pplab.snu.ac.kr/courses/adv_pl05/papers/p261-knuth.pdf)”

21
deploy.config.example
View file

@ -1,21 +0,0 @@
# Deployment Configuration
# Copy this file to deploy.config and fill in your server details
# deploy.config is gitignored to keep your credentials safe
# Remote server hostname or IP address
REMOTE_HOST="example.com"
# SSH username for the remote server
REMOTE_USER="username"
# Remote path where the website should be deployed
# This should be the directory served by your webserver (e.g., /var/www/html, /home/username/public_html)
REMOTE_PATH="/var/www/html"
# Optional: Path to SSH private key (if not using default ~/.ssh/id_rsa)
# Leave empty to use default SSH key
SSH_KEY=""
# Optional: SSH port (defaults to 22 if not specified)
# SSH_PORT="22"

34
deploy.sh
View file

@ -1,34 +0,0 @@
#!/bin/bash
# Deployment script for pablohere website
# This script syncs the public folder to a remote webserver
set -e # Exit on error
# Load deployment configuration
if [ ! -f "deploy.config" ]; then
echo "Error: deploy.config file not found!"
echo "Please copy deploy.config.example to deploy.config and fill in your server details."
exit 1
fi
source deploy.config
# Validate required variables
if [ -z "$REMOTE_HOST" ] || [ -z "$REMOTE_USER" ] || [ -z "$REMOTE_PATH" ]; then
echo "Error: Required variables not set in deploy.config"
echo "Please ensure REMOTE_HOST, REMOTE_USER, and REMOTE_PATH are set."
exit 1
fi
# Use rsync to sync files
echo "Deploying public folder to $REMOTE_USER@$REMOTE_HOST:$REMOTE_PATH"
rsync -avz --delete \
--exclude='.git' \
--exclude='.DS_Store' \
$SSH_OPTS \
public/ \
$REMOTE_USER@$REMOTE_HOST:$REMOTE_PATH
echo "Deployment complete!"

31
public/index.html
View file

@ -123,9 +123,6 @@
<a href="https://github.com/pmartincalvo/dni" target="_blank" rel="noopener noreferrer">My Python package to <a href="https://github.com/pmartincalvo/dni" target="_blank" rel="noopener noreferrer">My Python package to
handle Spanish DNIs better</a> handle Spanish DNIs better</a>
</li> </li>
<li>
<a href="https://bitcoininfra.contrapeso.xyz" target="_blank" rel="noopener noreferrer">My open access Bitcoin infrastructure that you can use freely.</a> It includes access to the peer port of my Bitcoin node, an Electrum server and a mempool.space instance.
</li>
</ul> </ul>
<p> <p>
There are also some other projects that I generally keep private but There are also some other projects that I generally keep private but
@ -147,34 +144,6 @@
<h2 id="writings-header">Writings</h2> <h2 id="writings-header">Writings</h2>
<p>Sometimes I like to jot down ideas and drop them here.</p> <p>Sometimes I like to jot down ideas and drop them here.</p>
<ul> <ul>
<li>
<a href="writings/my-fitness-journey.html" target="_blank"
rel="noopener noreferrer">My fitness journey</a>
</li>
<li>
<a href="writings/how-i-write-some-articles-im-lazy-about.html" target="_blank"
rel="noopener noreferrer">How I write some articles I have a hard time getting started with</a>
</li>
<li>
<a href="writings/replacing-a-failed-disk-in-a-zfs-mirror.html" target="_blank"
rel="noopener noreferrer">Replacing a Failed Disk in a ZFS Mirror</a>
</li>
<li>
<a href="writings/busy-mans-guide-to-optimizing-dbt-models-performance.html" target="_blank"
rel="noopener noreferrer">Busy man's guide to optimizing dbt models performance</a>
</li>
<li>
<a href="writings/fixing-a-degraded-zfs-mirror.html" target="_blank"
rel="noopener noreferrer">Fixing a Degraded ZFS Mirror: Reseat, Resilver, and Scrub</a>
</li>
<li>
<a href="writings/a-degraded-pool-with-a-healthy-disk.html" target="_blank"
rel="noopener noreferrer">A degraded pool with a healthy disk</a>
</li>
<li>
<a href="writings/why-i-put-my-vms-on-a-zfs-mirror.html" target="_blank"
rel="noopener noreferrer">Why I Put My VMs on a ZFS Mirror</a>
</li>
<li> <li>
<a href="writings/a-note-for-the-future-the-tax-bleeding-in-2025.html" target="_blank" <a href="writings/a-note-for-the-future-the-tax-bleeding-in-2025.html" target="_blank"
rel="noopener noreferrer">A note for the future: the tax bleeding in 2025</a> rel="noopener noreferrer">A note for the future: the tax bleeding in 2025</a>

11
public/styles.css
View file

@ -7,8 +7,7 @@ body {
h1, h1,
h2, h2,
h3, h3 {
h4 {
text-align: center; text-align: center;
} }
@ -26,12 +25,4 @@ figcaption {
font-style: italic; font-style: italic;
font-size: small; font-size: small;
text-align: center; text-align: center;
}
blockquote {
border-left: 3px solid #888;
margin-left: 0;
padding-left: 1em;
font-style: italic;
color: #555;
} }

133
public/writings/a-degraded-pool-with-a-healthy-disk.html
View file

@ -1,133 +0,0 @@
<!DOCTYPE HTML>
<html>
<head>
<title>Pablo here</title>
<meta charset="utf-8">
<meta viewport="width=device-width, initial-scale=1">
<link rel="stylesheet" href="../styles.css">
</head>
<body>
<main>
<h1>
Hi, Pablo here
</h1>
<p><a href="../index.html">back to home</a></p>
<section>
<h2>A degraded pool with a healthy disk</h2>
<p><em>Part 2 of 3 in my "First ZFS Degradation" series. See also <a href="why-i-put-my-vms-on-a-zfs-mirror.html">Part 1: The Setup</a> and <a href="fixing-a-degraded-zfs-mirror.html">Part 3: The Fix</a>.</em></p>
<h3>The "Oh Shit" Moment</h3>
<p>I wasn't even looking for trouble. I was clicking around the Proxmox web UI, exploring some storage views I hadn't noticed before, when I saw it: my ZFS pool was in <strong>DEGRADED</strong> state.</p>
<p>I opened the details. One of my two mirrored drives was listed as <strong>FAULTED</strong>.</p>
<p>I was very surprised. This box and disks were brand new and didn't even have three months of running on them. I was not expecting HW issues to come at me that fast. I SSH'd into the server and ran the command that would become my best friend over the next 24 hours:</p>
<pre><code>zpool status -v proxmox-tank-1</code></pre>
<p>No glitch. The pool was degraded. The drive had racked up over 100 read errors, 600+ write errors, and 129 checksum errors. ZFS had given up on it.</p>
<pre><code> NAME STATE READ WRITE CKSUM
proxmox-tank-1 DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
ata-ST4000NT001-3M2101_WX11TN0Z FAULTED 108 639 129 too many errors
ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0</code></pre>
<p>The good news: <code>errors: No known data errors</code>. ZFS was serving all my data from the healthy drive. Nothing was lost yet.</p>
<p>The bad news: I was running on a single point of failure. If AGAPITO2 decided to have a bad day too, I'd be in real trouble.</p>
<p>I tried the classic IT move: rebooting. The system came back up and ZFS immediately started trying to resilver (rebuild) the degraded drive. But within minutes, the errors started piling up again and the resilver stalled.</p>
<p>Time to actually figure out what was wrong.</p>
<h3>The Diagnostic Toolbox</h3>
<p>When a ZFS drive acts up, you have two main sources of truth: what the <strong>kernel</strong> sees happening at the hardware level, and what the <strong>drive itself</strong> reports about its health. This can be looked up with <code>dmesg</code> and <code>smartctl</code>.</p>
<h4>dmesg: The Kernel's Diary</h4>
<p>The Linux kernel maintains a ring buffer of messages about hardware events, driver activities, and system operations. The <code>dmesg</code> command lets you read it. For disk issues, you want to grep for SATA-related keywords:</p>
<pre><code>dmesg -T | egrep -i 'ata[0-9]|sata|reset|link|i/o error' | tail -100</code></pre>
<p>The <code>-T</code> flag gives you human-readable timestamps instead of seconds-since-boot.</p>
<p>What I saw was... weird. Here's an excerpt:</p>
<pre><code>[Fri Jan 2 22:25:13 2026] ata4.00: exception Emask 0x50 SAct 0x70220001 SErr 0xe0802 action 0x6 frozen
[Fri Jan 2 22:25:13 2026] ata4.00: irq_stat 0x08000000, interface fatal error
[Fri Jan 2 22:25:13 2026] ata4.00: failed command: READ FPDMA QUEUED
[Fri Jan 2 22:25:13 2026] ata4: hard resetting link
[Fri Jan 2 22:25:14 2026] ata4: SATA link down (SStatus 0 SControl 300)</code></pre>
<p>Let me translate: the kernel tried to read from the drive on <code>ata4</code>, got a "fatal error," and responded by doing a hard reset of the SATA link. Then the link went down entirely. The drive just... disappeared.</p>
<p>But it didn't stay gone. A few seconds later:</p>
<pre><code>[Fri Jan 2 22:25:24 2026] ata4: link is slow to respond, please be patient (ready=0)
[Fri Jan 2 22:25:24 2026] ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300)</code></pre>
<p>The drive came back! At full speed! But then...</p>
<pre><code>[Fri Jan 2 22:25:29 2026] ata4.00: qc timeout after 5000 msecs (cmd 0xec)
[Fri Jan 2 22:25:29 2026] ata4.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[Fri Jan 2 22:25:29 2026] ata4: limiting SATA link speed to 3.0 Gbps</code></pre>
<p>It failed again. The kernel, trying to be helpful, dropped the link speed from 6.0 Gbps to 3.0 Gbps. Maybe a slower speed would be more stable?</p>
<p>It wasn't. The pattern repeated: connect, fail, reset, reconnect at a slower speed. 6.0 Gbps, then 3.0 Gbps, then 1.5 Gbps. Eventually:</p>
<pre><code>[Fri Jan 2 22:27:06 2026] ata4.00: disable device</code></pre>
<p>The kernel gave up entirely.</p>
<p>This wasn't what a dying drive looks like. A dying drive throws read errors on specific bad sectors. This drive was connecting and disconnecting like someone was jiggling the cable. The kernel was calling it "interface fatal error", emphasis on <em>interface</em>.</p>
<h4>smartctl: Asking the Drive Directly</h4>
<p>Every modern hard drive has S.M.A.R.T. (Self-Monitoring, Analysis, and Reporting Technology) — basically a built-in health monitor. The <code>smartctl</code> command lets you get info out of it.</p>
<p>First, the overall health check:</p>
<pre><code>smartctl -H /dev/sdb</code></pre>
<pre><code>SMART overall-health self-assessment test result: PASSED</code></pre>
<p>Okay, that looks great. But if the disk is healthy, what the hell is going on, and where are all those errors that ZFS was spotting coming from?</p>
<p>Let's dig deeper with the extended info:</p>
<pre><code>smartctl -x /dev/sdb</code></pre>
<p>The key attributes I was looking for:</p>
<table>
<thead>
<tr>
<th>Attribute</th>
<th>Value</th>
<th>What it means</th>
</tr>
</thead>
<tbody>
<tr>
<td>Reallocated_Sector_Ct</td>
<td>0</td>
<td>Bad sectors the drive has swapped out. Zero is good.</td>
</tr>
<tr>
<td>Current_Pending_Sector</td>
<td>0</td>
<td>Sectors waiting to be checked. Zero is good.</td>
</tr>
<tr>
<td>UDMA_CRC_Error_Count</td>
<td>0</td>
<td>Data corruption during transfer. Zero is good.</td>
</tr>
<tr>
<td>Number of Hardware Resets</td>
<td>39</td>
<td>Times the connection has been reset. Uh...</td>
</tr>
</tbody>
</table>
<p>All the sector-level health metrics looked perfect. No bad blocks, no pending errors, no CRC errors. The drive's magnetic platters and read/write heads were fine.</p>
<p>But 39 hardware resets? That's not normal. That's the drive (or its connection) getting reset nearly 40 times.</p>
<p>I ran the short self-test to be sure:</p>
<pre><code>smartctl -t short /dev/sdb
# Wait a minute...
smartctl -l selftest /dev/sdb</code></pre>
<pre><code># 1 Short offline Completed without error 00%</code></pre>
<p>The drive passed its own self-test. The platters spin, the heads move, the firmware works, and it can read its own data just fine.</p>
<h3>Hypothesis</h3>
<p>At this point, the evidence was pointing clearly away from "the drive is dying" and toward "something is wrong with the connection."</p>
<p>What the kernel logs told me: the drive keeps connecting and disconnecting. Each time it reconnects, the kernel tries slower speeds. Eventually it gives up entirely. This is what you see with an unstable physical connection.</p>
<p>What SMART told me: the drive itself is healthy. No bad sectors, no media errors, no signs of wear. But there have been dozens of hardware resets — the connection keeps getting interrupted.</p>
<p>The suspects, in order of likelihood:</p>
<ol>
<li><strong>SATA data cable</strong>: the most common culprit for intermittent connection issues. Cables go bad, or weren't seated properly in the first place.</li>
<li><strong>Power connection</strong>: if the drive isn't getting stable power, it might brown out intermittently.</li>
<li><strong>SATA port on the motherboard</strong>: less likely, but possible.</li>
<li><strong>PSU</strong>: power supply issues could affect the power rail feeding the drive. Unlikely, since both disks where feeding from the same cable tread, but still an option.</li>
</ol>
<p>Given that I had just built this server a few weeks earlier, and a good part of that happened after midnight... I was beginning to suspect that perhaps I simply might not have plugged in the disk properly.</p>
<h3>The Verdict</h3>
<p>I was pretty confident now: the drive was fine, but the connection was bad. Most likely the SATA data cable, and most probably simply not connected properly.</p>
<p>The fix would require shutting down the server, opening the case, and reseating (or replacing) cables. Before doing that, I wanted to take the drive offline cleanly and document everything.</p>
<p>In <a href="fixing-a-degraded-zfs-mirror.html">Part 3</a>, I'll walk through exactly how I fixed it: the ZFS commands, the physical work, and the validation to make sure everything was actually okay afterward.</p>
<p><em>Continue to <a href="fixing-a-degraded-zfs-mirror.html">Part 3: The Fix</a> →</em></p>
<p><a href="../index.html">back to home</a></p>
</section>
</main>
</body>
</html>

358
public/writings/busy-mans-guide-to-optimizing-dbt-models-performance.html
View file

@ -1,358 +0,0 @@
<!DOCTYPE HTML>
<html>
<head>
<title>Pablo here</title>
<meta charset="utf-8">
<meta viewport="width=device-width, initial-scale=1">
<link rel="stylesheet" href="../styles.css">
</head>
<body>
<main>
<h1>
Hi, Pablo here
</h1>
<p><a href="../index.html">back to home</a></p>
<hr>
<section>
<h2>Busy man's guide to optimizing dbt models performance</h2>
<p>The below guide is a copy-paste of an internal doc file I created while working in Superhog. My team of
analysts were very smart guys, but had little knowledge on Postgres internals (we used Postgres for our
DWH) and low-level query optimization. That was understandable: they were analysts, busy with answering
business questions. Their value was not in fixing pipeline performance.</p>
<p>Nevertheless, giving them some degree of freedom in fixing performance was both great to avoid me
becoming a bottleneck and to also expand their knowledge, which they were eager to. This guide was
targeted to them, and the goal was to give them as many tools as possible without having to go into the
rabbit hole of <code>EXPLAIN ANALYZE</code>.</p>
<hr>
<p>You have a <code>dbt</code> model that takes ages to run in production. For some very valid reason, this
is a problem.</p>
<p>This is a small reference guide on things you can try. I suggest you try them from start to end, since
they are sorted in a descending way by value/complexity ratio.</p>
<p>Before you start working on a model, you might want to check <a href="#bonus">the bonus guide at the
bottom</a> to learn how to make sure you don't change the outputs of a model while refactoring it.
</p>
<p>If you've tried everything you could here and things still don't work, don't hesitate to call Pablo.</p>
<h3>1. Is your model <em>really</em> taking too long?</h3>
<blockquote>Before you optimize a model that is taking too long, make sure it actually takes too long.
</blockquote>
<p>The very first step is to really assess if you do have a problem.</p>
<p>We run our DWH in a Postgres server, and Postgres is a complex system. Postgres is doing many things at
all times and it's very stateful, which means you will pretty much never see <em>exactly</em> the same
performance twice for some given query.</p>
<p>Before going crazy optimizing, I would advise running the model or the entire project a few times and
observing the behaviour. It might be that <em>some day</em> it took very long for some reason, but
usually, it runs just fine.</p>
<p>You also might want to do this in a moment where there's little activity in the DWH, like very early or
late in the day, so that other users' activity in the DWH don't pollute your observations.</p>
<p>If this is a model that is already being run regularly, we can also leverage the statistics
collected by the <code>pg_stat_statements</code> Postgres extension to check what are the min, avg, and
max run times for it. Ask Pablo to get this.</p>
<h3>2. Reducing the amount of data</h3>
<blockquote>Make your query only bring in the data it needs, and not more. Reduce the amount of data as
early as possible.</blockquote>
<p>This option is a simple optimization trick that can be used in many areas and it's easy to pull off.</p>
<p>The two holy devils of slow queries are large amounts of data and monster lookups/sorts. Both can be
drastically reduced by simply reducing the amount of data that goes into the query, typically by applying
some smart <code>WHERE</code> or creative conditions on a <code>JOIN</code> clause. This can be either
done in your basic CTEs where you read from other models, or in the main <code>SELECT</code> of your
model.</p>
<p>Typically, try to make this as <em>early</em> as possible in the model. Early here refers to the steps
of your query. In your queries, you will typically:</p>
<ul>
<li>read a few tables,</li>
<li>do some <code>SELECTs</code></li>
<li>then do more crazy logic downstream with more <code>SELECTs</code></li>
<li>and the party goes on for as long and complex your case is</li>
</ul>
<p>Reducing the amount of data at the end is pointless. You will still need to read a lot of stuff early and
have monster <code>JOIN</code>s , window functions, <code>DISTINCT</code>s, etc. Ideally, you want to do
it when you first access an upstream table. If not there, then as early as possible within the logic.
</p>
<p>The specifics of how to apply this are absolutely query dependent, so I can't give you magic instructions
for the query you have at hand. But let me illustrate the concept with an example:</p>
<h4>Only hosts? Then only hosts</h4>
<p>You have a table <code>stg_my_table</code> with a lot of data, let's say 100 million records, and each
record has the id of a host. In your model, you need to join these records with the host user data to
get some columns from there. So right now your query looks something like this (tables fictional, this
is not how things look in DWH):</p>
<pre><code>with
stg_my_table as (select * from {{ ref("stg_my_table") }}),
stg_users as (select * from {{ ref("stg_users")}})
select
...
from stg_my_table t
left join
stg_users u
on t.id_host_user = id_user</code></pre>
<p>At the time I'm writing this, the real user table in our DWH has like 600,000 records. This means
that:</p>
<ul>
<li>The CTE <code>stg_users</code> will need to fetch 600,000 records, with all their data, and store
them.</li>
<li>Then the left join will have to join 100 million records from <code>my_table</code> with the 600,000
user records.</li>
</ul>
<p>Now, this is not working for you because it takes ages. We can easily improve the situation by applying
the principle of this section: reducing the amount of data.</p>
<p>Our user table in the DWH has both hosts and guests. Actually, it has a ~1,000 hosts and everything else
is just guests. This means that:</p>
<ul>
<li>We're fetching around 599,000 guest details that we don't care about at all.</li>
<li>Every time we join a record from <code>my_table</code>, we do so against 600,000 user records when
we only truly care about 1,000 of them.</li>
</ul>
<p>Stupid, isn't it?</p>
<p>Well, imagining that our fictional <code>stg_users</code> tables had a field called
<code>is_host</code>, we can rewrite the query this way to get exactly the same result in only a
fraction of the time:
</p>
<pre><code>with
stg_my_table as (select * from {{ ref("stg_my_table") }}),
stg_users as (
select *
from {{ ref("stg_users")}}
where is_host = true
)
select
...
from stg_my_table t
left join
stg_users u
on t.id_host_user = id_user</code></pre>
<p>It's simple to understand: the CTE will now only get the 1,000 records related to hosts, which means we
save performance in both fetching that data and having a much smaller join operation downstream against
<code>stg_my_table</code>.
</p>
<h3>3. Controlling CTE materialization</h3>
<blockquote>Tell Postgres when to cache intermediate results and when to optimize through them.</blockquote>
<p>This one requires a tiny bit of understanding of what happens under the hood, but the payoff is big and
the fix is easy to apply.</p>
<h4>What Postgres does with your CTEs</h4>
<p>When Postgres runs a CTE, it has two strategies:</p>
<ul>
<li><strong>Materialized</strong>: Postgres runs the CTE query, stores the full result in a temporary
buffer, and every downstream reference reads from that buffer. Think of it as Postgres creating a
temporary, index-less table with the CTE's output.</li>
<li><strong>Not materialized</strong>: Postgres treats the CTE as if it were a view. It doesn't store
anything — instead, it folds the CTE's logic into the rest of the query and optimizes everything
together. This means it can push filters down, use indexes from the original tables, and skip
reading rows it doesn't need.</li>
</ul>
<p>By default, Postgres decides for you: if a CTE is referenced once, it inlines it. If it's referenced
more than once, it materializes it.</p>
<p>The problem is that this default isn't always ideal, especially with how we write dbt models.</p>
<h4>Why this matters for our dbt models</h4>
<p>Following our conventions, we always import upstream refs as CTEs at the top of the file:</p>
<pre><code>with
stg_users as (select * from {{ ref("stg_users") }}),
stg_bookings as (select * from {{ ref("stg_bookings") }}),
some_intermediate_logic as (
select ...
from stg_users
join stg_bookings on ...
where ...
),
some_other_logic as (
select ...
from stg_users
where ...
)
select ...
from some_intermediate_logic
join some_other_logic on ...</code></pre>
<p>Notice that <code>stg_users</code> is referenced twice — once in <code>some_intermediate_logic</code>
and once in <code>some_other_logic</code>. This means Postgres will materialize it by default. What
happens then is:</p>
<ol>
<li>Postgres scans the entire <code>stg_users</code> table and copies all 600,000 rows into a temporary
buffer.</li>
<li>If the buffer exceeds available memory, it spills to disk.</li>
<li>Every downstream CTE that reads from <code>stg_users</code> does a sequential scan of that buffer.
Note this means indices can't be used, even if the original table had them.</li>
<li>Any filters that downstream CTEs apply to <code>stg_users</code> (like
<code>where is_host = true</code>) can't be pushed down to the original table scan. Postgres reads
all 600,000 rows first, stores them, and only then filters.
</li>
</ol>
<p>All of that, for a <code>select *</code> that does absolutely no computation worth caching.</p>
<h4>The fix</h4>
<p>You can explicitly control this behaviour by adding <code>MATERIALIZED</code> or
<code>NOT MATERIALIZED</code> to any CTE:
</p>
<pre><code>with
stg_users as not materialized (select * from {{ ref("stg_users") }}),
stg_bookings as not materialized (select * from {{ ref("stg_bookings") }}),
some_intermediate_logic as (
...
),
some_other_logic as (
...
)
select ...</code></pre>
<p>With <code>NOT MATERIALIZED</code>, Postgres treats those import CTEs as transparent aliases. It can see
straight through to the original table, use its indexes, and push filters down.</p>
<h4>When to use which</h4>
<p>The rule of thumb is simple:</p>
<ul>
<li><strong>Cheap CTE, referenced multiple times</strong> → <code>NOT MATERIALIZED</code>. This is the
typical case for our import CTEs at the top of the file. There's no computation to cache, so
materializing just wastes resources.</li>
<li><strong>Expensive CTE, referenced multiple times</strong> → leave it alone (or explicit
<code>MATERIALIZED</code>). If a CTE does heavy aggregations, complex joins, or window functions,
materializing means that work happens once. Without it, Postgres would repeat the expensive query
every time the CTE is referenced.
</li>
<li><strong>Any CTE referenced only once</strong> → doesn't matter. Postgres inlines it automatically.
</li>
</ul>
<p>If you're unsure whether a CTE is "expensive enough" to warrant materialization, just try both and
measure. There's no shame in that.</p>
<h3>4. Change upstream materializations</h3>
<blockquote>Materialize upstream models as tables instead of views to reduce computation on the model at
hand.</blockquote>
<p>Going back to basics, dbt offers <a href="https://docs.getdbt.com/docs/build/materializations"
target="_blank" rel="noopener noreferrer">multiple materializations strategies for our models</a>.
</p>
<p>Typically, for reasons that we won't cover here, the preferred starting point is to use views. We only go
for tables or incremental materializations if there are good reasons for this.</p>
<p>If you have a model that is having terrible performance, it's possible that the fault doesn't sit at the
model itself, but rather at an upstream model. Let me make an example.</p>
<p>Imagine we have a situation with three models:</p>
<ul>
<li><code>stg_my_simple_model</code>: a model with super simple logic and small data</li>
<li><code>stg_my_crazy_model</code>: a model with a crazy complex query and lots of data</li>
<li><code>int_my_dependant_model</code>: an int model that reads from both previous models.</li>
<li>Where the staging models are set to materialize as views and the int model is set to materialize as
a table.</li>
</ul>
<p>Because the two staging models are set to materialize as views, this means that every time you run
<code>int_my_dependant_model</code>, you will also have to execute the queries of
<code>stg_my_simple_model</code> and <code>stg_my_crazy_model</code>. If the upstream views model are
fast, this is not an issue of any kind. But if a model is a heavy query, this could be an issue.
</p>
<p>The point is, you might notice that <code>int_my_dependant_model</code> takes 600 seconds to run and
think there's something wrong with it, when actually the fault sits at <code>stg_my_crazy_model</code>,
which perhaps is taking 590 seconds out of the 600.</p>
<p>How can materializations solve this? Well, if <code>stg_my_crazy_model</code> was materialized as a table
instead of as view, whenever you ran <code>int_my_dependant_model</code> you would simply read from a
table with pre-populated results, instead of having to run the <code>stg_my_crazy_model</code> query
each time. Typically, reading the results will be much faster than running the whole query. So, in
summary, by making <code>stg_my_crazy_model</code> materialize as a table, you can fix your performance
issue in <code>int_my_dependant_model</code>.</p>
<h3>5. Switch the model to materialization to <code>incremental</code></h3>
<blockquote>Make the processing of the table happen in small batches instead of on all data to make it more
manageable.</blockquote>
<p>Imagine we want to count how many bookings were created each month.</p>
<p>As time passes, more and more months and more and more bookings appear in our history, making the size of
this problem ever increasing. But then again, once a month has finished, we shouldn't need to go back
and revisit history: what's done is done, and only the ongoing month is relevant, right?</p>
<p><a href="https://docs.getdbt.com/docs/build/incremental-models" target="_blank"
rel="noopener noreferrer">dbt offers a materialization strategy named <code>incremental</code></a>,
which allows you to only work on a subset of data. This means that every time you run
<code>dbt run</code> , your model only works on a certain part of the data, and not all of it. If the
nature of your data and your needs allows isolating each run to a small part of all upstream data, this
strategy can help wildly improve the performance.
</p>
<p>Explaining the inner details of <code>incremental</code> goes beyond the scope of this page. You can
check the official docs from <code>dbt</code> (<a
href="https://docs.getdbt.com/docs/build/incremental-models" target="_blank"
rel="noopener noreferrer">here</a>), ask the team for support or check some of the incremental
models that we already have in our project and use them as references.</p>
<p>Note that using <code>incremental</code> strategies makes life way harder than simple <code>view</code>
or <code>table</code> ones, so only pick this up if it's truly necessary. Don't make models incremental
without trying other optimizations first, or simply because you realise that you <em>could</em> use it
in a specific model.</p>
<h3>6. End of the line: general optimization</h3>
<p>The final tip is not really a tip. The above five things are the easy-peasy, low hanging fruit stuff that
you can try. This doesn't mean that there isn't more than you can do, just that I don't know of more
simple stuff that you can try without deep knowledge of how Postgres works beneath and a willingness to
get your hands <em>real</em> dirty.</p>
<p>If you've reached this point and your model is still performing poorly, you either need to put your Data
Engineer hat on and really deepen your knowledge… or call Pablo.</p>
<h3 id="bonus">Bonus: how to make sure you didn't screw up and change the output of the model</h3>
<p>The topic we are discussing in this guide is making refactors purely for the sake of performance, without
changing the output of the given model. We simply want to make the model faster, not change what data it
generates.</p>
<p>That being the case, and considering the complexity of the strategies we've presented here, being afraid
that you messed up and accidentally changed the output of the model is a very reasonable fear to have.
That's a kind of mistake that we definitely want to avoid.</p>
<p>Doing this manually can be a PITA and very time consuming, which doesn't help at all.</p>
<p>To make your life easier, I'm going to show you a new little trick.</p>
<h4>Hashing tables and comparing them</h4>
<p>I'll post a snippet of code here that you can run to compare if any pair of tables has <em>exactly</em>
the same contents. Emphasis on exactly. Changing the slightest bit of content will be detected.</p>
<pre><code>SELECT md5(array_agg(md5((t1.*)::varchar))::varchar)
FROM (
SELECT *
FROM my_first_table
ORDER BY &lt;whatever field is unique&gt;
) AS t1
SELECT md5(array_agg(md5((t2.*)::varchar))::varchar)
FROM (
SELECT *
FROM my_second_table
ORDER BY &lt;whatever field is unique&gt;
) AS t2</code></pre>
<p>How this works is: you execute the two queries, which will return a single value each. Some hexadecimal
gibberish.</p>
<p>If the output of the two queries is identical, it means their contents are identical. If they are
different, it means there's something different across both.</p>
<p>If you don't understand how this works, and you don't care, that's fine. Just use it.</p>
<p>If not knowing does bother, you should go down the rabbit holes of hash functions and deterministic
serialization.</p>
<h4>Including this in your refactoring workflow</h4>
<p>Right, now you know how to make sure that two tables are identical.</p>
<p>This is dramatically useful for your optimization workflow. You can know simply:</p>
<ul>
<li>Keep the original model</li>
<li>Create a copy of it, which is the one you will be working on (the working copy)</li>
<li>Prepare the magic query to check their contents are identical</li>
<li>From this point on, you can enter in this loop for as long as you want/need:
<ul>
<li>Run the magic query to ensure you start from same-output-state</li>
<li>Modify the working copy model to attempt whatever optimization thingie you wanna try</li>
<li>Once you are done, run the magic query again.</li>
<li>If the output is not the same anymore, you screwed up. Start again and avoid whatever
mistake you made.</li>
<li>If the output is still the same, you didn't cause a change in the model output. Either keep
on optimizing or call it day.</li>
</ul>
</li>
<li>Finally, just copy over the working copy model code into the old one and remove the working copy.
</li>
</ul>
<p>I hope that helps. I also recommend doing the loop as frequently as possible. The less things you change
between executions of the magic query, the easier is to realize what caused errors if they appear.</p>
<hr>
<p><a href="../index.html">back to home</a></p>
</section>
</main>
</body>
</html>

188
public/writings/fixing-a-degraded-zfs-mirror.html
View file

@ -1,188 +0,0 @@
<!DOCTYPE HTML>
<html>
<head>
<title>Pablo here</title>
<meta charset="utf-8">
<meta viewport="width=device-width, initial-scale=1">
<link rel="stylesheet" href="../styles.css">
</head>
<body>
<main>
<h1>
Hi, Pablo here
</h1>
<p><a href="../index.html">back to home</a></p>
<section>
<h2>Fixing a Degraded ZFS Mirror: Reseat, Resilver, and Scrub</h2>
<p><em>Part 3 of 3 in my "First ZFS Degradation" series. See also <a href="why-i-put-my-vms-on-a-zfs-mirror.html">Part 1: The Setup</a> and <a href="a-degraded-pool-with-a-healthy-disk.html">Part 2: Diagnosing the Problem</a>.</em></p>
<h3>The Game Plan</h3>
<p>By now I was pretty confident about what was wrong: not a dying drive, but a flaky SATA connection. The fix should be straightforward. Just take the drive offline, shut down, reseat the cables, bring it back up, and let ZFS heal itself.</p>
<p>But I wanted to do this methodically. ZFS is forgiving, but I didn't want to make things worse by rushing.</p>
<p>Here was my plan:</p>
<ol>
<li>Take the faulty drive offline in ZFS (tell ZFS "stop trying to use this drive")</li>
<li>Power down the server</li>
<li>Open the case, inspect and reseat cables</li>
<li>Boot up, verify the drive is detected</li>
<li>Bring the drive back online in ZFS</li>
<li>Let the resilver complete</li>
<li>Run a scrub to verify data integrity</li>
<li>Check SMART one more time</li>
</ol>
<p>Let's walk through each step.</p>
<h3>Step 1: Taking the Drive Offline</h3>
<p>Before touching hardware, I wanted ZFS to stop trying to use the problematic drive.</p>
<p>First, I set up some variables to avoid typos with that long disk ID:</p>
<pre><code>DISKID="ata-ST4000NT001-3M2101_WX11TN0Z"
DISKPATH="/dev/disk/by-id/$DISKID"</code></pre>
<p>Then I took it offline:</p>
<pre><code>zpool offline proxmox-tank-1 "$DISKID"</code></pre>
<p>Checking the status afterward:</p>
<pre><code>zpool status -v proxmox-tank-1</code></pre>
<pre><code> NAME STATE READ WRITE CKSUM
proxmox-tank-1 DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
ata-ST4000NT001-3M2101_WX11TN0Z OFFLINE 108 639 129
ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0</code></pre>
<p>The state changed from FAULTED to OFFLINE. ZFS knows I intentionally took it offline rather than it failing on its own. The error counts are still there as a historical record, but ZFS isn't actively trying to use the drive anymore.</p>
<p>Time to shut down and get my hands dirty.</p>
<h3>Step 2: Opening the Case</h3>
<p>I powered down the server and opened up the Fractal Node 804. This case has a lovely design with drive bays accessible from the side, which I love. No reaching out into weird corners in the case, just unscrew a couple screws, slide the drive bay out and there they are, handy and reachable.</p>
<p>I located AGAPITO1 (I had handwritten labels on the drives, lesson learned after many sessions of playing "which drive is which") and inspected the connections.</p>
<p>Here's the honest truth: everything looked fine. The SATA data cable was plugged in. The power connector was plugged in. Nothing was obviously loose or damaged. There was a bit of tension in the cable as it moved from one area of the case (where the motherboard is) to the drives area, but I really didn't think that was affecting the connection to either the drive or the motherboard itself.</p>
<p>But "looks fine" doesn't mean "is fine". So I did a full reseat:</p>
<ul>
<li>Unplugged and firmly replugged the SATA data cable at both ends (drive and motherboard).</li>
<li>Unplugged and firmly replugged the power connector.</li>
<li>While I was in there, checked the connections on the other disk of the mirror as well.</li>
</ul>
<p>I made sure each connector clicked in solidly. Then I closed up the case and hit the power button.</p>
<h3>Step 3: Verifying Detection</h3>
<p>The server booted up. Would Linux see the drive?</p>
<pre><code>ls -l /dev/disk/by-id/ | grep WX11TN0Z</code></pre>
<pre><code>lrwxrwxrwx 1 root root 9 Jan 2 23:15 ata-ST4000NT001-3M2101_WX11TN0Z -> ../../sdb</code></pre>
<p>The drive was there, mapped to <code>/dev/sdb</code>.</p>
<p>I opened a second terminal and started watching the kernel log in real time:</p>
<pre><code>dmesg -Tw</code></pre>
<p>This would show me immediately if the connection started acting flaky again. For now, it was quiet, showing just normal boot messages, the drive being detected successfully, etc. Nothing alarming.</p>
<h3>Step 4: Bringing It Back Online</h3>
<p>Moment of truth. I told ZFS to start using the drive again:</p>
<pre><code>zpool online proxmox-tank-1 "$DISKID"</code></pre>
<p>Immediately checked the status:</p>
<pre><code>zpool status -v proxmox-tank-1</code></pre>
<pre><code> pool: proxmox-tank-1
state: DEGRADED
status: One or more devices is currently being resilvered.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Jan 2 23:17:35 2026
0B resilvered, 0.00% done, no estimated completion time
NAME STATE READ WRITE CKSUM
proxmox-tank-1 DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
ata-ST4000NT001-3M2101_WX11TN0Z DEGRADED 0 0 0 too many errors
ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0</code></pre>
<p>Two things to notice: the drive's error counters are now at zero (we're starting fresh), and ZFS immediately started resilvering. It shows "too many errors" as the reason for the degraded state, which is historical, it remembers why the drive was marked bad before.</p>
<p>I kept watching both the status and the kernel log. No errors, no link resets.</p>
<h3>Step 5: The Resilver</h3>
<p>Resilvering is ZFS's term for rebuilding redundancy. Copying data from the healthy drive to the one that fell behind. In my case, the drive had been desynchronized for who knows how long (the pool had drifted 524GB out of sync before I noticed), so there was a lot to copy.</p>
<p>I shut down my VMs to reduce I/O contention and let the resilver have the disk bandwidth. Progress:</p>
<pre><code>scan: resilver in progress since Fri Jan 2 23:17:35 2026
495G / 618G scanned, 320G / 618G issued at 100M/s
320G resilvered, 51.78% done, 00:50:12 to go</code></pre>
<p>The kernel log stayed quiet the whole time. Everything was indicating the cable reseat had worked.</p>
<p>I went to bed and let it run overnight. The next morning:</p>
<pre><code>scan: resilvered 495G in 01:07:58 with 0 errors on Sat Jan 3 00:25:33 2026</code></pre>
<p>495 gigabytes resilvered in about an hour, zero errors. But the pool still showed DEGRADED with a warning about "unrecoverable error." I was very confused about this, but I solved that with some research. Apparently, ZFS is cautious and wants human acknowledgement before declaring everything healthy again.</p>
<pre><code>zpool clear proxmox-tank-1 ata-ST4000NT001-3M2101_WX11TN0Z</code></pre>
<p>This command clears the error flags. Immediately:</p>
<pre><code> pool: proxmox-tank-1
state: ONLINE
scan: resilvered 495G in 01:07:58 with 0 errors on Sat Jan 3 00:25:33 2026
NAME STATE READ WRITE CKSUM
proxmox-tank-1 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-ST4000NT001-3M2101_WX11TN0Z ONLINE 0 0 0
ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0</code></pre>
<p>Damn, seeing this felt nice.</p>
<h3>Step 6: The Scrub</h3>
<p>A resilver copies data to bring the drives back in sync, but it doesn't verify that all the existing data is still good. For that, you run a scrub. ZFS reads every block on the pool, verifies checksums, and repairs anything that doesn't match.</p>
<pre><code>zpool scrub proxmox-tank-1</code></pre>
<p>I let this run while I brought my VMs back up (scrubs can run in the background without blocking normal operations, though performance takes a hit). A few hours later:</p>
<pre><code>scan: scrub repaired 13.0M in 02:14:22 with 0 errors on Sat Jan 3 11:03:54 2026
NAME STATE READ WRITE CKSUM
proxmox-tank-1 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-ST4000NT001-3M2101_WX11TN0Z ONLINE 0 0 992
ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0</code></pre>
<p>Interesting. The scrub repaired 13MB of data and found 992 checksum mismatches on AGAPITO1. From what I read, checksum errors are typically a sign of the disk being in terrible shape and needing a replacement ASAP. That sounds scary, but I took the risk and assumed those were blocks that had been written incorrectly (or not at all) during the period when the connection was flaky, and not an issue with the disk itself. ZFS detected the bad checksums and healed them using the good copies from AGAPITO2.</p>
<p>I cleared the errors again and the pool was clean:</p>
<pre><code>zpool clear proxmox-tank-1 ata-ST4000NT001-3M2101_WX11TN0Z</code></pre>
<h3>Step 7: Final Validation with SMART</h3>
<p>One more check. I wanted to see if SMART had anything new to say about the drive after all that activity:</p>
<pre><code>smartctl -x /dev/sdb | egrep -i 'overall|Reallocated|Pending|CRC|Hardware Resets'</code></pre>
<pre><code>SMART overall-health self-assessment test result: PASSED
5 Reallocated_Sector_Ct PO--CK 100 100 010 - 0
197 Current_Pending_Sector -O--C- 100 100 000 - 0
199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
0x06 0x008 4 41 --- Number of Hardware Resets</code></pre>
<p>Still passing. The hardware reset count went from 39 to 41 — just the reboots I did during this process.</p>
<p>For completeness, I ran the long self-test. The short test only takes a minute and does basic checks, the long test actually reads every sector on the disk, which for a 4TB drive takes... a while.</p>
<pre><code>smartctl -t long /dev/sdb</code></pre>
<p>The estimated time was about 6 hours. In practice, it took closer to 12. Running VMs in parallel probably didn't help.</p>
<p>But eventually:</p>
<pre><code>SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 1563 -
# 2 Short offline Completed without error 00% 1551 -
# 3 Short offline Completed without error 00% 1462 -</code></pre>
<p>The extended test passed. Every sector on the disk is readable. The drive is genuinely healthy — it was just the connection that was bad.</p>
<h3>Lessons Learned</h3>
<ul>
<li><strong>ZFS did exactly what it's supposed to do:</strong> Despite 524+ gigabytes of desync and nearly a thousand checksum errors, I lost zero data and was back on action while keeping my VMs running. The healthy drive kept serving everything while the flaky drive was acting up, and once the connection was fixed, ZFS healed itself automatically. Also, I was operating for an unknown amount of time with only one drive. In this case it seems it was due to stupid me messing up cable management, but I'm very happy knowing if the disk had been genuinely faulty, services would have continued just fine.</li>
<li><strong>Physical connections matter:</strong> It's easy to not pay that much attention when building a new box. Well, it bites back.</li>
<li><strong>Monitor your pools.</strong> I only found this issue by accident, clicking around in the Proxmox UI. The pool had been degraded for who knows how long before I noticed. I'm already working in setting up a monitor to my Uptime Kuma instance so that next time the pool status stops being ONLINE I get notified immediately.</li>
</ul>
<p>I'm happy I was able to test out recoverying from a faulty disk with such a tiny issue. I learned a lot fixing it, and now I'm even more happy than before having decided to go for this ZFS pool setup.</p>
<h3>Quick Reference: The Commands</h3>
<p>For future me (and anyone else who ends up here with a degraded pool):</p>
<pre><code># Check pool status
zpool status -v <pool>
# Watch kernel logs in real time
dmesg -Tw
# Check SMART health
smartctl -H /dev/sdX
smartctl -x /dev/sdX
# Take a drive offline before physical work
zpool offline <pool> <device-id>
# Bring a drive back online
zpool online <pool> <device-id>
# Clear error flags after recovery
zpool clear <pool> <device-id>
# Run a scrub to verify all data
zpool scrub <pool>
# Run SMART self-tests
smartctl -t short /dev/sdX # Quick test (~1 min)
smartctl -t long /dev/sdX # Full surface scan (hours)
smartctl -l selftest /dev/sdX # Check test results</code></pre>
<p><em>Thanks for reading! This was <a href="fixing-a-degraded-zfs-mirror.html">Part 3: The Fix</a>. You might also enjoy <a href="why-i-put-my-vms-on-a-zfs-mirror.html">Part 1: The Setup</a> and <a href="a-degraded-pool-with-a-healthy-disk.html">Part 2: Diagnosing the Problem</a>.</em></p>
<p><a href="../index.html">back to home</a></p>
</section>
</main>
</body>
</html>

106
public/writings/how-i-write-some-articles-im-lazy-about.html
View file

@ -1,106 +0,0 @@
<!DOCTYPE HTML>
<html>
<head>
<title>Pablo here</title>
<meta charset="utf-8">
<meta viewport="width=device-width, initial-scale=1">
<link rel="stylesheet" href="../styles.css">
</head>
<body>
<main>
<h1>
Hi, Pablo here
</h1>
<p><a href="../index.html">back to home</a></p>
<hr>
<section>
<h2>How I write some articles I have a hard time getting started with</h2>
<p>
I have a lot of shower thoughts. Way more than I can handle. Many times I feel like some of them
would make for a nice piece on my webpage, or I should make a personal note out of. But often times
they won't catch me in the right place and time to sit down, focus, and type them out. You have to
figure out the key points, lay out a plan, think about what you want to say and how, type it all
out, work on the text... It burns some calories. And sometimes it's enough friction to kill the idea
entirely.
</p>
<p>
Recently I found a set of AI tools that allow me to capture some of these ideas and lower the
friction to get them on ink. My goal is not to have the AI write things for me, but just to quickly
drop a pretty decent first draft with very little effort, on the spot. With this, I've happily
rescued many good ideas from getting by falling through the cracks before I even hit my desk (or
getting ignored because I don't have the energy to go through it all).
</p>
<h3>Step one: recording myself</h3>
<p>
I'm self-hosting an open source audio transcription app called
<a href="https://github.com/rishikanthc/Scriberr" target="_blank" rel="noopener noreferrer">scriberr</a>.
It's just a fancy little webpage where you can drop audio and get a transcript. The neat bit is that
it also allows you to simply record on the spot. I used to host
<a href="https://github.com/pluja/whishper" target="_blank" rel="noopener noreferrer">Whishper</a>,
but you had to first record then upload a file. I really liked it, but sadly I found that was
enough friction when rushing on the phone to not use it. I can access scriberr from my laptop, my
desktop, my phone, from anywhere basically.
</p>
<p>
So anytime I feel like it, I can pull out my phone and start ranting about whatever topic is on my
mind. The recording gets sent to my server back home, and scriberr transcribes the whole thing into
text automatically. Transcription itself takes a bit because I'm not using a GPU, but I'm not in a
rush usually since the important thing is just to get the ideas out of my brain and into text easily.
</p>
<p>
The transcript that comes out is usually quite decent in terms of accuracy. I'd say around 95% of
what I say gets picked up perfectly. The whisper models also do pretty decent in different
languages, so I can record in whatever I feel like at each moment.
</p>
<p>
What's not great, in case you've never used transcription tools, is that the output is just a long
stream of words. The model tries its best at punctuation, but it's rather crappy. And obviously, no
paragraphs. Plus, with unprepared, spontaneous ranting, structure tends not to be top notch either.
Definitely not article-grade text out of the box.
</p>
<h3>Letting an LLM clean it up</h3>
<p>
But hey, those are not issues now that we have LLMs! I have a little script that fetches the
transcript results from my server to my laptop. Once I have my raw transcript locally, I just pass
it on to Claude, with a little prompt saying something like: "This is a transcript of me talking
about this and that. Process it for me, I want the output to be like XYZ."
</p>
<p>
Sometimes I really just ask to have the transcript nicely formatted into paragraphs and proper
sentences, with the actual sentences being totally respected. Other times I already ask for some
restructuring of the ideas, so it's not just cleaning up the writing but actually shaping it into
something that resembles an article. It depends on how clear I was during my rant or what I'm really
planning to get out of it.
</p>
<p>
After this bit, I'll either abandon the idea altogether because it wasn't as interesting as I felt
initially, or I'll really work properly in the text when I can and come up with something I'm happy
with. It will still take some desk time to get to a final result, but then the article is really my
writing and not a mix of my own slop with a layer of LLM slopification on top of it. I can't help
but think people who just copy paste LLM output and put their signature under it have little respect
for themselves and their reputation, and little love for the act of thinking and writing.
</p>
<h3>A great starter</h3>
<p>
And there you go. In just a few minutes, without having to focus deeply in front of a blank file, I
end up with a first version of my thoughts that I can already start polishing and thinking from. Not
all transcripts make it into clean texts in the end, but at least I ensure some of the ones which
would otherwise get lost do survive. The stuff that comes out of the LLM is not always great, to be
honest. Sometimes I'll change a lot of things from what I said in the audio. Sometimes I'll end up
adding a lot of stuff that I wasn't covering in my original rant. But the first draft helps me get
my thoughts out there and gives me something to begin with.
</p>
<p>
And that's often all I need.
</p>
<hr>
<p><a href="../index.html">back to home</a></p>
</section>
</main>
</body>
</html>

202
public/writings/my-fitness-journey.html
View file

@ -1,202 +0,0 @@
<!DOCTYPE HTML>
<html>
<head>
<title>Pablo here</title>
<meta charset="utf-8">
<meta viewport="width=device-width, initial-scale=1">
<link rel="stylesheet" href="../styles.css">
</head>
<body>
<main>
<h1>
Hi, Pablo here
</h1>
<p><a href="../index.html">back to home</a></p>
<hr>
<section>
<h2>My fitness journey</h2>
<p>Nowadays I find myself in the best physical shape I've ever been at. I'm stronger than I've ever been.
Pretty decent cardio-wise, not at my peak but close. I'm overall very active, and I can jump into any
physical activity without having second thoughts. I sleep like a baby, perform like a machine, eat like a
pig and it's all good. I'm at a very good weight. And there are so many other things in my life that work
nicely because my body is working nicely.</p>
<p>I'm very happy about this because, for a long time, I really didn't think I would find myself in this
situation. Historically I was not an active person at all. It took many years of back and forth, making
mistakes and trying things, to get my groove in and settle for the habits I have now. And those habits
are great because it's literally costing me no effort to maintain myself in this state.</p>
<p>So I thought I could look back a little, reflect on how I got here. Perhaps you can learn from some of
the mistakes I made and some of the things that worked for me.</p>
<h3>Growing up</h3>
<p>When I was a kid, I was a normal kid. I liked to play and have fun, but I was pretty terrible at sports.
Not a cripple, but surely a rather clumsy kid. I wasn't really talented for most sports, usually falling
in the bottom 25% of the distribution in terms of overall skill and performance.</p>
<p>I guess part of that was just genetics and natural talent, and part of it was simply that my parents were
not the most physically active people. They never engaged in any sports, never did any physical exercise
or went to the gym. So I had no input from them in terms of physical activity or motivation around that
area. I would just play because I was a kid, not because of my parents pushing me to go for it. As I
grew up and playing slowly became less of a thing due to age, sports also slowly faded from my life (and
I was quite happy about it: it's not fun to be the loser systematically).</p>
<p>The only exception in this path is that I did do a couple of years of martial arts when I was around 11
or 12. That was nice, it was a lot of fun and it helped me boost my confidence. And surprisingly, I got
kind of decent at it. Being used to the bottom 25%, just being average at something felt like a massive
success. But then I went into my teenage years and my head was quickly filled with other stuff. I
eventually dropped out of it.</p>
<h3>The dark ages</h3>
<p>During my teenage years, I didn't do shit. I played no sports nor trained my body in any way. I don't
think I ever tried to do anything until I was 18 or 19, already in university. I signed up for the gym a
couple of times, checked online for routines, downloaded them, tried to follow them. Would usually last
two or three months and then drop out. It was boring, not motivating, and I also had no clue what I was
doing. When I look back at how I was training at those times, I can see so many mistakes. I would try to
go too hard, have overly ambitious routines that were completely not sustainable. Nutrition wasn't in
place, technique wasn't in place, exercise choices were probably extremely poor.</p>
<p>I also tried jogging for a bit, but similar experience. Would do it for a while, then eventually lose
consistency. On and off for years.</p>
<p>When I was around 20 or 21, I was living in Germany and really short on money, so signing up for a gym
was not an option. But near where I was living there was a park with some bars, and some days there was
this German fella, I think his name was Leo. He was insane good: super ripped, in extremely good
condition, did all sorts of complicated exercises like it was nothing. The kind of guy you see training
and go "oh wow, that's amazing." He would give me some tips and I really enjoyed training around there
with him. Still, I wasn't extremely consistent. But it was a good intro to doing calisthenics in a bar
park, since the closest I had done was bodyweight training when I did martial arts in my younger years.
</p>
<p>And in that park I fucked up my ribcage doing dips on the parallel bars. One day I went too hard, and I
would guess the technique must have been absolutely terrible as well. That day, after finishing a set, I
just felt this little stab in my chest, which I didn't give much importance to on the spot. Then the next
morning I had super intense pain in my chest every time I tried to move or breathe. I'm pretty confident
I searched online whether I was having a heart attack. It slowly faded but not completely, and it stuck
around for at least a couple of years. That was my first big fuck-up with training and injuring myself.
After that, I learned not to exceed myself with risky exercise like dips, to respect my natural ranges of
motion, and to just take it easy for the most part.</p>
<p>That chest pain got me away from exercise for a long time because any chest exercise would hurt quite a
bit. Push-ups were a no-no. And how's a man supposed to train if he can't do push-ups, right?</p>
<h3>Some progress, then rock bottom</h3>
<p>Around 23 or 24, I went a bit harder with running. Started jogging more regularly, signed up for a 10k,
did a decent job, ran a couple more races, and eventually finished a half marathon in under two hours. A
year or two where I was jogging regularly. That was nice. I eventually lost my habit, but it was a good
experience and there I learned how to do resistance training decently. Also proved myself I was capable
of training for a feat like a half marathon, which was nice since I would have never identified myself as
a runner before.</p>
<p>Then, by the time Covid hit, I was not active at all. Absolutely zero gym, calisthenics, no jogging. And
then we were thrown in our cages with the curfews. Being locked down all day was absolutely terrible. I
wouldn't move around. I was working quite hard at my desk, long days of sitting. My back was hurting like
I was a grandpa. I was smoking a pack of cigarettes a day. Probably drinking more than I should. For a
few months it just spiralled down into some mornings where I would wake up and go "god, I feel like shit
every single day."</p>
<p>Actually, let me backtrace a little. A few months before Covid hit, I had a traffic accident and broke my
leg. Had to get surgery on my knee. I was bedridden for a couple of months, then had to slowly learn how
to walk again. It's hard to explain how bad it is to be bedridden for that long.</p>
<p>So Covid hit on top of that, and I just went worse and worse. Maybe one year, one year and a half into it,
I was in absolute shit shape. At that point, it started to click in my brain that this was not normal, not
good, and not something I wanted. I guess sometimes you really need to lack something to truly appreciate
it and feel the need for it. Greener grasses, yadayada.</p>
<h3>The turnaround</h3>
<p>First thing, I quit smoking. One morning I woke up with my usual chest congestion, you know, where you
have to go to the bathroom and spit the hell out of yourself for five or ten minutes. All morning I was
just feeling it and I was like "I'm so done with this." I quit cold turkey that day, never went back.
Took no effort. Since then, I always advise smokers who want to quit to consider as a possible strategy
to just smoke themselves out. Smoke every single minute, as much as you want. Heck, smoke even when you
don't want to. Maybe that way you'll just get sudden rejection for it like I did.</p>
<p>Then I started training a little at home with a few dumbbells. Basic stuff: rows, squats, floor presses.
And I took it really easy because I was in such terrible shape and I had learned from my leg surgery
rehab that when you're at the bottom, you really have to take it easy coming up. That was one of the most
enlightening things I figured out. When you're in a really bad shape, don't rush it. You will go up
eventually and things will work out, but don't force yourself to do stuff you're not ready to do. Do
something that feels comfy, even trivial I would say, and start doing it regularly. From that point on,
you can start pumping the numbers.</p>
<p>I was extremely busy with work, working from home, so what I would do is train during my lunch break. 15,
20 minutes. Four exercises, a few rows, a few presses, a few squats, maybe just nine sets, done. Cook my
lunch, continue with life. Very humble training for a busy guy.</p>
<p>Precisely because what I was doing was not ambitious, it was pretty convenient. We're talking about
working out for 15 to 20 minutes in a lightweight way at home. I didn't change clothes. I didn't get in
my car and drive to a gym. I would sometimes cook as I was training, leveraging the rest time between
sets. It was extremely convenient and humble, and because of that, I was actually able to make a habit
out of it. One month, two months, three months. Eventually it just grew into a habit. I stayed with this
attitude for at least a year. That was the first time in my life I was actually consistent with strength
training.</p>
<p>The habit stuck and slowly, I increased the intensity, but with an extremely relaxed attitude. I wasn't
tracking myself heavily. I wasn't pushing hard to pump the numbers. I was just happily trying to feel
well and then every now and then I'd throw another half a kilo here, another half a kilo there. Do 4
sets where I used to do 3. Do 8 pushups instead of 6.</p>
<p>I also started jogging casually, doing 5k's here and there. Same principle: very humble jogs with no
intense pace goals, trivial to include in daily life.</p>
<h3>Calisthenics and the bars</h3>
<p>Eventually I started working for a company with a nice office close to the beach, literally a few meters
from the sand. Right in front of the office there was an absolutely amazing calisthenics bar park, with
all the pull-up bars you could figure out, all the grips and heights, parallel bars, row bars, everything
you can imagine. Training in the sand was really nice.</p>
<p>I made a habit of training whenever I went to the office. Sometimes in the morning, sometimes at the end
of the day. I would train at the bars regularly and some days go for a 5k near the beach. I just kept my
sneakers at the office and grabbed them whenever I felt I had the energy.</p>
<p>At this stage I had clearly recovered all of my shitty physical condition from the curfew, smoking and
drinking. By the way, by the beach stage I had already stopped drinking as well. I've been smoke-free
and alcohol-free for a few years now.</p>
<p>Things kept improving. I was feeling so great that I could actually be a bit more ambitious with my
training. And it just worked. It didn't feel like I was pushing myself too hard and I didn't have weird
pains. Also, because I was in decent condition and not smoking or drinking, I was getting incredibly good
sleep, so recovery was much better. I also lost quite a bit of weight due to it all (though most of the
merit I think goes to quitting alcohol completely).</p>
<p>Eventually, I kind of got addicted to the bars. The habit had completely stuck with me. Since then, I
train at different bar parks depending on where I'm working or where life takes me, but I'm always
finding some bar park and going there 2-3 times a week. I even got a weighted vest, so some training
sessions I'll go quite hard with heavy pull-ups and dips.</p>
<p>And that's where I sit today. I'm stronger than I've ever been. I can crank out 10 pull-ups any day of
the week when I'm fresh, or squat cleanly with the missus on my back. And what I today call an easy day
would have been outright impossible for me to perform in 2020. That's pretty cool.</p>
<p>I'm not doing great with jogging right now because I stopped for a few months, tried to pump the numbers
too quickly when I came back (lesson learned), and hurt my Achilles tendon. Working on recovering that,
and then I'll just be very humble again with my 5k's. But right before that, I could run you a sub-hour
10K on any normal day, even if I had not been prepping intensely for it the weeks before.</p>
<p>The only thing I might be missing is some stretch work for flexibility and mobility. I've been thinking I
should probably try some yoga classes. Not to become a master at it, but to have my joints pushed to the
extreme positions every now and then. I'll probably do that at some point.</p>
<h3>Lessons</h3>
<p>If I look back, here's what I would tell my past self, or anyone who's starting out or struggling to make
exercise stick.</p>
<p>The most important thing is to start ridiculously small. So little volume, so little frequency, that it
almost feels pointless. A couple of days a week, five minutes of something that doesn't even feel like a
challenge. You might think that's not getting you anywhere. You're so wrong. Listen: first, you need to
prove yourself that you're capable of doing something trivial consistently, because doing something
consistently is all that matters. Once you prove you can do that, slowly increase, but very slowly.
Don't be in a rush. Don't try to make every session a challenge. You have a lifetime of exercising ahead
of you. Get to a certain level, stay there for a few weeks until you're bored, and then bump it up a
little bit. So little you have a hard time noticing any difference. And then repeat. Same for cardio.
Never jogged? Just go out and do two kilometres. You should be able to pull that in under 15 minutes. Do
that for a month, a couple of times a week, then do 2.5 kilometres. Take years to get to something
significant. But eventually you will get there, which is the important thing. And you'll make it because
you took it easy, you didn't injure yourself, and you didn't give up because your training plan was too
hard and completely incompatible with living a normal life. It feels counterintuitive, but by being kind
of lazy and doing very little, you eventually grow into doing a lot.</p>
<p>Along the same lines, make it trivial to start exercising in your daily life. Remove every bit of
friction you can. If you can afford to have equipment at home, have it ready and accessible, not buried
in some inconvenient closet. If you just have to reach for it and start, there's no excuse. The moment
it takes a couple of minutes to prep for exercise, you can talk yourself out of it. Same for jogging:
have your sneakers and clothes ready to grab, have one route you always do, don't plan it, don't think
about it, just go. Calisthenics bar parks have worked nicely for me precisely because they need zero
prep. You show up and the bars are there, ready for you to jump to them. And don't shy away from short
workouts either. In 15 minutes at a bar park you can do a lot. You're not going to break any records,
but it's way better than doing nothing. I never do long strength training sessions. I'm pretty confident
80-90% of my workouts sit somewhere between 15 and 30 minutes. You can get quite decent results with
that. And if someday you have the time and energy to punish yourself for an hour, be my guest. But don't
feel you have to.</p>
<p>Finally, get your weight sorted. Being at a good weight is insanely convenient for exercise. Running,
lifting, all of it. Being overweight is a fucking pain in the ass, messes with your health in many ways
and just doesn't let your body work properly. If you're clearly overweight, getting your weight down
before you start training hard will make your life a lot easier. Things like running are terribly tolling
when you're heavy, whereas at a healthy weight it feels so much more pleasant, which helps you stick
with it.</p>
<p>That's it. It took me many years to get here, but I'm rolling with it. I'm having fun with the bars,
I'll be having fun jogging with no pressure very soon, might throw the yoga in. I keep it easy. Whenever
I feel like I have some extra energy and bandwidth, I push myself a bit harder for fun. Not on a
schedule, not with any pressure, just because I feel like doing it. Whenever I seem to have no energy or
time, I just do what I can afford to. But the key is: I always do a bit, no matter what.</p>
<hr>
<p><a href="../index.html">back to home</a></p>
</section>
</main>
</body>
</html>

242
public/writings/replacing-a-failed-disk-in-a-zfs-mirror.html
View file

@ -1,242 +0,0 @@
<!DOCTYPE HTML>
<html>
<head>
<title>Pablo here</title>
<meta charset="utf-8">
<meta viewport="width=device-width, initial-scale=1">
<link rel="stylesheet" href="../styles.css">
</head>
<body>
<main>
<h1>
Hi, Pablo here
</h1>
<p><a href="../index.html">back to home</a></p>
<section>
<h2>Replacing a Failed Disk in a ZFS Mirror</h2>
<p>If you've been following along, you know the story: I set up a <a href="why-i-put-my-vms-on-a-zfs-mirror.html">ZFS mirror for my Proxmox VMs</a>, then one of the drives <a href="a-degraded-pool-with-a-healthy-disk.html">started acting flaky</a>, and I <a href="fixing-a-degraded-zfs-mirror.html">diagnosed and fixed what turned out to be a bad SATA connection</a>.</p>
<p>Well, the connection wasn't the whole story. A few weeks after that fix, the same drive, AGAPITO1, started dropping off again. Same symptoms: link resets, speed downgrades, kernel giving up on the connection. I went through the cable swap dance again, tried different SATA ports on the motherboard, tried different cables. Nothing helped. The SATA PHY on the drive itself was failing.</p>
<p>I contacted PcComponentes (where I bought it), RMA'd the drive, and ran degraded on AGAPITO2 alone for about two weeks. Then the replacement arrived. This article covers the process of physically installing a new drive and getting it into the ZFS mirror, from "box on the desk" to "pool healthy, mirror whole."</p>
<h3>The starting point</h3>
<p>Before doing anything, this is what the pool looked like:</p>
<pre><code> pool: proxmox-tank-1
state: DEGRADED
status: One or more devices have been removed.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using zpool online' or replace the device with
'zpool replace'.
scan: scrub repaired 0B in 06:55:06 with 0 errors on Tue Feb 17 20:40:50 2026
config:
NAME STATE READ WRITE CKSUM
proxmox-tank-1 DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
ata-ST4000NT001-3M2101_WX11TN0Z REMOVED 0 0 0
ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0
errors: No known data errors</code></pre>
<p><code>DEGRADED</code> with one drive <code>REMOVED</code>. The old drive (WX11TN0Z) was physically gone, shipped back to PcComponentes. AGAPITO2 (WX11TN2P) was holding down the fort alone.</p>
<p>This is the beauty and the terror of a degraded mirror: everything works fine. Your VMs keep running, your data is intact, reads and writes happen normally. But you have zero redundancy. If that surviving drive has a bad day, you lose everything. Two weeks of running like this was two weeks of hoping AGAPITO2 stayed healthy.</p>
<h3>Before you touch hardware</h3>
<p>Before doing anything physical, I wanted to capture the current state. When things go wrong during maintenance, you want to be able to compare "before" and "after."</p>
<p>Three things to record while the server is still running:</p>
<p><strong>Pool status</strong>, the <code>zpool status</code> output above. You want to know exactly what ZFS thinks the world looks like right now.</p>
<p><strong>SATA layout</strong>, which drive is on which port:</p>
<pre><code>dmesg -T | grep -E 'ata[0-9]+\.[0-9]+: ATA-|ata[0-9]+: SATA link up'</code></pre>
<p>In my case, AGAPITO2 was on ata4 and ata3 was empty (the old drive's port). This matters because after you install the new drive, you want to confirm it shows up on the expected port.</p>
<p><strong>Surviving drive health</strong>, to make sure the drive you're depending on is actually healthy before you start:</p>
<pre><code>smartctl -H /dev/disk/by-id/ata-ST4000NT001-3M2101_WX11TN2P</code></pre>
<pre><code>SMART overall-health self-assessment test result: PASSED</code></pre>
<p>If this says anything other than <code>PASSED</code>, stop and deal with that first. You don't want to discover your only remaining copy of data is on a failing drive while you're in the middle of hardware work.</p>
<p>Once you've got your reference snapshots, shut down the server gracefully:</p>
<pre><code>shutdown now</code></pre>
<h3>Physical installation</h3>
<p>I won't write a hardware installation tutorial, every case and drive bay is different. But a few practical tips for homelabbers doing this for the first time:</p>
<ul>
<li><strong>Inspect your cables before connecting them.</strong> If the SATA data cable has been sitting disconnected in the case, check the connector pins. Bent pins or dust can cause exactly the kind of intermittent issues that started this whole saga.</li>
<li><strong>Label the new drive.</strong> I labeled mine "TOMMY" with its serial number (WX120LHQ) written on a sticker. Yes, I name my drives. It makes debugging much easier than squinting at serial numbers.</li>
<li><strong>Push connectors until they click.</strong> Both SATA data and power. Then do the wiggle test: grab the connector gently and try to move it. If it shifts at all, it's not fully seated.</li>
</ul>
<p>Seat the drive, connect both cables, close the case, and power on.</p>
<h3>Boot and verify detection</h3>
<p>First thing after boot: did the kernel see the new drive?</p>
<pre><code>dmesg -T | grep -E 'ata[0-9]+\.[0-9]+: ATA-|ata[0-9]+: SATA link up'</code></pre>
<pre><code>[Fri Feb 20 22:57:06 2026] ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[Fri Feb 20 22:57:06 2026] ata3.00: ATA-11: ST4000NT001-3M2101, EN01, max UDMA/133
[Fri Feb 20 22:57:07 2026] ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[Fri Feb 20 22:57:07 2026] ata4.00: ATA-11: ST4000NT001-3M2101, EN01, max UDMA/133</code></pre>
<p>Both drives detected at full 6.0 Gbps: TOMMY on ata3, AGAPITO2 on ata4.</p>
<p>Next, verify it shows up with its expected serial in <code>/dev/disk/by-id/</code>:</p>
<pre><code>ls -l /dev/disk/by-id/ | grep WX120LHQ</code></pre>
<pre><code>ata-ST4000NT001-3M2101_WX120LHQ -> ../../sda</code></pre>
<p>And confirm identity with SMART:</p>
<pre><code>smartctl -i /dev/disk/by-id/ata-ST4000NT001-3M2101_WX120LHQ</code></pre>
<pre><code>Device Model: ST4000NT001-3M2101
Serial Number: WX120LHQ
Firmware Version: EN01
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)</code></pre>
<p>Correct model, serial, firmware, and running at full speed.</p>
<p>One more critical check: look for SATA errors in the kernel log.</p>
<pre><code>dmesg -T | grep -E 'ata[0-9]' | grep -iE 'error|fatal|reset|link down|slow|limiting'</code></pre>
<p>I saw <code>ata1: SATA link down</code> and <code>ata2: SATA link down</code>, which are just unused ports. Nothing on ata3 or ata4. If you see errors on the port your new drive is on, <strong>stop</strong>. A brand new drive throwing SATA errors on a known-good cable is likely dead on arrival.</p>
<h3>Health-check before trusting it</h3>
<p>A drive can be detected and still be dead on arrival. Before resilvering 1.3 terabytes of data onto it, I wanted to know it was actually healthy.</p>
<p><strong>SMART overall health:</strong></p>
<pre><code>smartctl -H /dev/disk/by-id/ata-ST4000NT001-3M2101_WX120LHQ</code></pre>
<pre><code>SMART overall-health self-assessment test result: PASSED</code></pre>
<p><strong>Baseline SMART attributes</strong>, the important ones to check on a new drive:</p>
<pre><code>smartctl -A /dev/disk/by-id/ata-ST4000NT001-3M2101_WX120LHQ | grep -E 'Reallocated|Pending|Offline_Uncorrect|CRC'</code></pre>
<pre><code> 5 Reallocated_Sector_Ct ... - 0
197 Current_Pending_Sector ... - 0
198 Offline_Uncorrectable ... - 0
199 UDMA_CRC_Error_Count ... - 0</code></pre>
<p>All zeros. Reallocated sectors would mean the drive has already had to remap bad spots. Pending sectors are blocks the drive suspects are bad but hasn't confirmed yet. CRC errors indicate data corruption during transfer. On a new or refurbished drive, all of these should be zero.</p>
<p><strong>Short self-test:</strong></p>
<pre><code>smartctl -t short /dev/disk/by-id/ata-ST4000NT001-3M2101_WX120LHQ
# Wait ~2 minutes...
smartctl -l selftest /dev/disk/by-id/ata-ST4000NT001-3M2101_WX120LHQ</code></pre>
<pre><code># 1 Short offline Completed without error 00% 0 -</code></pre>
<p>Passed with 0 power-on hours, a fresh drive. If any of these checks fail, don't proceed. Contact the seller and get another replacement.</p>
<h3>The replacement: <code>zpool replace</code></h3>
<p>This is the moment. One command:</p>
<pre><code>zpool replace proxmox-tank-1 ata-ST4000NT001-3M2101_WX11TN0Z ata-ST4000NT001-3M2101_WX120LHQ</code></pre>
<p>This tells ZFS "the drive identified as WX11TN0Z (currently <code>REMOVED</code>) is being replaced by WX120LHQ." ZFS starts resilvering immediately, copying all data from the surviving drive (AGAPITO2) onto the new one (TOMMY).</p>
<p>Checking status right after:</p>
<pre><code> pool: proxmox-tank-1
state: DEGRADED
scan: resilver in progress since Fri Feb 20 23:10:58 2026
config:
NAME STATE READ WRITE CKSUM
proxmox-tank-1 DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
replacing-0 DEGRADED 0 0 0
ata-ST4000NT001-3M2101_WX11TN0Z REMOVED 0 0 0
ata-ST4000NT001-3M2101_WX120LHQ ONLINE 0 0 7.73K
ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0</code></pre>
<p>Notice the <code>replacing-0</code> vdev. That's a temporary structure ZFS creates during the replacement, showing both the old (<code>REMOVED</code>) and new (<code>ONLINE</code>) drive while the resilver is in progress.</p>
<p>The 7.73K cksum count on the new drive might look alarming, but it's expected during a resilver. Those are blocks that haven't been written yet. ZFS is aware of them and they'll clear up as the resilver progresses.</p>
<p>I monitored progress with:</p>
<pre><code>watch -n 30 "zpool status -v proxmox-tank-1"</code></pre>
<p>I also kept <code>dmesg -Tw</code> running in another terminal, watching for any SATA errors. The kernel log stayed quiet the entire time.</p>
<p>In my case, the VMs had auto-started on boot, so the resilver was competing with production I/O. It completed in about 3.5 hours: 1.34 terabytes resilvered with 0 errors. Not bad for a pair of 4TB IronWolf drives running alongside active workloads.</p>
<h3>Post-resilver verification</h3>
<p>The resilver finished. Time to verify everything is actually good.</p>
<p><strong>Pool status:</strong></p>
<pre><code> pool: proxmox-tank-1
state: ONLINE
scan: resilvered 1.34T in 03:32:55 with 0 errors on Sat Feb 21 02:43:53 2026
config:
NAME STATE READ WRITE CKSUM
proxmox-tank-1 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-ST4000NT001-3M2101_WX120LHQ ONLINE 0 0 7.73K
ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0
errors: No known data errors</code></pre>
<p><code>ONLINE</code>. The <code>replacing-0</code> vdev is gone and the mirror now has the new drive in place. The 7.73K cksum on TOMMY is a residual counter from the resilver, so let's clear it:</p>
<pre><code>zpool clear proxmox-tank-1</code></pre>
<p>Now for the real test. A resilver copies data to rebuild the mirror, but a <strong>scrub</strong> reads every block on the pool, verifies all checksums, and repairs any mismatches. This is the definitive integrity check:</p>
<pre><code>zpool scrub proxmox-tank-1</code></pre>
<p>This ran for about 3.5 hours across 1.34T of data:</p>
<pre><code> scan: scrub repaired 0B in 03:27:50 with 0 errors on Sat Feb 21 11:38:02 2026
NAME STATE READ WRITE CKSUM
proxmox-tank-1 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-ST4000NT001-3M2101_WX120LHQ ONLINE 0 0 0
ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0
errors: No known data errors</code></pre>
<p>Zero bytes repaired, zero errors, both drives at 0/0/0. Clean.</p>
<p>One last thing: a post-I/O SMART check on the new drive. After hours of heavy writes during the resilver and reads during the scrub, any hardware weakness should have surfaced:</p>
<pre><code>smartctl -x /dev/disk/by-id/ata-ST4000NT001-3M2101_WX120LHQ | grep -E 'Reallocated|Pending|Offline_Uncorrect|CRC|Hardware Resets|COMRESET|Interface'</code></pre>
<pre><code>Reallocated_Sector_Ct ... 0
Current_Pending_Sector ... 0
Offline_Uncorrectable ... 0
UDMA_CRC_Error_Count ... 0
Number of Hardware Resets ... 2
Number of Interface CRC Errors ... 0
COMRESET ... 2</code></pre>
<p>All clean. The 2 hardware resets and 2 COMRESETs are just from the server booting, perfectly normal.</p>
<h3>The commands, all in one place</h3>
<p>For future me and anyone else replacing a disk in a ZFS mirror:</p>
<pre><code># --- Before shutdown ---
# Record pool status
zpool status -v &lt;pool&gt;
# Record SATA layout
dmesg -T | grep -E 'ata[0-9]+\.[0-9]+: ATA-|ata[0-9]+: SATA link up'
# Check surviving drive health
smartctl -H /dev/disk/by-id/&lt;surviving-disk-id&gt;
# Shut down
shutdown now
# --- After boot with new drive ---
# Verify detection
dmesg -T | grep -E 'ata[0-9]+\.[0-9]+: ATA-|ata[0-9]+: SATA link up'
ls -l /dev/disk/by-id/ | grep &lt;new-serial&gt;
smartctl -i /dev/disk/by-id/&lt;new-disk-id&gt;
# Check for SATA errors
dmesg -T | grep -E 'ata[0-9]' | grep -iE 'error|fatal|reset|link down'
# Health-check the new drive
smartctl -H /dev/disk/by-id/&lt;new-disk-id&gt;
smartctl -A /dev/disk/by-id/&lt;new-disk-id&gt; | grep -E 'Reallocated|Pending|Offline_Uncorrect|CRC'
smartctl -t short /dev/disk/by-id/&lt;new-disk-id&gt;
smartctl -l selftest /dev/disk/by-id/&lt;new-disk-id&gt;
# --- Replace and resilver ---
# Replace old drive with new
zpool replace &lt;pool&gt; &lt;old-disk-id&gt; &lt;new-disk-id&gt;
# Monitor resilver progress
watch -n 30 "zpool status -v &lt;pool&gt;"
# Watch kernel log for SATA errors during resilver
dmesg -Tw
# --- Post-resilver verification ---
# Check final status
zpool status -v &lt;pool&gt;
# Clear residual cksum counters
zpool clear &lt;pool&gt;
# Run a full scrub
zpool scrub &lt;pool&gt;
# Post-I/O SMART check
smartctl -x /dev/disk/by-id/&lt;new-disk-id&gt; | grep -E 'Reallocated|Pending|Offline_Uncorrect|CRC'</code></pre>
<p>The mirror degradation that started on February 8th is resolved. Two weeks of running on a single drive, an RMA, and one evening of work later, the pool is whole again. Full redundancy restored, zero data lost throughout the entire saga. ZFS did exactly what it was designed to do.</p>
<p><em>This is the fourth and final article in this series. If you're just arriving, start with <a href="why-i-put-my-vms-on-a-zfs-mirror.html">Part 1: Why I Put My VMs on a ZFS Mirror</a>, then <a href="a-degraded-pool-with-a-healthy-disk.html">Part 2: A Degraded Pool with a Healthy Disk</a>, and <a href="fixing-a-degraded-zfs-mirror.html">Part 3: Fixing a Degraded ZFS Mirror</a>.</em></p>
<p><a href="../index.html">back to home</a></p>
</section>
</main>
</body>
</html>

120
public/writings/why-i-put-my-vms-on-a-zfs-mirror.html
View file

@ -1,120 +0,0 @@
<!DOCTYPE HTML>
<html>
<head>
<title>Pablo here</title>
<meta charset="utf-8">
<meta viewport="width=device-width, initial-scale=1">
<link rel="stylesheet" href="../styles.css">
</head>
<body>
<main>
<h1>
Hi, Pablo here
</h1>
<p><a href="../index.html">back to home</a></p>
<section>
<h2>Why I Put My VMs on a ZFS Mirror</h2>
<p><em>Part 1 of 3 in my "First ZFS Degradation" series. Also read <a href="a-degraded-pool-with-a-healthy-disk.html">Part 2: Diagnosing the Problem</a> and <a href="fixing-a-degraded-zfs-mirror.html">Part 3: The Fix</a>.</em></p>
<h3>Why This Series Exists</h3>
<p>A few weeks into running my new homelab server, I stumbled upon something I wasn't expecting to see that early: my ZFS pool was in "DEGRADED" state. One of my two mirrored drives had gone FAULTED.</p>
<p>This was the first machine I had set up with a ZFS mirror, precisely to be able to deal with disk issues smoothly, without losing data and having downtime. Although it felt like a pain in the ass to spot the problem, I was also happy because it gave me a chance to drill the kind of disk maintenance I was hoping to do in this new server.</p>
<p>But here's the thing: when I was in the middle of it, I couldn't find a single resource that walked through the whole experience in detail. Plenty of docs explain what ZFS <em>is</em>. Plenty of forum posts have people asking "help my pool is degraded." But nothing that said "here's what it actually feels like to go through this, step by step, with all the commands and logs and reasoning behind the decisions."</p>
<p>So I wrote it down. I took a lot of notes during the process and crafted a more or less organized story from them. This three-part series is for fellow amateur homelabbers who are curious about ZFS, maybe a little intimidated by it, and want to know what happens when things go sideways. I wish I had found a very detailed log like this when I was researching ZFS initially. Hope it helps you.</p>
<h3>The server and disks</h3>
<p>My homelab server is a modest but capable box I built in late 2025. It has decent consumer hardware, but nothing remarkable. I'll only specify that I have currently three disks on it:</p>
<ul>
<li><strong>OS Drive</strong>: Kingston KC3000 512GB NVMe. Proxmox lives here.</li>
<li><strong>Data Drives</strong>: Two Seagate IronWolf Pro 4TB drives (ST4000NT001). This is where my Proxmox VMs get their disks stored.</li>
</ul>
<p>The two IronWolf drives are where this story takes place. I labeled them AGAPITO1 and AGAPITO2 because... well, every pair of drives deserves a silly name. I have issues remembering serial numbers.</p>
<p>The server runs Proxmox and hosts most of my self-hosted life: personal services, testing VMs, and my Bitcoin infrastructure (which I share over at <a href="https://bitcoininfra.contrapeso.xyz" target="_blank" rel="noopener noreferrer">bitcoininfra.contrapeso.xyz</a>). If this pool goes down, everything goes down.</p>
<h3>Why ZFS?</h3>
<p>I'll be honest: I didn't overthink this decision. ZFS is the default storage recommendation for Proxmox, it has a reputation for being rock-solid, and I'd heard enough horror stories about silent data corruption to want something with checksumming built in.</p>
<p>What I was most interested in was the ability to define RAID setups in software and deal easily with disks going in and out of them. I had never gone beyond the naive "one disk for the OS, one disk for data" setup in previous servers. After having disks failing on me in previous boxes, I decided it was time to gear up and do it proper this time. My main concern initially was just saving time: it's messy when a "simple" host has disk issues, and I hoped mirroring would allow me to invest less time in cleaning up disasters.</p>
<h3>Why a Mirror?</h3>
<p>When I set up the pool, I had two 4TB drives. That gave me a few options:</p>
<ol>
<li><strong>Single disk</strong>: Maximum space (8TB usable), zero redundancy. One bad sector and you're crying.</li>
<li><strong>Mirror</strong>: Half the space (4TB usable from 8TB raw), but everything is written to both drives. One drive can completely die and you lose nothing.</li>
<li><strong>RAIDZ</strong>: Needs at least 3 drives, gives you parity-based redundancy. More space-efficient than mirrors at scale.</li>
</ol>
<p>I went with the mirror for a few reasons.</p>
<p>First, I only had two drives to start with, so RAIDZ wasn't even an option yet.</p>
<p>Second, mirrors are <em>simple</em>. Data goes to both drives. If one dies, the other has everything. No parity calculations, no write penalties, no complexity.</p>
<p>Third (and this is the one that sold me), <strong>mirrors let you expand incrementally</strong>. With ZFS, you can add more mirror pairs (called "vdevs") to your pool later. You can even mix sizes: start with two 4TB drives, add two 8TB drives later, and ZFS will use all of it. RAIDZ doesn't give you that flexibility; once you set your vdev width, you're stuck with it.</p>
<h4>When Would RAIDZ Make More Sense?</h4>
<p>If you're starting with 4+ drives and you want to maximize usable space, RAIDZ starts looking attractive:</p>
<table>
<thead>
<tr>
<th>Configuration</th>
<th>Drives</th>
<th>Usable Space</th>
<th>Fault Tolerance</th>
</tr>
</thead>
<tbody>
<tr>
<td>Mirror</td>
<td>2</td>
<td>50%</td>
<td>1 drive</td>
</tr>
<tr>
<td>RAIDZ1</td>
<td>3</td>
<td>~67%</td>
<td>1 drive</td>
</tr>
<tr>
<td>RAIDZ1</td>
<td>4</td>
<td>75%</td>
<td>1 drive</td>
</tr>
<tr>
<td>RAIDZ2</td>
<td>4</td>
<td>50%</td>
<td>2 drives</td>
</tr>
<tr>
<td>RAIDZ2</td>
<td>6</td>
<td>~67%</td>
<td>2 drives</td>
</tr>
</tbody>
</table>
<p>RAIDZ2 is popular for larger arrays because it can survive <em>two</em> drive failures, which matters more as you add drives (more drives = higher chance of one failing during a resilver).</p>
<p>But for a two-drive homelab that might grow to four drives someday, I felt a mirror was the right call. I can always add another mirror pair later.</p>
<h3>The Pool: proxmox-tank-1</h3>
<p>My ZFS pool is called <code>proxmox-tank-1</code>. Here's what it looks like when everything is healthy:</p>
<pre><code> pool: proxmox-tank-1
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
proxmox-tank-1 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-ST4000NT001-3M2101_WX11TN0Z ONLINE 0 0 0
ata-ST4000NT001-3M2101_WX11TN2P ONLINE 0 0 0</code></pre>
<p>That's it. One pool, one mirror vdev, two drives. The drives are identified by their serial numbers (the <code>WX11TN0Z</code> and <code>WX11TN2P</code> parts), which is important — ZFS uses stable identifiers so it doesn't get confused if Linux decides to shuffle around <code>/dev/sda</code> and <code>/dev/sdb</code>.</p>
<p>All my Proxmox VMs store their virtual disks on this pool. When I create a new VM, I point its storage at <code>proxmox-tank-1</code> and ZFS handles the rest.</p>
<h3>What Could Possibly Go Wrong?</h3>
<p>Everything was humming along nicely. VMs were running fine and I was feeling pretty good about my setup.</p>
<p>Then, a few weeks in, I was poking around the Proxmox web UI and noticed something that caught my eye.</p>
<p>The ZFS pool was DEGRADED. One of my drives — AGAPITO1, serial <code>WX11TN0Z</code> — was FAULTED.</p>
<p>In <a href="a-degraded-pool-with-a-healthy-disk.html">Part 2</a>, I'll walk through how I diagnosed what was actually wrong. Spoiler: the drive itself was fine. The problem was much dumber than that.</p>
<p><em>Continue to <a href="a-degraded-pool-with-a-healthy-disk.html">Part 2: Diagnosing the Problem</a></em></p>
<p><a href="../index.html">back to home</a></p>
</section>
</main>
</body>
</html>