No description
counterweight
0112bb2c69
Investigating a dashboard full of DOWN services that were plainly healthy turned
up three bugs, all introduced by me.
── 1. The disk check has been lying since it was written ───────────────────
df: options -P and --output are mutually exclusive
`df -P ... --output=pcent,target` errors out. `2>/dev/null` swallowed it, the
read loop got nothing, `worst` stayed 0, and the check reported "max 0% on /"
and exited 0 on EVERY host regardless of real usage. vipy is at 58%. It would
never have caught a full disk - a green light wired to nothing, which is worse
than no check at all.
Dropped -P. More importantly, "df returned no filesystems" is now a FAILURE
rather than being read as 0% - the original bug was only invisible because an
empty result was indistinguishable from an empty disk.
Testing the fix immediately surfaced a second problem it had been masking:
/sys/firmware/efi/efivars sits at 87% on a healthy machine, so a working check
would have paged daily. efivarfs and ramfs join tmpfs/devtmpfs/squashfs/overlay
in the exclusions, all of them pseudo-filesystems whose "usage" is not a fact an
operator can act on. It now reports "max 73% on / (3 filesystems)".
── 2. The false DOWNs were a half-finished deploy ──────────────────────────
The previous commit tightened heartbeats from 30h to 7h AND moved the checks
from daily to 6-hourly - but only the registration side was deployed, with
--limit observability. The hosts kept pushing once a day against a 7h window, so
after seven hours every disk, ZFS and backup-store endpoint went red.
Nothing was wrong with the services and nothing was wrong with Gatus: it
correctly reported that pushes were not arriving. Changing a heartbeat window
without deploying the matching cadence is a guaranteed false alarm, and the two
have to ship together.
── 3. The new OnCalendar was invalid ───────────────────────────────────────
"*-*-* 05:30:00,11:30:00,17:30:00,23:30:00" is not valid systemd syntax - a
comma-separated list of FULL TIMES is rejected with "bad unit file setting", and
the timer silently failed to install. Corrected to "*-*-* 05/6:30:00" and
validated with `systemd-analyze calendar` before deploying, which is how this
should have been written in the first place.
Verified: 11 disk checks and the ZFS check all HEALTHY on a manual run; timers
confirmed on the hosts as 00/6:00:00, 00/6:20:00 and 05/6:30:00; 86 UP / 0 DOWN.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|---|---|---|
| ansible | ||
| archive/uptime_kuma | ||
| tofu/nodito | ||
| .gitignore | ||
| 01_infra_setup.md | ||
| 02_vps_core_services_setup.md | ||
| 03_vm_disk_enlargement.md | ||
| README.md | ||
| requirements.txt | ||
Personal infra
My repo documenting my personal infra, along with artifacts, scripts, etc.
How to use
Go through the different numbered markdowns in the repo root to do the different parts.
How to edit secrets
ansible-vault edit ansible/your_file_with_secrets.yml
Assumes that you've set ansible/.vault_pass with chmod 600.
Overview
Services
- Reverse Proxy
- Deployed on Vipy
- Caddy
- Plan install
- File based config
- Crossbackup to Desky via rsync
Uptime Kuma— decommissioned 2026-09-11, seearchive/uptime_kuma/- Deployed on Vipy
- Crossbackup to Desky via rsync
- Vaultwarden
- Deployed on Desky
- Crossbackup to Vipy via rsync
- Gitea
- Deployed on Desky
- Crossbackup to Vipy via rsync
- Immich
- Deployed on Desky
- VPN
- All set up on Vipy
- Bitcoin Knots
- Deployed on Desky
- electrs
- Synapse Server
- Phoenix D + LNBits
- Backups
Infra
- Laptop (Lapy)
- One beefy desktop (Desky)
- One VPS (Vipy)