ansible: add site.yml, and rename the monitoring group off the host's name
site.yml is a TABLE OF CONTENTS, not a second source of truth. It is 25
import_playbook: lines and comments - no `hosts:`, no `roles:`. Which hosts get
what stays on the `hosts:` line inside each playbook, exactly where it already
was; nothing moved. Every role is already wrapped in a thin playbook carrying
its own `hosts:` line, so there is no roles-vs-playbooks split to reconcile:
from here everything is a playbook.
What it buys:
What runs on a host? ansible-playbook site.yml --limit <host> --list-hosts
Who gets thing Y? the `hosts:` line in Y's own playbook
What is a host? ansible-inventory --graph
Note --list-hosts, not --list-tasks: the latter prints every play regardless of
--limit, so it will happily show you the bitcoin play under memos-box.
Nine playbooks are deliberately excluded and the file names every one with a
reason, so it accounts for all of them: the three infra/4xx monitoring plays
(still assert on the removed Uptime Kuma credentials and fail immediately),
910_docker (says `hosts: managed`, but Docker is on 5 of 11 managed hosts and
those 5 are exactly the ones that need it - running it installs Docker on the
Bitcoin node and the hypervisor), two nodito one-shots, the Kuma notification
setup, and two deliberate manual actions.
Writing it surfaced an inventory collision. There is a HOST named `monitoring`
in [vps] AND a group [monitoring], so Ansible warned and resolved `hosts:
monitoring` to the host:
[WARNING]: Found both group and host with same name: monitoring
The group is renamed to [observability]; the host keeps its name. [caddy:children]
and the two ntfy playbooks follow. Behaviour is unchanged - `hosts: monitoring`
already resolved to the host - but the ambiguity is gone and the warning with it.
Verified: inventory graph is warning-free, site.yml passes --syntax-check, and
per-host play counts are identical before and after the rename.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-13 21:24:09 +02:00
|
|
|
---
|
|
|
|
|
# Everything, in the order it has to happen.
|
|
|
|
|
#
|
|
|
|
|
# This file is a TABLE OF CONTENTS, not a second source of truth. It says what
|
|
|
|
|
# runs and in what order. It does NOT say which hosts get what — that stays on
|
|
|
|
|
# the `hosts:` line inside each playbook, exactly where it is today. Nothing
|
|
|
|
|
# moves; this file only makes the set readable in one place.
|
|
|
|
|
#
|
|
|
|
|
# What runs on a host? ansible-playbook site.yml --limit <host> --list-hosts
|
|
|
|
|
# Who gets thing Y? the `hosts:` line in Y's own playbook
|
|
|
|
|
# What is a host? ansible-inventory --graph
|
|
|
|
|
#
|
|
|
|
|
# Run a slice with --limit, or run one playbook directly as before. Nothing here
|
|
|
|
|
# changes how any individual playbook behaves.
|
|
|
|
|
|
|
|
|
|
# ── Baseline: every managed machine ─────────────────────────────────────────
|
|
|
|
|
- import_playbook: infra/01_user_and_access_setup_playbook.yml
|
|
|
|
|
- import_playbook: infra/02_firewall_and_fail2ban_playbook.yml
|
|
|
|
|
- import_playbook: infra/900_install_rsync.yml
|
|
|
|
|
- import_playbook: infra/920_join_headscale_mesh.yml
|
monitoring: retire the Uptime-Kuma-era checks, add ZFS pool capacity
Five things deprecated, each verified against the DEPLOYED script before being
deleted rather than assumed superseded:
infra/410_disk_usage_alerts.yml -> disk-usage check (infra/400)
infra/420_system_healthcheck.yml -> liveness check (infra/400)
infra/430_cpu_temp_alerts.yml -> cpu-temp check (infra/400)
32_zfs play 2 (monitoring half) -> zfs-health check (infra/400)
34_nut play 2 (entirely) -> ups-status check (infra/400)
Nothing is lost by the swap. The old system_healthcheck.sh only computed uptime
and pushed, which is exactly a liveness heartbeat. The old disk monitor was
WEAKER than its replacement: it checked "/" alone at 80%, where the new one
walks every real filesystem at 85%.
Deleting the playbooks was not the hard part. The units they installed live on
the hosts, enabled, and keep firing regardless of what the repo says - two of
them were still pushing to uptime.contrapeso.xyz every 15 minutes across nine
machines. A playbook deleted without a cleanup leaves its output running
forever with nothing left to explain it. So infra/409_remove_legacy_monitoring
stops, disables and removes the units, deletes /opt/{disk-monitoring,
system-healthcheck,nodito-monitoring,zfs-monitoring}, and removes the orphaned
hand-written ups-heartbeat.sh. It ends by grepping for any surviving Kuma
reference and reporting it. Kept permanently and idempotent, so a rebuilt or
restored host cannot quietly bring them back.
A trap avoided: the monthly ZFS scrub lived INSIDE 32_zfs play 2. Deleting the
play wholesale would have silently stopped scrubbing the pool - and an
unscrubbed pool makes the health check meaningless, because it would have
nothing true to report. That play is now scrub-only and check-runs ok=5
changed=0.
ZFS pool capacity added as a sixth condition to the zfs-health check. `zpool
status` reports a 95% full pool as perfectly ONLINE, so capacity has to be read
separately with `zpool list` - and it is the failure you get warning of rather
than the one you discover. Threshold 80%, because ZFS allocation degrades badly
past roughly that and fragmentation is hard to undo. The pool is at 47%.
Verified both directions: passes on the real pool, and a simulated 91% exits 1.
Also fixed the waste recorded in c2de6db: the healthcheck role installed its
dependencies once per CHECK rather than per HOST - 29 apt transactions
estate-wide for a curl already present, and the slowest part of every deploy.
It now deduplicates within a play run, and the redundant standalone
daemon_reload is gone (the systemd task already does one).
site.yml updated, which exposed that services/gatus was never in it. It now runs
before the three registration playbooks, since registering endpoints against a
Gatus that is not yet serving would simply fail.
Verified: no legacy timer remains on any host; 83 endpoints, 83 UP, 0 DOWN.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-14 10:18:57 +02:00
|
|
|
# Idempotent and kept permanently: guarantees a rebuilt or restored host cannot
|
|
|
|
|
# quietly bring the Uptime-Kuma-era monitoring back.
|
|
|
|
|
- import_playbook: infra/409_remove_legacy_monitoring.yml
|
|
|
|
|
|
|
|
|
|
# ── Monitoring ──────────────────────────────────────────────────────────────
|
|
|
|
|
# Gatus first: the three plays below register endpoints with it, and registering
|
|
|
|
|
# against a host that is not serving yet would simply fail.
|
|
|
|
|
- import_playbook: services/gatus/deploy_gatus_playbook.yml
|
|
|
|
|
- import_playbook: infra/400_host_monitoring.yml
|
|
|
|
|
- import_playbook: infra/401_service_monitoring.yml
|
|
|
|
|
- import_playbook: infra/402_public_monitoring.yml
|
ansible: add site.yml, and rename the monitoring group off the host's name
site.yml is a TABLE OF CONTENTS, not a second source of truth. It is 25
import_playbook: lines and comments - no `hosts:`, no `roles:`. Which hosts get
what stays on the `hosts:` line inside each playbook, exactly where it already
was; nothing moved. Every role is already wrapped in a thin playbook carrying
its own `hosts:` line, so there is no roles-vs-playbooks split to reconcile:
from here everything is a playbook.
What it buys:
What runs on a host? ansible-playbook site.yml --limit <host> --list-hosts
Who gets thing Y? the `hosts:` line in Y's own playbook
What is a host? ansible-inventory --graph
Note --list-hosts, not --list-tasks: the latter prints every play regardless of
--limit, so it will happily show you the bitcoin play under memos-box.
Nine playbooks are deliberately excluded and the file names every one with a
reason, so it accounts for all of them: the three infra/4xx monitoring plays
(still assert on the removed Uptime Kuma credentials and fail immediately),
910_docker (says `hosts: managed`, but Docker is on 5 of 11 managed hosts and
those 5 are exactly the ones that need it - running it installs Docker on the
Bitcoin node and the hypervisor), two nodito one-shots, the Kuma notification
setup, and two deliberate manual actions.
Writing it surfaced an inventory collision. There is a HOST named `monitoring`
in [vps] AND a group [monitoring], so Ansible warned and resolved `hosts:
monitoring` to the host:
[WARNING]: Found both group and host with same name: monitoring
The group is renamed to [observability]; the host keeps its name. [caddy:children]
and the two ntfy playbooks follow. Behaviour is unchanged - `hosts: monitoring`
already resolved to the host - but the ambiguity is gone and the warning with it.
Verified: inventory graph is warning-free, site.yml passes --syntax-check, and
per-host play counts are identical before and after the rename.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-13 21:24:09 +02:00
|
|
|
# 910_docker says `hosts: managed`, but only 5 of 11 managed hosts have or need
|
|
|
|
|
# Docker. Left out until it has a [docker] group — see the note in PLAN_7.
|
|
|
|
|
|
|
|
|
|
# ── The hypervisor ──────────────────────────────────────────────────────────
|
|
|
|
|
- import_playbook: infra/nodito/31_proxmox_community_repos_playbook.yml
|
|
|
|
|
- import_playbook: infra/nodito/32_zfs_pool_setup_playbook.yml
|
|
|
|
|
- import_playbook: infra/nodito/34_nut_ups_setup_playbook.yml
|
|
|
|
|
|
|
|
|
|
# ── Reverse proxy, before anything that registers a vhost ───────────────────
|
|
|
|
|
- import_playbook: services/caddy_playbook.yml
|
|
|
|
|
|
|
|
|
|
# ── Services ────────────────────────────────────────────────────────────────
|
|
|
|
|
- import_playbook: services/bitcoin-knots/deploy_bitcoin_knots_playbook.yml
|
|
|
|
|
- import_playbook: services/fulcrum/deploy_fulcrum_playbook.yml
|
|
|
|
|
- import_playbook: services/datum-gateway/deploy_datum_gateway_playbook.yml
|
|
|
|
|
- import_playbook: services/mempool/deploy_mempool_playbook.yml
|
|
|
|
|
- import_playbook: services/memos/deploy_memos_playbook.yml
|
|
|
|
|
- import_playbook: services/forgejo-runner/deploy_forgejo_runner_playbook.yml
|
|
|
|
|
- import_playbook: services/phoenixd/deploy_phoenixd_playbook.yml
|
|
|
|
|
- import_playbook: services/headscale/deploy_headscale_playbook.yml
|
|
|
|
|
- import_playbook: services/vaultwarden/deploy_vaultwarden_playbook.yml
|
|
|
|
|
- import_playbook: services/forgejo/deploy_forgejo_playbook.yml
|
|
|
|
|
- import_playbook: services/lnbits/deploy_lnbits_playbook.yml
|
|
|
|
|
- import_playbook: services/ntfy/deploy_ntfy_playbook.yml
|
|
|
|
|
- import_playbook: services/ntfy-emergency-app/deploy_ntfy_emergency_app_playbook.yml
|
|
|
|
|
- import_playbook: services/personal-blog/deploy_personal_blog_playbook.yml
|
|
|
|
|
|
|
|
|
|
# ── Backups: each source dumps itself, the box pulls ────────────────────────
|
|
|
|
|
- import_playbook: services/headscale/setup_backup_headscale.yml
|
|
|
|
|
- import_playbook: services/vaultwarden/setup_backup_vaultwarden.yml
|
|
|
|
|
- import_playbook: services/forgejo/setup_backup_forgejo.yml
|
|
|
|
|
- import_playbook: services/lnbits/setup_backup_lnbits.yml
|
|
|
|
|
- import_playbook: services/memos/setup_backup_memos.yml
|
|
|
|
|
- import_playbook: playbooks/backups.yml
|
|
|
|
|
|
|
|
|
|
# Deliberately not here. Every playbook in the repo is either imported above or
|
|
|
|
|
# listed below, so this file accounts for all of them:
|
|
|
|
|
#
|
|
|
|
|
# infra/910_docker_playbook.yml says `hosts: managed`, but Docker is on 5
|
|
|
|
|
# of 11 managed hosts and those 5 are exactly
|
|
|
|
|
# the ones that need it. Running it would
|
|
|
|
|
# install Docker on the Bitcoin node and the
|
|
|
|
|
# hypervisor. Needs a [docker] group first.
|
|
|
|
|
#
|
|
|
|
|
# infra/nodito/30_proxmox_bootstrap one-shot: bare-metal bootstrap, run once
|
|
|
|
|
# infra/nodito/33_..._cloud_template one-shot: builds the VM template
|
|
|
|
|
#
|
|
|
|
|
# services/ntfy/setup_ntfy_uptime_ creates a notification channel INSIDE
|
|
|
|
|
# kuma_notification.yml Uptime Kuma. Kuma-specific tooling, not
|
monitoring: retire the Uptime-Kuma-era checks, add ZFS pool capacity
Five things deprecated, each verified against the DEPLOYED script before being
deleted rather than assumed superseded:
infra/410_disk_usage_alerts.yml -> disk-usage check (infra/400)
infra/420_system_healthcheck.yml -> liveness check (infra/400)
infra/430_cpu_temp_alerts.yml -> cpu-temp check (infra/400)
32_zfs play 2 (monitoring half) -> zfs-health check (infra/400)
34_nut play 2 (entirely) -> ups-status check (infra/400)
Nothing is lost by the swap. The old system_healthcheck.sh only computed uptime
and pushed, which is exactly a liveness heartbeat. The old disk monitor was
WEAKER than its replacement: it checked "/" alone at 80%, where the new one
walks every real filesystem at 85%.
Deleting the playbooks was not the hard part. The units they installed live on
the hosts, enabled, and keep firing regardless of what the repo says - two of
them were still pushing to uptime.contrapeso.xyz every 15 minutes across nine
machines. A playbook deleted without a cleanup leaves its output running
forever with nothing left to explain it. So infra/409_remove_legacy_monitoring
stops, disables and removes the units, deletes /opt/{disk-monitoring,
system-healthcheck,nodito-monitoring,zfs-monitoring}, and removes the orphaned
hand-written ups-heartbeat.sh. It ends by grepping for any surviving Kuma
reference and reporting it. Kept permanently and idempotent, so a rebuilt or
restored host cannot quietly bring them back.
A trap avoided: the monthly ZFS scrub lived INSIDE 32_zfs play 2. Deleting the
play wholesale would have silently stopped scrubbing the pool - and an
unscrubbed pool makes the health check meaningless, because it would have
nothing true to report. That play is now scrub-only and check-runs ok=5
changed=0.
ZFS pool capacity added as a sixth condition to the zfs-health check. `zpool
status` reports a 95% full pool as perfectly ONLINE, so capacity has to be read
separately with `zpool list` - and it is the failure you get warning of rather
than the one you discover. Threshold 80%, because ZFS allocation degrades badly
past roughly that and fragmentation is hard to undo. The pool is at 47%.
Verified both directions: passes on the real pool, and a simulated 91% exits 1.
Also fixed the waste recorded in c2de6db: the healthcheck role installed its
dependencies once per CHECK rather than per HOST - 29 apt transactions
estate-wide for a curl already present, and the slowest part of every deploy.
It now deduplicates within a play run, and the redundant standalone
daemon_reload is gone (the systemd task already does one).
site.yml updated, which exposed that services/gatus was never in it. It now runs
before the three registration playbooks, since registering endpoints against a
Gatus that is not yet serving would simply fail.
Verified: no legacy timer remains on any host; 83 endpoints, 83 UP, 0 DOWN.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-14 10:18:57 +02:00
|
|
|
# a deployment, and Kuma is being retired.
|
ansible: add site.yml, and rename the monitoring group off the host's name
site.yml is a TABLE OF CONTENTS, not a second source of truth. It is 25
import_playbook: lines and comments - no `hosts:`, no `roles:`. Which hosts get
what stays on the `hosts:` line inside each playbook, exactly where it already
was; nothing moved. Every role is already wrapped in a thin playbook carrying
its own `hosts:` line, so there is no roles-vs-playbooks split to reconcile:
from here everything is a playbook.
What it buys:
What runs on a host? ansible-playbook site.yml --limit <host> --list-hosts
Who gets thing Y? the `hosts:` line in Y's own playbook
What is a host? ansible-inventory --graph
Note --list-hosts, not --list-tasks: the latter prints every play regardless of
--limit, so it will happily show you the bitcoin play under memos-box.
Nine playbooks are deliberately excluded and the file names every one with a
reason, so it accounts for all of them: the three infra/4xx monitoring plays
(still assert on the removed Uptime Kuma credentials and fail immediately),
910_docker (says `hosts: managed`, but Docker is on 5 of 11 managed hosts and
those 5 are exactly the ones that need it - running it installs Docker on the
Bitcoin node and the hypervisor), two nodito one-shots, the Kuma notification
setup, and two deliberate manual actions.
Writing it surfaced an inventory collision. There is a HOST named `monitoring`
in [vps] AND a group [monitoring], so Ansible warned and resolved `hosts:
monitoring` to the host:
[WARNING]: Found both group and host with same name: monitoring
The group is renamed to [observability]; the host keeps its name. [caddy:children]
and the two ntfy playbooks follow. Behaviour is unchanged - `hosts: monitoring`
already resolved to the host - but the ambiguity is gone and the warning with it.
Verified: inventory graph is warning-free, site.yml passes --syntax-check, and
per-host play counts are identical before and after the rename.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-13 21:24:09 +02:00
|
|
|
#
|
|
|
|
|
# services/vaultwarden/disable_ deliberate manual actions, not convergence
|
|
|
|
|
# vaultwarden_sign_ups_playbook.yml
|
|
|
|
|
# services/personal-blog/setup_
|
|
|
|
|
# deploy_alias_lapy.yml
|