counterweight/personal_infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Compare commits

base: counterweight:master
Branches Tags
counterweight:master
counterweight:refactor-ansible
counterweight:zfs-monitoring
counterweight:nodito-setup
counterweight:before-refactor
..
compare: counterweight:zfs-monitoring
Branches Tags
counterweight:master
counterweight:refactor-ansible
counterweight:zfs-monitoring
counterweight:nodito-setup
counterweight:before-refactor

No commits in common. "master" and "zfs-monitoring" have entirely different histories.
master ... zfs-monito

302 changed files with 6615 additions and 13919 deletions
Download patch file Download diff file Expand all files Collapse all files

24
.gitignore vendored
View file

@ -1,23 +1,21 @@
# OpenTofu / Terraform # OpenTofu / Terraform
.terraform/ .terraform/
.tofu/ .tofu/
.terraform.lock.hcl
.tofu.lock.hcl
terraform.tfstate terraform.tfstate
terraform.tfstate.* terraform.tfstate.*
crash.log crash.log
*.tfvars
*.tfvars.json
# Provider credentials only. Non-secret infra config (the vms map) is committed test-inventory.ini
# as *.auto.tfvars, and *.lock.hcl is committed on purpose so provider hashes inventory.ini
# are pinned.
terraform.tfvars
terraform.tfvars.json
*secrets.auto.tfvars
venv/* venv/*
.env .env
# Secrets are ansible-vault encrypted and ARE committed. # Secrets and sensitive files
# Anything matching *_secrets.plain.yml is a working decryption — never commit those. *_secrets.yml
*_secrets.plain.yml *_secrets.yaml
secrets/
# Vault password — never commit .secrets/
ansible/.vault_pass

6
01_infra_setup.md
View file

@ -162,12 +162,6 @@ Note that, by applying these playbooks, both the root user and the `counterweigh
```bash ```bash
cp ansible/infra_secrets.yml.example ansible/infra_secrets.yml cp ansible/infra_secrets.yml.example ansible/infra_secrets.yml
``` ```
> **DEPRECATED (2026-09-11).** Uptime Kuma has been decommissioned. The server
> deployment was removed from this repo; what it monitored and how it was set up is
> preserved in [`archive/uptime_kuma/`](archive/uptime_kuma/). The monitoring blocks in
> the playbooks are kept but inert (`uptime_kuma_enabled: false`) so the check logic
> survives for whatever replaces it. The credentials below no longer exist in the vault.
* Edit `ansible/infra_secrets.yml` and add your Uptime Kuma credentials: * Edit `ansible/infra_secrets.yml` and add your Uptime Kuma credentials:
```yaml ```yaml
uptime_kuma_username: "admin" uptime_kuma_username: "admin"

74
02_vps_core_services_setup.md
View file

@ -49,12 +49,6 @@ Checklist:
## Uptime Kuma ## Uptime Kuma
> **DEPRECATED (2026-09-11).** Uptime Kuma has been decommissioned. The server
> deployment was removed from this repo; what it monitored and how it was set up is
> preserved in [`archive/uptime_kuma/`](archive/uptime_kuma/). The monitoring blocks in
> the playbooks are kept but inert (`uptime_kuma_enabled: false`) so the check logic
> survives for whatever replaces it. The credentials below no longer exist in the vault.
Uptime Kuma gets used to monitor the availability of services, keep track of their uptime and notify issues. Uptime Kuma gets used to monitor the availability of services, keep track of their uptime and notify issues.
### Deploy ### Deploy
@ -204,58 +198,9 @@ Checklist:
- [ ] SSH cloning works after setting up your SSH pub key - [ ] SSH cloning works after setting up your SSH pub key
## Phoenixd
phoenixd is the server version of the Phoenix wallet: a self-custodial Lightning node that leans on the ACINQ node for liquidity and channel management. It is the Lightning backend that LNBits talks to, so deploy it first.
It is not exposed to the internet. Its HTTP API is bound to `127.0.0.1` and only local consumers (LNBits, running on the same host) use it. There is no subdomain and no Caddy config for it.
### Deploy
* Review `ansible/services/phoenixd/phoenixd_vars.yml`. The things worth a look:
* `phoenixd_version`: pinned release tag.
* `phoenixd_auto_liquidity`: how much inbound liquidity phoenixd buys when it needs some (`off`, `2m`, `5m`, `10m`). This costs sats, it is not free.
* `phoenixd_http_bind_port`: the loopback API port LNBits will point at.
* No new secrets are needed: phoenixd generates its own API password on first start.
* Run the deployment playbook: `ansible-playbook -i inventory.ini services/phoenixd/deploy_phoenixd_playbook.yml`.
* The playbook will:
* Install `phoenixd` and `phoenix-cli` into `/usr/local/bin`
* Create the `phoenix` system user and a `0700` data dir at `/opt/phoenixd/.phoenix`
* Create and start the `phoenixd` systemd service
* Create a health check timer that runs `phoenix-cli getinfo` every minute
* Register a push monitor named `Phoenixd` in Uptime Kuma under the `services` group
### Back up the seed
**Do this immediately after the first deploy.** The seed is the only thing that recovers the funds.
* Run the backup playbook: `ansible-playbook -i inventory.ini services/phoenixd/setup_backup_phoenixd_to_lapy.yml`.
* This gpg encrypts `seed.dat` and `phoenix.conf` to Lapy daily and keeps 14 days. It does not stop the node and it does not back up the channel database on purpose: phoenixd keeps channel state with its peer, and restoring a stale channel db to a live node can force close channels and cost you a penalty.
* Also write the 12 words down offline, once: `sudo cat /opt/phoenixd/.phoenix/seed.dat`.
### Operate
* Status: `sudo systemctl status phoenixd`, logs: `sudo journalctl -u phoenixd -f`.
* CLI: `sudo PHOENIX_DATADIR=/opt/phoenixd/.phoenix phoenix-cli getinfo` (also `getbalance`, `listchannels`, `createinvoice`). If you changed `phoenixd_http_bind_port`, add `--http-bind-port <port>` before the subcommand.
* API password, needed to wire anything up to the node: `sudo grep '^http-password=' /opt/phoenixd/.phoenix/phoenix.conf`.
### Restoring on a fresh host
* Deploy phoenixd but stop the service before it generates a new seed, or just drop the file in before the first run:
* `sudo mkdir -p /opt/phoenixd/.phoenix`
* `echo "your twelve words here" | sudo tee /opt/phoenixd/.phoenix/seed.dat`
* `sudo chown -R phoenix:phoenix /opt/phoenixd/.phoenix && sudo chmod 600 /opt/phoenixd/.phoenix/seed.dat`
* Then run the deployment playbook. Never run two nodes on the same seed at once.
Checklist:
- [ ] `phoenix-cli getinfo` returns a node id
- [ ] The `Phoenixd` monitor in Uptime Kuma is green
- [ ] The seed is backed up to Lapy *and* written down offline
## LNBits ## LNBits
LNBits is a Lightning Network wallet and accounts system. It uses the phoenixd node deployed above as its Lightning backend. LNBits is a Lightning Network wallet and accounts system.
### Deploy ### Deploy
@ -269,7 +214,6 @@ LNBits is a Lightning Network wallet and accounts system. It uses the phoenixd n
* From that point on, you can configure through the Web UI. * From that point on, you can configure through the Web UI.
* Some advice around specifics of LNbits: * Some advice around specifics of LNbits:
* The default setup uses a FakeWallet backend for testing. Configure a real Lightning backend as needed by modifying the `.env` file located or using the superuser UI. * The default setup uses a FakeWallet backend for testing. Configure a real Lightning backend as needed by modifying the `.env` file located or using the superuser UI.
* To use the phoenixd node deployed above, set the backend to `PhoenixdWallet` with endpoint `http://127.0.0.1:9740` and the API password from `/opt/phoenixd/.phoenix/phoenix.conf`. The deployment playbook does not do this for you, since flipping the wallet backend on a live instance is not something to do behind your back.
* For security, disable the new users registration. * For security, disable the new users registration.
### Set up backups to Lapy ### Set up backups to Lapy
@ -285,6 +229,22 @@ LNBits is a Lightning Network wallet and accounts system. It uses the phoenixd n
* Start it up again. * Start it up again.
## ntfy-emergency-app
ntfy-emergency-app is a simple web application that allows trusted people to send emergency messages via ntfy notifications. Perfect for situations where you need to be alerted immediately but don't want to enable notifications on your regular messaging apps.
### Deploy
* Decide what subdomain you want to serve the emergency app on and update `ansible/services_config.yml` under `ntfy_emergency_app`.
* Note that you will have to add a DNS entry to point to the VPS public IP.
* Configure the ntfy settings in `ntfy_emergency_app_vars.yml`:
* `ntfy_emergency_app_topic`: The ntfy topic to send messages to (default: "emergency")
* `ntfy_emergency_app_ui_message`: Custom message displayed in the web interface
* Ensure `infra_secrets.yml` contains `ntfy_username` and `ntfy_password` with the credentials the app should use.
* Make sure docker is available on the host.
* Run the deployment playbook: `ansible-playbook -i inventory.ini services/ntfy-emergency-app/deploy_ntfy_emergency_app_playbook.yml`.
## Headscale ## Headscale
Headscale is a self-hosted Tailscale control server that allows you to create your own Tailscale network. Headscale is a self-hosted Tailscale control server that allows you to create your own Tailscale network.

69
03_vm_disk_enlargement.md
View file

@ -1,69 +0,0 @@
# 03 VM Disk Enlargement
How to enlarge the disk of an existing Proxmox VM that was provisioned via the tofu project.
## Overview
Since the tofu VM resource uses `ignore_changes = all` in its lifecycle block, disk resizes are performed manually through the Proxmox UI and then expanded inside the guest OS. You should still update the tofu vars to keep them in sync with reality.
## Step 1: Update disk size in tofu vars
* Edit `tofu/nodito/terraform.tfvars` and bump the `disk_size_gb` value for the VM you want to resize.
* This won't trigger any actual change (because of `ignore_changes = all`), but keeps the declared state consistent so future readers know the real disk size.
## Step 2: Resize the disk in Proxmox
* Open the Proxmox web UI and navigate to the VM.
* Go to the **Hardware** tab.
* Select the **Hard Disk** entry (scsi0).
* Click **Disk Action** > **Resize**.
* Enter the amount of additional space you want to add (this is the increment, not the total). For example, to go from 10 GB to 30 GB, enter `20`.
* Click **Resize disk**.
The VM does **not** need to be stopped for this operation. Proxmox will grow the underlying block device while the VM is running.
## Step 3: Expand the partition and filesystem inside the VM
SSH into the VM and run these steps as root (or with `sudo`).
### Check the current layout
```bash
lsblk
```
You should see that the disk (`sda`) now shows the new total size, but the root partition (`sda1`) still has the old size.
### Grow the partition
Install `cloud-guest-utils` if `growpart` is not available:
```bash
apt install -y cloud-guest-utils
```
Then grow the root partition to fill all available space:
```bash
growpart /dev/sda 1
```
> Note the space between the device and the partition number.
### Resize the filesystem
For ext4 (the default for Debian cloud images):
```bash
resize2fs /dev/sda1
```
This works online; no reboot is required.
### Verify
```bash
df -h /
```
The root filesystem should now reflect the new size.

8
README.md
View file

@ -6,12 +6,6 @@ My repo documenting my personal infra, along with artifacts, scripts, etc.
Go through the different numbered markdowns in the repo root to do the different parts. Go through the different numbered markdowns in the repo root to do the different parts.
## How to edit secrets
`ansible-vault edit ansible/your_file_with_secrets.yml`
Assumes that you've set `ansible/.vault_pass` with `chmod 600`.
## Overview ## Overview
### Services ### Services
@ -22,7 +16,7 @@ Assumes that you've set `ansible/.vault_pass` with `chmod 600`.
+ Plan install + Plan install
+ File based config + File based config
+ Crossbackup to Desky via rsync + Crossbackup to Desky via rsync
* ~~Uptime Kuma~~ — decommissioned 2026-09-11, see `archive/uptime_kuma/` * Uptime Kuma
+ Deployed on Vipy + Deployed on Vipy
+ Crossbackup to Desky via rsync + Crossbackup to Desky via rsync
* Vaultwarden * Vaultwarden

14
ansible/ansible.cfg
View file

@ -1,14 +0,0 @@
[defaults]
inventory = inventory.ini
roles_path = roles
collections_path = collections
interpreter_python = auto_silent
stdout_callback = yaml
retry_files_enabled = False
host_key_checking = True
forks = 10
vault_password_file = .vault_pass
[ssh_connection]
pipelining = True
ssh_args = -o ControlMaster=auto -o ControlPersist=300s

84
ansible/group_vars/all/main.yml
View file

@ -1,84 +0,0 @@
new_user: counterweight
ssh_port: 22
allow_ssh_from: "any"
root_domain: contrapeso.xyz
# Uptime Kuma was decommissioned on 2026-09-11. The monitoring blocks in the
# playbooks are kept deliberately — the check logic is meant to be rewired to
# whatever replaces it. This flag keeps them inert until then. See archive/uptime_kuma/.
# age recipient for all backup artefacts
age_backup_recipient: "age192wwdaseqej2ggwyp884gtm05c396anp7chr0vr8m47g50fahpyqr9fsza"
# Public key small-backups-box pulls with
# Authorised on each source host for an unprivileged, dedicated user only
backup_pull_public_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOfIixKMhA9z+Nvyx6ToZIniC8aEgyiInRiboaTTemgX offsite-backup-pull"
# ─────────────────────────────────────────────────────────────────────────────
# Subdomains. Global because the edge host proxies for services that live on
# other machines, so no single inventory group covers the readers. Combine with
# root_domain above to build an FQDN.
#
# Moved here from services_config.yml, which 30 plays had to remember to name in
# vars_files: - a file everyone must opt into is a file someone will forget.
# ─────────────────────────────────────────────────────────────────────────────
subdomains:
# Monitoring
gatus: status
# VPN infrastructure (spacey)
headscale: headscale
# Core services (vipy)
vaultwarden: vault
forgejo: forgejo
lnbits: wallet
# Secondary services (vipy)
personal_blog: pablohere
# Memos (memos-box)
memos: memos
# Mempool block explorer (mempool-box, proxied via vipy)
mempool: mempool
# DATUM Gateway dashboard (knots-box, proxied via vipy)
datum_gateway: datum
# Monero restricted RPC (monerod-box, proxied via vipy). Not a dashboard:
# this is the JSON-RPC endpoint Feather points at. Caddy terminates TLS and
# monerod's own Digest auth does the authentication.
monero: monero
# Monero block explorer (monerod-box, proxied via vipy). Fully public and
# unauthenticated, like any block explorer - the data is the public chain.
monero_explorer: exmrplore
# Navidrome music server (music-box, proxied via vipy). No basic auth:
# Navidrome has its own login, and the Subsonic API the mobile clients speak
# has nowhere to carry a second credential. Same call made for monero above.
music: music
# Jar Lid Archive staging (jar-lid-archive-staging-box, proxied via vipy).
# Public with no basic auth: the one-off photo links are for people without
# an account, and the app has its own login for everything else.
jar_lid_archive_staging: jars-staging
# Read by plays across several groups, so global rather than group_vars/<group>.
headscale_namespace: counter-net
# ─────────────────────────────────────────────────────────────────────────────
# Domains whose registration expiry is monitored (infra/402_public_monitoring).
#
# Registration renewal is a manual act at the registrar, and losing a domain is
# not recoverable in the way losing a host is - so these are checked daily and
# alarm with two weeks of runway.
#
# root_domain is the estate's own domain; the rest are domains we own that are
# served from it or from a host in the inventory.
# ─────────────────────────────────────────────────────────────────────────────
monitored_domains:
- "{{ root_domain }}"
- arbret.com

229
ansible/group_vars/all/vault.yml
View file

@ -1,229 +0,0 @@
$ANSIBLE_VAULT;1.1;AES256
62636131396363616337633131353962356639613766396163383633353937303334306661656662
3734393736396462323634396236386339623336663033310a366563623930616331616333343337
35613165633465653630626139393930333263323238353530663065613734646334663833323638
3364613961656433360a633136376664356262633338356163363631656665623562653034376338
33376661633533653766636135616261633133313338383238313335363461373064646261646262
36383030393235653566666537373232666538616435356332373530323536386535666532656461
32363638613131616137383133613966353232323537373938306232643738353163306336613238
36396539623730333232653338393636333464353536623337323935613461396136313131326133
35646561323165323431633039366435623965396363353535343165656631626166393234643639
37396139376531653839646233313837373136613337326439373062393030393130623638633334
30343266323938333333633064376632663737343332306533653232653837376637656166393935
32353231333532326436383532646336356162333738333037336337363133396465363937343330
38333335316663333232663966376234323261353036363262353362643061396533646139633539
35343065653763393032346333373734376133613039396161343438666566613530653463623239
63643762663933653333363665383634373762303462313064323336353862363530343831643336
64653030386438326633356138316338353838316562643138353065623262373639326638646365
38653637633731326364653834393132336637663135366230316437316365363366343530316236
31616466383462626266316330303532346536653435643234646534373064313564343661383139
33363864386266313563643264306537303066366463633062346237653963383131633436303234
32613462616261623632343566653836623539666539643865656366313264616635616638386237
65396135303739616635356332353336303861373965323834303261396162393235663963383963
38376136616537396162333566326465323761353030393961623262336165373931633333353330
30313137326661363137366265376132333063326130356334616537613330373664336462666137
39366336356332613836343039623330343231613530373463316435323034653732333362336362
65313763393863303935353430333430333632633036363932396238303464353465363032613536
66663766356262313135653964636437396665323461386337303731326166386331363134373133
65636436323336393034666638646632643336363031306163643266323333633031653734376662
61313734666436343062373037336262633333356462303562346536663862393431373334323363
32616663383135613933303430323030656130333264343231393164336536326464623964313131
38366139393732376665373537376563656332626530636363613433666237653835306433363031
62323734393933363064636464616132383636353965366365386261303530646233336266326366
63663332656233393232376235373836306432373434316532356264613733643530373230656135
32626463613663653336303064306563323133383834313936316366623931396537396165333563
31633264663030643164353138653565363533373336333833386330313335636230353030613061
39333462663965333565363730313731623538306436303337653361663035386130333665623463
63643536653831393361336665616361653764356163363832656463363038343330636239356136
33316238343538306538306564326337396633666166333863646462326164376365303836323530
33636230376635323166343631373530623437366461393638383532656564303233383865626332
65343063666462656236346634396464386437616332303437333932663966333162623463353733
66646563313363633361393862616563353232656136376138306239383561386636333231373436
30333336363831666530363132613862306334656133666231633063313263373336373562353238
35323166366163343430323439646164646161623264663966643334653237366335666464373131
37303663643638316334336665303562363266373130656230356336303531323161393365383430
33366165613236363138636237363766366366363539643361323664613461383965616333346661
30653436373030333232393366326435653061383963316632613163346165633335643731633961
34396261663939636237383439613461303735393233653164363837333235303132626436626132
65383661353261383437366235316533343664383033333966333531353131666530303439353464
39396337373331663634333839656362653761306130363136353861653033303430323938663339
64376636393935313633336464333930326432383164376139323533356438393761376462323764
35313339363032373463613335666438306139396634386466323039663735643563333664636364
63623636333837366165373637613431613962343966613934663065383333353761336331343466
31393466393661336539383161613963313661313366316437393835303564393335336630366434
34383434613637356232383863366465613133303964393033646239303932663563333535333931
32326630336233663131613063363533333133336463366466313362333331633430353433336131
39346639306630326336306537663262386264383938643362633762306364643435303533633635
33333364333466393038353434363334633965336533313135396436336432393333653832363531
38356633633632346263663935326264396632626263326663303861306363626134313761396233
31396435653463303530633739323930396438366166366137386566346338366566393961643865
33613365663863326331616163326666346633663136663330623331373439343330643261623665
63656439653935653637353134623865633666343136396361363835363861366162323234363737
62333135613539643261363462373534623631616438663035396162646161373665383364656438
61653337316537336337313231326639386665636263383566303164353535643238386563366661
63633037386536656332623331316361613064343833623766366665643232636265363731346538
30353037393365353333393261666532303035393665356165633263663066623464313935343536
61636462613738623861386534353332613734643534383430613564616235313432303566633963
61616662633638336261326365373735343264393432323965316330373430656635323937346239
61363433303235613630666331626361633938376564353831623664323334626233383133656235
66643732613432376561636436306236663734316136643735376564383362356262666131373038
66393931623637333237646332356531326533613932393232306461623235376664383334366562
30356365613331646530616431306330363063623234396265633936343832663036333364646337
32376138316533643839323766613838323265653632643430356234643665303666386664323031
66343764643038666237343536393737623862663363666464396535336335616136373930336433
37303862343834306266366335623036653738393234626434343337646338643631313662666634
64383034333936643164323736303639383237623235356639383165646433303935343062396633
32633436396530396330336238313466316334386531383962336165336461633836643963333164
61663434326463623737333530353965366361616639396238386565393533363339333235643761
64663230343964313435663335333039616165623466376638616534356638396538346234663436
38636233643265353561356666306165616462373639306135393866386433366536373938353764
64613637616431623963393534316631616439383161373135393161313332653134643237353037
64313966636566343766326132376336623065363937303633363634356438396161303437656636
34353562316663666430396630343665646165663061633235383232323363636637383465343538
37303733653166316564666163333963656265623264656666613539666166636466333866316634
32383466623962356338353334613166363932633330633836353037336139383732613766653030
32373064363766303436643331363732326665646334316663323562376561353063663837346537
39386531653762393263363261353136313961626663643865616266613834333664376664643761
31393661393634383030643462643062333635366135313564666336613965623730313138646365
62396565623636656335343737303732386161643537663236393831336336363764643762613164
66396262363265386164346665383961333332346139363337396639343731346131373932653332
62386564343639343930306164376432653961386165353333646134306231383334316366613337
62326339366130653333643333616636653937323130396563636666373832383066343666376465
37373932326538666664313534633661373035333765323733643064373865356338376130626137
32373763373734383539313839666665653536356335373933626361646139303235656132343033
30346435396637313234346436303964393863303735323864613038643136343237376138346334
31333637623339343263646533383434393862626134303463663832613836373565373164326435
62663461366639353537376664313463666437336264323165333734386362313430653839353437
66373130613338333034333663366564616562363632336236363237653536663963326230633136
30383635356362616263653637356234613537363364313437636662623038643537666431653533
35383165366537663839633638663938383836616366313831643630306435616431643464393562
30613634303266383535373965643534623338623466633365643839396564613236663631343439
38346164653764383333393432343565326537313639336630366163653937343566303938653665
32316262653630623638373964623430633466333835366439666530356535636630643063393562
37643966346437396463383133306466336365303137313932376536666462313262343131623161
64376261633964623761623666636134313038396564633436383839613730396139636530366333
36346338313330353437343464383232666338613337336464653663613662383230643939623136
63383163663062643436326166313432306166646264623933363935663862373737383434636234
36336464353934383034373166373733343133353165623661643333663332303266643538666530
35653063366463333961303761633763373266636639356239653636653061303865636332636632
35633333623236313566376131333830353734303665383433366239353562666664643662323065
35363164393138636135663462333131313632626535653239323463653634623339393061663861
61626531383964373730396536663434306531383437333335316166663864653561643939643837
30396137393665316239633330343161643562353735316330346565343366616536613330306664
32343537363466386664353638623034616631346366633862353538386566616137653162303466
38326531666130373864303235636261616565386264663335663862616265666366333739326631
38323766663864353331396334613735613436363263653938646661623735393038393661613266
37313932313434313735653330666639623730666462396433356364393931313632383634353334
66346539663836636139393839653237623932333537653666333832626333396235383731373237
33616634336662396538323436613166616339636235663933313437353737376539663438343164
34383138363833616364393537333331396566643562346464373134313433333931343939336261
32653231656265326662653639376365643234383265363466393864363562633066376664373261
62343233633034633261613839333461336639643136656166326534383366343239323262303630
36336265313233386361366139623663393638613538313265366462356463633862336364623962
31336363653964303466626239356435666562386535663832373333353062343034646432383832
35383831383331393962376665613634303634333864613066323530346134303934386564313666
36656162343965306461306132626336353435643334303531633736613633646536353433626437
63373033323434616539363035626632653663613838653232303039356633636264643432323236
35313461616231613563393436633933363935306236316466313166636564323139393137323534
34613432306237356663356434656331626161663138663665613663313263366162623561616234
63663865623765643236343035353739633266623366393139333261353539313462306132326463
61636665353861663838396236643563353134663831643061313234653830643864386131366562
65643866633632353932313733363163373237333635623736666139393766393634626430333039
63353536383138313165366362633230383137626364326333353037353736633664663738343766
35653536626531393832346634666131333730643465656136653433373134306461383231613963
63393366663137653635383830613239333337626432666436666363343637346331303731626137
32326366643439333966616165636632396366383364313832356636643032306131396534653339
39383732633132646464376233383139653865393061623763313865343964613031643965653939
36633963633033313530666661363331353836666137386536396533306165646131363235303336
32663537316164653935663034313833633135333837636331303363653864313231353734633461
31383035376639393335346531626266323730363934393062366263313136343665303036393933
62643965653932633334366363633763353862333335643265386533383062346266636639653064
31613464393866323363393731363766363063303234323861363166373430383266663234336332
61386561363562393736343939356434386464626134353634303931396662313665336531373235
36366463623562393139383033303630663062323564626135613837623335633763613435353964
30323161666264646630623532326431616135396133323838653736356338616339656638386464
31363331303164373031326161353666613337653536656330653865653062316636656237376562
34386431346330636464616136336531333331663865656266333964316261613265306434653038
35333935626461353033323733326634373036663133326336373665383233343636396466313566
37396462363764326361363561323633633537353165396362366166643362636639646536633065
34643633383264663137363264663964626335643938333933306232363466346430666230353561
37643836623262343664323664646161353230393435393033306536393531323166363739643432
32396236363839313131306266383866323861343537373738326265326637636531626431373335
65386564333961346633663633313737396565383431373265656131666638626462653335393330
33356362613535313063316330306330656263353463313538643232663136636136333539363663
30363066653633306537626664323339326565633134373831636633623930343635396362393561
35303237616335653263643132386237386639373939363536356238656163343133613065376531
66386361643465633938353438333730323933366465643533323738653762353437353136373134
36636533313665633362323361373839633839653338633336373561386663343361313733393031
34626566386635346439386564643261373363323466633839316564336137333636353364613062
37343038373933316665376539396263393765613034306139393962346563353065316638613738
66393739663736383265383735393331663739306462393939626661656534366665333132623831
34623962393535353665343537643938353033343634646664323535393762366639376132626362
34373230663937336161333934333935343564336237313837316363623665313564333134393133
64653562313036303137316436633065643062303335643266366361316163306433376565363034
63633830633937343530653537663465336563373435376366623537383165663033623539323333
66313238363266653065623532616331623635643038353032303266373565646136646538623530
31613038646331313738373634353737613435303964316161323938316261303237653362383233
64616133623432353139323835366238633562336161626564633530623035336232346465386332
64343036366430646161633461306661663564343132346564346633303934663539643239323536
61386238663037373534336535396230613565303566626631323138643365656132333635613532
34613633313633366638376662366464333664636538373133373939613565356637383863613035
30353261643161633734386331623765326664616636333965363161343661613937356636363864
38633132326237623964643734613165613133633832396633393432303535623237663864373135
34643964636565353635663633623933643164303734353766643665313933343565373530333833
36636165323030336562313764633039313165323630313665353366313135653636313338663233
66663431346234333364666661643165653963616230376130663062616366383238666561326364
36636530333633653763363435373534616435326533383030386164646130646164323262663330
32323163623534613036316134366463616639633934396363653539343830643239396633333064
31323264663866353530343033323066393332323161363031626136336531666566343332663230
62653635313763313461376236663432306130383264313763356531646637363432396365613766
37323935623031656438643330616563343632376230666236356534346466323533376631346564
63333536313132393164356337643465373737613939666666363361303636333436306632626166
63396637313235396236646237383162366531303133343962333639326139613231663737313532
37373732383734646330326662636330646135666462343133663336666462643264313565313933
31373035626564396665323235616138326539336133636232633034383262396466383138626264
61663166356162616465353565646166636238376464383033303636623334666236396637633635
30643861623632663632616130316630333234666439393030303961666239613530333937656563
30356363313733393663376433346138383966616465343934663734666437653863663133643437
64613236633736333237383834323338313431326634333730656536323138626438306231333035
35373936646264363337613134396337613039366363323638656437656564623831666461373064
66633232623666383738613834663938626438653430376534306164646363306663643362623137
61376433636336346266393532363131613938356262303564393636613833643738646230323938
32353763326631656561666533366236656134343638393435666437646231633463316232666563
63653635393334336266653265326239626531323035653930323438393235633962386438306133
34333537326530383363343139363461646339383064396637663465393961306139643761333538
39346362663830373535386665386533643435666438656638373861376537383538366331323461
66363138326432363432343335636434656139386161313162643838343566356264313734613630
64366162393932313131306533633666393564613835363664306639343161643339316537346338
64303764366136303737666464393463343261383538333563313164343666383132333761306463
62656138633939383239343533356136383735663834323763623437386434313333306230326237
38373666376332646635366233646237373431336531366432323130353034666265303236363937
36356536653465326139306635633032326535373734333664656339386266363365303139623463
38396664613639613737343062326538653835373861393939656636366337366134393236346463
33306462633639626463326230656338636664613730373731323531636337333062646233306139
61303661313764623565383265353061616638663636336635356364383933313363663330306638
39613637363935626363633635333533323735623963346334303661353030313066633636396361
65303738383131363237653866636637346464336334646362326538303232333661623038366332
36323736333538373931616638303030613765326165663066333131373734373766343066626131
32626462653538376431373635363835643036376230623061633963393065323262353931613165
39633136656666653231353036353034333634646266386533316232303462663161613239383931
64623034333137376134373235626165356466323931333131376238306536633861366430616362
63636233653339376665346635656239306161663538656166363563326564643661316563346666
38666161366430663234396237316534323263363436613435643436376535643430323432333639
32373436353936623966626263653037366263646232336665626533343366366530393535663733
32616639386433303032376565626535393466356136316662626364376666373934613166303233
39336136666437323766336430653433393134656230663961666239306639323039396432653936
65363736343637636538646130616362373965376437333762383138393134353332663734326565
31303136613739643965623437643663653339666638363861343164663538313234313330663738
37646362636130663431633762323634613964666166323962346234643765336639613934393938
64323239343832326564333731663961626661643936643233336662303335663861653331326161
64396564316465353163633066383562643636363562376136363733646439356530643432336139
63366265353435323565333161386563636134306261653366623036396266356137333364343930
30313465313834623530663436623333663262646331653832656266353834323463666332643130
32656539393838393766643962396133333038343166336166336539333434363433656265313339
31613238643339656130396461306134646666386430313636376633333465356230333433303666
37313534343037336432373639303534633235623264363064653635333837656132643665396132
66336463313139336163623637383164643237636134653837663062303238633237306562616161
62613536653036333431346131363864393730333432383632333034646361373731316431343662
39656264323965653738316264636361343361303466326336663438643430366235643165323632
34313331353530386266

24
ansible/group_vars/nodito_vms.yml
View file

@ -1,24 +0,0 @@
---
# Reach the VMs over Tailscale, and fall back to the LAN if the tailnet is down.
#
# ansible_host is a MagicDNS name. If tailscaled is not running on the control
# node that name does not resolve, the probe fails, and the LAN address recorded
# as lan_ip in inventory.ini takes over.
#
# Why the probe-then-connect shape rather than a plain `nc -w5 %h %p`:
# netcat-openbsd's -w is an IDLE timeout as well as a connect timeout, so a
# single `nc -w5` silently tears down the SSH session after five quiet seconds.
# That produced intermittent "Data could not be sent to remote host" failures on
# exactly the long, quiet operations (apt) where a dropped connection costs most.
# `nc -z` probes, then `exec nc` carries the session with no timeout at all.
#
# Safe against the LAN addresses drifting again (which is how fulcrum/mempool
# came to be transposed): known_hosts is keyed to the MagicDNS NAME, so if
# lan_ip ever points at a different machine the host key will not match and ssh
# aborts. Verified by pointing fulcrum-box at mempool-box's address:
# "Host key verification failed."
#
# lan_ip is a convenience, not an identity. If it goes stale the fallback stops
# working; it will never connect you to the wrong box.
ansible_ssh_common_args: >-
-o ProxyCommand="sh -c 'nc -z -w5 %h %p 2>/dev/null && exec nc %h %p || exec nc {{ lan_ip }} %p'"

17
ansible/host_vars/forgejo_runner_local/main.yml
View file

@ -1,17 +0,0 @@
---
# Systemd services deployed on this host, monitored every 5 minutes.
#
# The fact lives with the machine rather than in a central map, for the same
# reason the cross-host ports do: "what runs here" is a property of the host,
# and a central list is one more thing to forget to update when a service moves.
#
# Only units WE deploy belong here. Distro units (ssh, cron) have their own
# supervision and would be noise.
monitored_services:
- forgejo-runner
# The isolated Docker daemon that Actions jobs build on. Listed separately
# from forgejo-runner because they fail separately and the interesting case
# is the asymmetric one: dind down, runner up, every job failing while the
# runner's own unit stays green.
- forgejo-dind

17
ansible/host_vars/fulcrum_box_local/main.yml
View file

@ -1,17 +0,0 @@
# fulcrum-box: the Electrum server.
#
# Read by the fulcrum role here and by the socket-proxy play on the edge host,
# which publishes the port. See host_vars/knots_box_local/main.yml for why this
# lives in host_vars rather than in the role's defaults.
fulcrum_ssl_port: 50002
# Systemd services deployed on this host, monitored every 5 minutes.
#
# The fact lives with the machine rather than in a central map, for the same
# reason the cross-host ports do: "what runs here" is a property of the host,
# and a central list is one more thing to forget to update when a service moves.
#
# Only units WE deploy belong here. Distro units (ssh, cron) have their own
# supervision and would be noise.
monitored_services:
- fulcrum

28
ansible/host_vars/jar_lid_staging_local/main.yml
View file

@ -1,28 +0,0 @@
---
# Jar Lid Archive STAGING. Secrets are in vault.yml beside this file.
#
# These live here rather than in the role defaults because three plays read
# them: the deploy on this box, the Caddy vhost on the edge host (port, domain)
# and the backup (retention). A role default cannot reach a second play.
# Names this deployment everywhere it must be unique across hosts: the Caddy site
# file, the Gatus probe (probe_<instance>), the backup artefacts and the
# backup-dump/backup-store endpoints.
jar_lid_archive_instance: jar-lid-archive-staging
# Fixed by the image's CMD (gunicorn --bind 0.0.0.0:8000); changing this moves
# what Caddy and the probe call, not where gunicorn listens.
jar_lid_archive_port: 8000
jar_lid_archive_domain: "{{ subdomains.jar_lid_archive_staging }}.{{ root_domain }}"
# Staging data is disposable: three days locally and on small-backups-box.
jar_lid_archive_backup_retention_days: 3
jar_lid_archive_seed_users:
- {username: alice, email: alice@example.com, password: "{{ jar_lid_archive_seed_password_alice }}"}
- {username: bob, email: bob@example.com, password: "{{ jar_lid_archive_seed_password_bob }}"}
- {username: admin, email: admin@example.com, password: "{{ jar_lid_archive_seed_password_admin }}", is_superuser: true}
# No systemd units of ours run here - the stack is docker compose - so infra/401
# has nothing to watch. The probe in the deploy playbook calls /health/, which
# checks the database and the bucket as well as gunicorn.

34
ansible/host_vars/jar_lid_staging_local/vault.yml
View file

@ -1,34 +0,0 @@
$ANSIBLE_VAULT;1.1;AES256
37383535306133363031316535656461613038653562373839363432623163356234303061643566
6533333535613064653336313664643063326562666639640a373530386539316363346332663137
31303230323065623733663666383039346337343239646465393166663239643836356639373765
3564313065363266360a643239333330616463616533623435656164386264303634356134303831
36356131323637363734316161383065653663623732386638363165633265356362386538643039
64636464323730316638316430643339356662353736366332663738656566663137366166393735
34633430363931323330393234336661383532653366633735313365363562396137356438353330
65313737313131326361326631363864396536343633623963303632623465363964373964393430
61323161663939346435386435616364653336646165356232346335353235356339656336306138
65613636353636623562613362633761333637613066663838396638613161663532393030303035
35316464356539666631636265646433353537396564366538643962373364383730626164316265
63393034663835346362636564316135326430633963393139646365333666663630336462323361
31353830633335366635366435643864313665366563396134313938666537663232343162383366
34636231636261663739383366363962366438323962393732393864623238343961363035333866
61386630396334626564306536336561333265383464623730386263373132613438396332356631
30386435333739653637633032313966626437303634613838653735373666343565306663376230
61303734623865353835626464613033346437623234626136353664303434386262346634356463
34396433353431393935386136333331643539633138636135353339323930386634393639666665
36643462653061363062383039313936303062393364353366623965313530303031306336396336
65643663343163636338393666386134366337633064376530303463393737396165663930383562
64316363663636383764656139366433393766613262613433623637393538633130616232373136
31376636633666626365376533356262393065656539613266326465643663383439373661393631
30636136363066653936653837376634356333353036633165316662383337636435323162316430
30316331353361396132653233623034376539323064363135306262376638353664643163653334
37643638323038663531376336313064383238366430663363353733356530393661646138303230
30633862626166653362346632626261636531373130353034656532386230616330643134383631
63396138326336626431656130306531626539323934386332306361396239376466623866613364
36666632316331326361373136326465323365653034363337313135656262623465306439646362
30636639663163643430363165316231366232623332393635623461306339313762323265623732
30633464343235323566356434376535313238663533633630356163336532303765633366316166
38633462303762323930393337306265393033653061616337646163333335333737666332666632
35613132646636333335313263343136323337616662623465653939336637653134383766336265
35633638663636353537333238643236623033636163333164663062353863643164

26
ansible/host_vars/knots_box_local/main.yml
View file

@ -1,26 +0,0 @@
# knots-box: Bitcoin Knots and the DATUM Gateway.
#
# These ports are read twice: by the role that deploys the service here, and by
# the socket-proxy / Caddy plays that run on the EDGE host and publish them.
# A role default is invisible to that second play, which is why these live in
# host_vars rather than roles/<svc>/defaults/ - the edge play reads them as
# hostvars['knots_box_local'].<name>, and the role picks them up automatically
# because host_vars outranks role defaults.
#
# They used to live in services_config.yml, a file 30 plays had to remember to
# name in vars_files: and that four role defaults silently depended on.
bitcoin_p2p_port: 8333
datum_gateway_api_port: 7152
datum_gateway_stratum_port: 23334
# Systemd services deployed on this host, monitored every 5 minutes.
#
# The fact lives with the machine rather than in a central map, for the same
# reason the cross-host ports do: "what runs here" is a property of the host,
# and a central list is one more thing to forget to update when a service moves.
#
# Only units WE deploy belong here. Distro units (ssh, cron) have their own
# supervision and would be noise.
monitored_services:
- bitcoind
- datum-gateway

12
ansible/host_vars/memos_box_local/main.yml
View file

@ -1,12 +0,0 @@
---
# Systemd services deployed on this host, monitored every 5 minutes.
#
# The fact lives with the machine rather than in a central map, for the same
# reason the cross-host ports do: "what runs here" is a property of the host,
# and a central list is one more thing to forget to update when a service moves.
#
# Only units WE deploy belong here. Distro units (ssh, cron) have their own
# supervision and would be noise.
monitored_services:
- memos

6
ansible/host_vars/mempool_box_local/main.yml
View file

@ -1,6 +0,0 @@
# mempool-box: the Mempool block explorer.
#
# Read by the mempool role here and by the Caddy play on the edge host, which
# proxies to it. See host_vars/knots_box_local/main.yml for why this lives in
# host_vars rather than in the role's defaults.
mempool_frontend_port: 8080

31
ansible/host_vars/monerod_box_local/main.yml
View file

@ -1,31 +0,0 @@
# monerod-box: the Monero node.
#
# These ports are read twice: by the role that deploys the service here, and by
# the Caddy play that runs on the EDGE host and publishes the restricted RPC.
# A role default is invisible to that second play, which is why they live in
# host_vars - the edge play reads them as
# hostvars['monerod_box_local'].<name>, and the role picks them up
# automatically because host_vars outranks role defaults.
monero_p2p_port: 18080
monero_restricted_rpc_port: 18089
# The explorer's HTTP port, read by the Caddy play on the edge host.
xmrblocks_port: 8081
# The 500 GB chain disk, attached in Proxmox as scsi1. Named by that slot, which
# is fixed when the disk is attached and survives reboots and storage moves -
# not /dev/sdb, which it stopped being on 2026-10-04. Only read when the disk has
# no monero-data filesystem yet; see roles/monero/defaults/main.yml.
monero_data_device: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1
# Systemd services deployed on this host, monitored every 5 minutes.
#
# The fact lives with the machine rather than in a central map, for the same
# reason the cross-host ports do: "what runs here" is a property of the host,
# and a central list is one more thing to forget to update when a service moves.
#
# Only units WE deploy belong here. Distro units (ssh, cron) have their own
# supervision and would be noise.
monitored_services:
- monerod
- xmrblocks

11
ansible/host_vars/music_box_local/main.yml
View file

@ -1,11 +0,0 @@
---
# Navidrome's HTTP port. Read TWICE: by roles/navidrome on this box, and by the
# Caddy play on the edge host, which reads it as
# hostvars['music_box_local'].navidrome_port. A role default cannot reach a
# second play, which is why this lives here and not in defaults/main.yml.
navidrome_port: 4533
# Systemd units monitored every 5 minutes by infra/401_service_monitoring.yml.
# beets is not here: it is an interactive CLI, not a service.
monitored_services:
- navidrome

11
ansible/host_vars/nodito/vault.yml
View file

@ -1,11 +0,0 @@
$ANSIBLE_VAULT;1.1;AES256
30333035323663393939343061323234336164396465623665346165393534646366333332376463
3364373463333664363334373964323838336531353364310a636636373539623464336630666164
61376532616339376562373238383436306664313564663266303534346461666466383965323538
3163313239626663310a613033336332653165333537313366636361663036383031376561613761
31313563373062333033323037653939663762343161656264633436343361663737626366663732
39366666626338323436383134646263643538333564313566346336323563663534653161396136
39333565393538366238643563323630346166643461643063393631643665363566623631373762
36643866646637306231653837363838656163613766636265383139333838396535626335343163
32613935353330636263616333666230323436663935326133636362343836323535623237646235
6266316363366335323162663039366137633865396237373632

20
ansible/host_vars/spacey/main.yml
View file

@ -1,20 +0,0 @@
---
# Systemd services deployed on this host, monitored every 5 minutes.
#
# The fact lives with the machine rather than in a central map, for the same
# reason the cross-host ports do: "what runs here" is a property of the host,
# and a central list is one more thing to forget to update when a service moves.
#
# Only units WE deploy belong here. Distro units (ssh, cron) have their own
# supervision and would be noise.
monitored_services:
- headscale
- caddy
# Gatus moved onto this host in the 2026-10-17 rebuild (the `monitoring` droplet
# is gone). Headscale already listens on 0.0.0.0:8080, so Gatus takes 8081.
# Host vars rather than role defaults: the Caddy vhost play in
# services/gatus/deploy_gatus_playbook.yml does not load the role and reads
# this same variable.
gatus_port: 8081

20
ansible/host_vars/vipy/main.yml
View file

@ -1,20 +0,0 @@
---
# Systemd services deployed on this host, monitored every 5 minutes.
#
# The fact lives with the machine rather than in a central map, for the same
# reason the cross-host ports do: "what runs here" is a property of the host,
# and a central list is one more thing to forget to update when a service moves.
#
# Only units WE deploy belong here. Distro units (ssh, cron) have their own
# supervision and would be noise.
monitored_services:
- forgejo
- lnbits
- caddy
- phoenixd
# Deployed by the arbret repo, not this one (see arbret_handoff.md); listed
# here because "what runs on this host" is still a property of the host.
- arbret-server
- arbret-worker
- postgresql

4
ansible/infra/01_user_and_access_setup_playbook.yml
View file

@ -1,5 +1,7 @@
- name: Secure Debian - name: Secure Debian
hosts: managed hosts: all
vars_files:
- ../infra_vars.yml
become: true become: true
tasks: tasks:

4
ansible/infra/02_firewall_and_fail2ban_playbook.yml
View file

@ -1,5 +1,7 @@
- name: Secure Debian - name: Secure Debian
hosts: managed hosts: all
vars_files:
- ../infra_vars.yml
become: true become: true
tasks: tasks:

216
ansible/infra/400_host_monitoring.yml
View file

@ -1,216 +0,0 @@
---
# Host-level monitoring for the whole estate, reported to Gatus.
#
# Every check here PUSHES. Gatus never reaches out, which matters because nodito
# and its VMs sit behind NAT, and because four of the five checks are internal
# state with no pollable surface at all - disk usage, CPU temperature, ZFS pool
# health and UPS mains status cannot be observed from outside the machine.
#
# Liveness is a push too, and that is a choice rather than a limitation. A
# heartbeat proves the host is running AND can reach the internet; an ICMP probe
# from one vantage point only proves it answers pings from there. And because
# Gatus alerts when a heartbeat window expires, a check that stops running
# raises the alarm by itself - a dead timer looks exactly like a dead host,
# which is the correct reading.
#
# Each host has ONE bearer token, shared across its own checks: a token can only
# write results for that host's endpoints, so a compromised host can lie about
# itself, which it could do anyway.
#
# The push URL must use Gatus's own key format (config/key/key.go):
# key = sanitize(group) + "_" + sanitize(name)
# where sanitize lowercases and replaces / _ . , space # + & with "-". So
# knots_box_local becomes knots-box-local in the URL but stays readable in the
# name. host_key below is the Jinja equivalent; do not hand-write these.
# ─────────────────────────────────────────────────────────────────────────────
# Register everything with Gatus.
#
# This play runs FIRST on purpose. Gatus reloads its config within 30s, and the
# host plays below take minutes, so every endpoint exists before its first push
# arrives. Registering afterwards would 404 every first report.
#
# Heartbeat windows are several times the check interval, so one missed run - a
# slow apt run, a reboot - does not raise an alarm, but a check that has
# genuinely stopped does.
# ─────────────────────────────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# Alerting thresholds, and why they differ by check type.
#
# `failure-threshold` counts CONSECUTIVE failures, but "consecutive" means a
# different amount of wall-clock time per check:
#
# push/heartbeat endpoints a failure is produced once per heartbeat window
# pulled endpoints a failure is produced once per interval
#
# So the default of 3 would mean 33 minutes on an 11m heartbeat and over a day
# on a 7h one - and the heartbeat window ALREADY encodes the tolerance. An 11m
# window on a 5-minute push is precisely "one missed push forgiven"; stacking a
# threshold of 3 on top triples a tolerance that was already chosen.
#
# Hence: push endpoints alert on the FIRST heartbeat failure. Pulled endpoints
# have no built-in tolerance, so the threshold is where it belongs for them.
# ─────────────────────────────────────────────────────────────────────────────
- name: Register the host checks with Gatus
hosts: observability
become: yes
vars:
monitored: "{{ groups['managed'] | sort }}"
tasks:
- name: Build the liveness endpoint list
ansible.builtin.set_fact:
liveness_endpoints: "{{ liveness_endpoints | default([]) + [{
'name': item,
'group': 'liveness',
'token': gatus_push_tokens[item],
'heartbeat': '11m'}] }}"
loop: "{{ monitored }}"
- name: Build the disk endpoint list
ansible.builtin.set_fact:
disk_endpoints: "{{ disk_endpoints | default([]) + [{
'name': item,
'group': 'disk',
'token': gatus_push_tokens[item],
'heartbeat': '7h'}] }}"
loop: "{{ monitored }}"
- name: Register liveness endpoints
ansible.builtin.include_role:
name: gatus_endpoint
vars:
gatus_endpoint_default_alerts:
- type: signal
# 1, not 3: the heartbeat window is the tolerance. See the note above.
failure-threshold: 1
success-threshold: 2
send-on-resolved: true
minimum-reminder-interval: 6h
gatus_endpoint_name: liveness
gatus_endpoint_external: "{{ liveness_endpoints }}"
- name: Register disk endpoints
ansible.builtin.include_role:
name: gatus_endpoint
vars:
gatus_endpoint_default_alerts:
- type: signal
# 1, not 3: the heartbeat window is the tolerance. See the note above.
failure-threshold: 1
success-threshold: 2
send-on-resolved: true
minimum-reminder-interval: 6h
gatus_endpoint_name: disk
gatus_endpoint_external: "{{ disk_endpoints }}"
- name: Register the hypervisor endpoints
ansible.builtin.include_role:
name: gatus_endpoint
vars:
gatus_endpoint_default_alerts:
- type: signal
# 1, not 3: the heartbeat window is the tolerance. See the note above.
failure-threshold: 1
success-threshold: 2
send-on-resolved: true
minimum-reminder-interval: 6h
gatus_endpoint_name: hypervisor
gatus_endpoint_external:
- name: cpu
group: hypervisor
token: "{{ gatus_push_tokens['nodito'] }}"
heartbeat: "11m"
- name: zfs
group: hypervisor
token: "{{ gatus_push_tokens['nodito'] }}"
heartbeat: "7h"
- name: ups
group: hypervisor
token: "{{ gatus_push_tokens['nodito'] }}"
heartbeat: "11m"
- name: Deploy host liveness and disk checks
hosts: managed
become: yes
vars:
gatus_api: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints"
host_key: "{{ inventory_hostname | lower | regex_replace('[/_.,# +&]', '-') }}"
host_token: "{{ gatus_push_tokens[inventory_hostname] }}"
tasks:
- name: Is the host up?
ansible.builtin.include_role:
name: healthcheck
vars:
healthcheck_name: liveness
healthcheck_description: "Liveness heartbeat for {{ inventory_hostname }}"
healthcheck_check: liveness
healthcheck_interval: "5min"
healthcheck_boot_delay: "1min"
healthcheck_push_url: "{{ gatus_api }}/liveness_{{ host_key }}/external"
healthcheck_push_token: "{{ host_token }}"
- name: Is the disk packed?
ansible.builtin.include_role:
name: healthcheck
vars:
healthcheck_name: disk-usage
healthcheck_description: "Disk usage for {{ inventory_hostname }}"
healthcheck_check: disk-usage
# Every 6h rather than daily. Disk usage itself moves slowly, but the
# heartbeat can only be as tight as the push frequency - a daily push
# forces a >24h window, and a stuck check then hides for a day and a
# half. Six-hourly buys a 7h window. RandomizedDelaySec spreads the
# hosts so twelve boxes do not all report in the same second.
healthcheck_on_calendar: "*-*-* 00/6:00:00"
healthcheck_randomized_delay: "900"
healthcheck_boot_delay: "5min"
healthcheck_push_url: "{{ gatus_api }}/disk_{{ host_key }}/external"
healthcheck_push_token: "{{ host_token }}"
- name: Deploy the hypervisor-only checks
hosts: hypervisor
become: yes
vars:
gatus_api: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints"
host_token: "{{ gatus_push_tokens[inventory_hostname] }}"
tasks:
- name: Is the CPU hot?
ansible.builtin.include_role:
name: healthcheck
vars:
healthcheck_name: cpu-temp
healthcheck_description: "CPU temperature for {{ inventory_hostname }}"
healthcheck_check: cpu-temp
healthcheck_packages: [curl, lm-sensors]
healthcheck_interval: "5min"
healthcheck_push_url: "{{ gatus_api }}/hypervisor_cpu/external"
healthcheck_push_token: "{{ host_token }}"
- name: Is ZFS broken?
ansible.builtin.include_role:
name: healthcheck
vars:
healthcheck_name: zfs-health
healthcheck_description: "ZFS pool health for {{ zfs_pool_name }}"
healthcheck_check: zfs-health
healthcheck_packages: [curl, jq]
healthcheck_zfs_pool: "{{ zfs_pool_name }}"
healthcheck_on_calendar: "*-*-* 00/6:20:00"
healthcheck_boot_delay: "10min"
healthcheck_push_url: "{{ gatus_api }}/hypervisor_zfs/external"
healthcheck_push_token: "{{ host_token }}"
- name: Is the UPS online?
ansible.builtin.include_role:
name: healthcheck
vars:
healthcheck_name: ups-status
healthcheck_description: "UPS mains status for {{ ups_name }}"
healthcheck_check: ups-status
healthcheck_ups_name: "{{ ups_name }}"
healthcheck_interval: "5min"
healthcheck_push_url: "{{ gatus_api }}/hypervisor_ups/external"
healthcheck_push_token: "{{ host_token }}"

124
ansible/infra/401_service_monitoring.yml
View file

@ -1,124 +0,0 @@
---
# Is each systemd-deployed service actually running?
#
# Every 5 minutes, with an 11-minute Gatus heartbeat - one missed run before
# it alarms, so a reboot or a slow check does not page anyone, but a host that
# stops reporting does.
#
# This closes the gap that let a real bug run unnoticed: a backup script left
# forgejo, lnbits, headscale and memos stopped, and NOTHING caught it. The dumps
# exited 0, the artefacts were correct, the deploy said failed=0, and liveness
# only proves the HOST is up - not that anything on it is serving.
#
# One endpoint PER UNIT, not per host. A host running four services needs four
# endpoints, or a single red light says "something on vipy is down" without
# saying which - and that is the question you actually have at 3am. But only ONE
# timer per host: the check iterates that host's units and pushes a result for
# each, the same way check-backups.sh reports per source. Four units on vipy
# would otherwise mean four scripts, four services and four timers.
#
# Which units each host runs is in host_vars/<host>/main.yml as
# monitored_services, because "what runs here" is a property of the machine.
#
# Keys are host-qualified because unit names collide - caddy runs on four
# machines. Gatus computes sanitize(group)_sanitize(name), so group "services"
# and name "vipy/caddy" give services_vipy-caddy.
# ─────────────────────────────────────────────────────────────────────────────
# Register one endpoint per unit. Runs first: Gatus reloads within 30s, and the
# host play above takes minutes, so every endpoint exists before its first push.
# ─────────────────────────────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# Alerting thresholds, and why they differ by check type.
#
# `failure-threshold` counts CONSECUTIVE failures, but "consecutive" means a
# different amount of wall-clock time per check:
#
# push/heartbeat endpoints a failure is produced once per heartbeat window
# pulled endpoints a failure is produced once per interval
#
# So the default of 3 would mean 33 minutes on an 11m heartbeat and over a day
# on a 7h one - and the heartbeat window ALREADY encodes the tolerance. An 11m
# window on a 5-minute push is precisely "one missed push forgiven"; stacking a
# threshold of 3 on top triples a tolerance that was already chosen.
#
# Hence: push endpoints alert on the FIRST heartbeat failure. Pulled endpoints
# have no built-in tolerance, so the threshold is where it belongs for them.
#
# EXCEPT here, where the threshold is 2. The heartbeat only forgives a MISSING
# push. These checks also push an explicit `false` whenever a unit is caught
# mid-restart - a deploy, an unattended upgrade, a backup stopping the service
# for its dump - and at 1 every one of those pages, then resolves ten minutes
# later. 2 means "down on two consecutive 5-minute runs", i.e. 5-10 minutes.
# A host that stops reporting altogether still alarms on the heartbeat alone.
# ─────────────────────────────────────────────────────────────────────────────
- name: Register the service checks with Gatus
hosts: observability
become: yes
tasks:
# Two plain steps rather than one clever expression: first collect which
# units each host declares, then flatten that into endpoints.
- name: Collect the units each host declares
ansible.builtin.set_fact:
host_units: "{{ host_units | default([]) + [{'host': item, 'units': hostvars[item].monitored_services}] }}"
loop: "{{ groups['managed'] | sort }}"
when: hostvars[item].monitored_services | default([]) | length > 0
- name: Build one endpoint per unit
ansible.builtin.set_fact:
service_endpoints: "{{ service_endpoints | default([]) + [{
'name': (item.0.host | lower | regex_replace('[/_.,# +&]', '-')) ~ '/' ~ item.1,
'group': 'services',
'token': gatus_push_tokens[item.0.host],
'heartbeat': hostvars[item.0.host].service_health_heartbeat | default('11m')}] }}"
loop: "{{ host_units | subelements('units') }}"
- name: Register the service endpoints
ansible.builtin.include_role:
name: gatus_endpoint
vars:
gatus_endpoint_default_alerts:
- type: signal
# 2: tolerate one explicit `false` from a unit caught mid-restart.
# See the note above.
failure-threshold: 2
success-threshold: 2
send-on-resolved: true
minimum-reminder-interval: 6h
gatus_endpoint_name: services
gatus_endpoint_external: "{{ service_endpoints }}"
- name: Monitor systemd services on every host that has them
hosts: managed
become: yes
vars:
gatus_api: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints"
host_key: "{{ inventory_hostname | lower | regex_replace('[/_.,# +&]', '-') }}"
tasks:
- name: Is every deployed service running?
ansible.builtin.include_role:
name: healthcheck
vars:
healthcheck_name: service-health
healthcheck_description: "systemd services on {{ inventory_hostname }}"
healthcheck_check: systemd-units
healthcheck_units: "{{ monitored_services }}"
healthcheck_units_key_prefix: "services_{{ host_key }}"
# Cadence and heartbeat window are ONE decision, and both are a property
# of the host: a host can set service_health_on_calendar and a matching
# service_health_heartbeat in its host_vars. None does today; everything
# runs 5min/11m. (monerod-box was daily during its initial sync.)
#
# An empty healthcheck_on_calendar means "use healthcheck_interval" -
# see the timer template in roles/healthcheck.
healthcheck_interval: "5min"
healthcheck_on_calendar: "{{ service_health_on_calendar | default('') }}"
healthcheck_randomized_delay: "{{ service_health_randomized_delay | default('30') }}"
healthcheck_boot_delay: "2min"
# The per-unit results go to keys under this collection; the role's own
# single-result push is unused here, so only the base is set.
healthcheck_push_base: "{{ gatus_api }}"
healthcheck_push_token: "{{ gatus_push_tokens[inventory_hostname] }}"
when: monitored_services | default([]) | length > 0

213
ansible/infra/402_public_monitoring.yml
View file

@ -1,213 +0,0 @@
---
# Domain expiry, DNS correctness, and public endpoint reachability.
#
# These are the first checks in the estate that PULL rather than push, and that
# is the right way round for them: all three are about how the outside world
# sees us, so they must be measured from outside. Gatus polls from the
# observability host and needs nothing installed anywhere else - there is no
# script, no timer and no push token, because nothing is reporting in. (The
# provider-balance check does carry an API credential - see below.)
#
# That also means these have no heartbeat. A heartbeat answers "did the thing
# that was supposed to report in do so"; when Gatus does the checking itself,
# failure is immediate and self-evident.
- name: Register the public-facing checks with Gatus
hosts: observability
become: yes
vars:
# Expected A records, derived from inventory rather than written down again.
# The estate's recurring bug is an address recorded in a second place and
# then left behind when the machine moved, so the check asserts against
# ansible_host - if a box is renumbered, inventory is the one edit.
dns_records:
- {sub: "{{ subdomains.gatus }}", host: spacey}
- {sub: "{{ subdomains.headscale }}", host: spacey}
- {sub: "{{ subdomains.vaultwarden }}", host: vipy}
- {sub: "{{ subdomains.forgejo }}", host: vipy}
- {sub: "{{ subdomains.lnbits }}", host: vipy}
- {sub: "{{ subdomains.personal_blog }}", host: vipy}
- {sub: "{{ subdomains.memos }}", host: vipy}
- {sub: "{{ subdomains.mempool }}", host: vipy}
- {sub: "{{ subdomains.datum_gateway }}", host: vipy}
- {sub: "{{ subdomains.monero }}", host: vipy}
- {sub: "{{ subdomains.monero_explorer }}", host: vipy}
- {sub: "{{ subdomains.music }}", host: vipy}
- {sub: "{{ subdomains.jar_lid_archive_staging }}", host: vipy}
# Not under root_domain, so named in full. arbret.com's zone is NOT in
# Porkbun (mynymdns) - this is the check that would have caught it still
# pointing at prd-arbret's recycled IP after the 2026-10-17 rebuild.
- {fqdn: arbret.com, host: vipy}
- {fqdn: www.arbret.com, host: vipy}
# A public resolver on purpose: this must test what the internet sees, not
# what a local cache or the tailnet's MagicDNS happens to answer.
dns_resolver: "1.1.1.1"
# Expected status per site, checked live before being written down.
# 401 is the CORRECT answer for the two behind basic auth - asserting 200
# there would go green precisely when the auth broke.
public_sites:
- {name: gatus, sub: "{{ subdomains.gatus }}", path: "/", status: 401}
- {name: headscale, sub: "{{ subdomains.headscale }}", path: "/health", status: 200}
- {name: vaultwarden, sub: "{{ subdomains.vaultwarden }}", path: "/", status: 200}
- {name: forgejo, sub: "{{ subdomains.forgejo }}", path: "/", status: 200}
- {name: lnbits, sub: "{{ subdomains.lnbits }}", path: "/", status: 200}
- {name: blog, sub: "{{ subdomains.personal_blog }}", path: "/", status: 200}
- {name: memos, sub: "{{ subdomains.memos }}", path: "/", status: 200}
- {name: mempool, sub: "{{ subdomains.mempool }}", path: "/", status: 200}
- {name: datum, sub: "{{ subdomains.datum_gateway }}", path: "/", status: 401}
# 401 is CORRECT: monerod answers an unauthenticated request with an HTTP
# Digest challenge. Asserting 200 here would go green precisely when
# rpc-login broke and the node started serving the world for free.
- {name: monero, sub: "{{ subdomains.monero }}", path: "/get_info", status: 401}
- {name: exmrplore, sub: "{{ subdomains.monero_explorer }}", path: "/", status: 200}
# /app/, not "/": checked live, "/" answers 302 (Navidrome redirects to
# the SPA) and only /app/ returns 200 with the real UI. 200 rather than
# 401 because Navidrome has no basic auth in front of it - the Subsonic
# API the mobile clients speak cannot carry a proxy credential, so its
# own login is the gate. See roles/navidrome/README.md.
- {name: music, sub: "{{ subdomains.music }}", path: "/app/", status: 200}
# /health/ through the whole public path: DNS, vipy's Caddy and cert, the
# tailnet hop, gunicorn, and the database and bucket behind it (503 if
# either is down). "/" would only prove the login redirect works.
- {name: jars-staging, sub: "{{ subdomains.jar_lid_archive_staging }}", path: "/health/", status: 200}
# arbret (runs on vipy since 2026-10-17; deployed by the arbret repo, not
# this one). /api/health answers 200 + "status":"ok" when healthy and 503
# "degraded" otherwise, DB down included - per the arbret owner. The body
# condition guards against a 200 that is not actually healthy.
- {name: arbret, fqdn: arbret.com, path: "/", status: 200}
- {name: arbret-health, fqdn: arbret.com, path: "/api/health", status: 200,
conditions: ['[BODY].status == ok']}
# Ports published from the edge host by socket_proxy.
public_tcp:
- {name: bitcoin-p2p, host: vipy, port: "{{ hostvars['knots_box_local'].bitcoin_p2p_port }}"}
- {name: fulcrum-ssl, host: vipy, port: "{{ hostvars['fulcrum_box_local'].fulcrum_ssl_port }}"}
- {name: datum-stratum, host: vipy, port: "{{ hostvars['knots_box_local'].datum_gateway_stratum_port }}"}
tasks:
# ── Domain expiry ────────────────────────────────────────────────────────
# Each domain needs a URL SCHEME: Gatus derives the endpoint type from the
# prefix (endpoint.Type()), so a bare "example.com" is UNKNOWN and the whole
# config is rejected. No status is asserted, only the WHOIS/RDAP expiry, so
# whatever the apex serves - a real site, or the registrar's parking page -
# is irrelevant.
#
# 24h, and upstream enforces a 5m minimum for DOMAIN_EXPIRATION anyway
# because it uses a free whois service that must not be hammered.
# 336h = 14 days of runway, because renewal is a manual act at the registrar.
- name: Build the domain endpoints
ansible.builtin.set_fact:
domain_endpoints: "{{ domain_endpoints | default([]) + [{
'name': item,
'group': 'domain',
'url': 'https://' ~ item,
'interval': '24h',
'conditions': ['[DOMAIN_EXPIRATION] > 336h'],
'alerts': [{'type': 'signal', 'failure-threshold': 1,
'success-threshold': 1, 'send-on-resolved': true,
'minimum-reminder-interval': '168h'}]}] }}"
loop: "{{ monitored_domains }}"
# ── DNS ──────────────────────────────────────────────────────────────────
# 6h, not daily: a DNS query is cheap and a wrong record is an outage. The
# domain check stays at 24h because it does a WHOIS/RDAP lookup against a
# free service. Alert on the first failure - at a 6h interval, waiting for
# three would be nearly a day.
- name: Build the DNS endpoints
ansible.builtin.set_fact:
dns_endpoints: "{{ dns_endpoints | default([]) + [{
'name': (item.fqdn if item.fqdn is defined else item.sub ~ '.' ~ root_domain),
'group': 'dns',
'url': dns_resolver,
'interval': item.interval | default('6h'),
'dns': {'query-type': 'A', 'query-name': (item.fqdn if item.fqdn is defined else item.sub ~ '.' ~ root_domain)},
'conditions': ['[DNS_RCODE] == NOERROR',
'[BODY] == ' ~ hostvars[item.host].ansible_host],
'alerts': [{'type': 'signal', 'failure-threshold': 1,
'success-threshold': 1, 'send-on-resolved': true,
'minimum-reminder-interval': '24h'}]}] }}"
loop: "{{ dns_records }}"
# ── Public HTTP ──────────────────────────────────────────────────────────
# failure-threshold 3 at a 5m interval = 15 minutes. A pulled endpoint has
# no heartbeat window, so unlike the push checks the tolerance has to live
# in the threshold - and one failed poll of a public site is usually a blip.
- name: Build the public HTTP endpoints
ansible.builtin.set_fact:
http_endpoints: "{{ http_endpoints | default([]) + [{
'name': item.name,
'group': 'public',
'url': 'https://' ~ (item.fqdn if item.fqdn is defined else item.sub ~ '.' ~ root_domain) ~ item.path,
'interval': item.interval | default('5m'),
'conditions': ['[STATUS] == ' ~ item.status,
'[CERTIFICATE_EXPIRATION] > 168h']
+ item.conditions | default([]),
'alerts': [{'type': 'signal', 'failure-threshold': item.threshold | default(3),
'success-threshold': 2, 'send-on-resolved': true,
'minimum-reminder-interval': '6h'}]}] }}"
loop: "{{ public_sites }}"
# ── Public TCP ───────────────────────────────────────────────────────────
- name: Build the public TCP endpoints
ansible.builtin.set_fact:
tcp_endpoints: "{{ tcp_endpoints | default([]) + [{
'name': item.name,
'group': 'public',
'url': 'tcp://' ~ hostvars[item.host].ansible_host ~ ':' ~ item.port,
'interval': '5m',
'conditions': ['[CONNECTED] == true'],
'alerts': [{'type': 'signal', 'failure-threshold': 3,
'success-threshold': 2, 'send-on-resolved': true,
'minimum-reminder-interval': '6h'}]}] }}"
loop: "{{ public_tcp }}"
# ── Provider balance ─────────────────────────────────────────────────────
# 99stack is prepaid: when the balance runs out the servers stop. The one
# pulled check here that carries a credential - unlike everything above,
# the API will not answer anonymously (403 "Authorization token is
# missing").
#
# The condition is RUNWAY, not balance. billing.secondsdue is the
# provider's own "seconds until you run out at current usage", so it
# follows the spend as servers come and go; a fixed balance threshold
# would be one more number written down once and left behind. 336h of
# runway, same as domain expiry, because topping up is a manual act too.
#
# The token is account-wide - 99stack has no read-only scope - so it can
# create and destroy servers. It lands in Gatus's endpoints file (0640
# root:gatus), the same place as the push tokens.
- name: Require the 99stack API token
ansible.builtin.assert:
that:
- ninetyninestack_api_token is defined
- ninetyninestack_api_token | length > 0
fail_msg: >-
ninetyninestack_api_token must be in group_vars/all/vault.yml - an
API token from the 99stack control panel.
quiet: true
- name: Build the provider endpoints
ansible.builtin.set_fact:
provider_endpoints:
- name: 99stack-balance
group: provider
url: https://api.99stack.com/v1.5/account/info
interval: 6h
headers:
Authorization: "Bearer {{ ninetyninestack_api_token }}"
conditions:
- "[STATUS] == 200"
- "[BODY].billing.secondsdue > {{ 14 * 86400 }}"
alerts:
- {type: signal, failure-threshold: 1, success-threshold: 1,
send-on-resolved: true, minimum-reminder-interval: 24h}
- name: Register the public-facing endpoints
ansible.builtin.include_role:
name: gatus_endpoint
vars:
gatus_endpoint_name: public
gatus_endpoint_pulled: "{{ domain_endpoints + dns_endpoints + http_endpoints + tcp_endpoints + provider_endpoints }}"

98
ansible/infra/403_service_probe_registration.yml
View file

@ -1,98 +0,0 @@
---
# The per-service health probes.
#
# These are NOT the same thing as the systemd checks in infra/401. Those answer
# "is the unit running"; these answer "does the service actually respond" - an
# RPC call to bitcoind, a TCP connect to Fulcrum's Electrum port, an HTTP fetch
# from the Mempool backend. A process can be perfectly `active` and useless,
# which is precisely the gap these close.
#
# The checks themselves live in each service's own role, deployed by that
# service's playbook. This play only registers where they report, because the
# endpoints must exist in Gatus before the first push arrives.
#
# They used to push to Uptime Kuma. The scripts now POST with a bearer token
# instead of GETting ?status=up, and each host uses its own token.
# ─────────────────────────────────────────────────────────────────────────────
# Alerting thresholds, and why they differ by check type.
#
# `failure-threshold` counts CONSECUTIVE failures, but "consecutive" means a
# different amount of wall-clock time per check:
#
# push/heartbeat endpoints a failure is produced once per heartbeat window
# pulled endpoints a failure is produced once per interval
#
# So the default of 3 would mean 33 minutes on an 11m heartbeat and over a day
# on a 7h one - and the heartbeat window ALREADY encodes the tolerance. An 11m
# window on a 5-minute push is precisely "one missed push forgiven"; stacking a
# threshold of 3 on top triples a tolerance that was already chosen.
#
# Hence: push endpoints alert on the FIRST heartbeat failure. Pulled endpoints
# have no built-in tolerance, so the threshold is where it belongs for them.
# ─────────────────────────────────────────────────────────────────────────────
- name: Register the per-service probes with Gatus
hosts: observability
become: yes
vars:
probes:
# failure_threshold 4: this probe runs every MINUTE, and bitcoind turns RPC
# away for 1-3 minutes when a block lands - validation on the saturated
# HDD plus Fulcrum/mempool/DATUM all fetching the block at once (93
# "work queue depth exceeded" in one day, 2026-10-18). At 1, every slow
# block paged. 4 consecutive failed minutes is past any block burst seen,
# and a real outage still alerts within ~5 minutes.
- {name: bitcoin-knots, host: knots_box_local, failure_threshold: 4}
- {name: datum-gateway, host: knots_box_local}
- {name: fulcrum, host: fulcrum_box_local}
- {name: phoenixd, host: vipy}
- {name: forgejo-runner, host: forgejo_runner_local}
- {name: mempool-mariadb, host: mempool_box_local}
- {name: mempool-backend, host: mempool_box_local}
- {name: mempool-frontend, host: mempool_box_local}
# Every 5 minutes like the rest since 2026-10-17 (they were daily while
# the node synced on a rotating disk). Cadence lives in
# roles/{monero,xmrblocks}/defaults/main.yml; it and this window are one
# decision and must move together.
- {name: monerod, host: monerod_box_local}
- {name: xmrblocks, host: monerod_box_local}
- {name: navidrome, host: music_box_local}
- {name: jar-lid-archive-staging, host: jar_lid_staging_local}
# arbret (on vipy, deployed by the arbret repo). `host` here is the KEY in
# gatus_push_tokens, and arbret has a dedicated `arbret` token rather than
# vipy's - so it cannot write vipy's own results, and vice versa.
# Windows are two runs plus slack, as for the 5m probes above:
# analytics hourly at :00 -> 130m
# beancount daily at 03:00 -> 26h
# Cadence lives in the arbret repo; tell us if it changes.
- {name: arbret-analytics, host: arbret, heartbeat: '130m'}
- {name: arbret-beancount, host: arbret, heartbeat: '26h'}
# The alert every probe gets. failure-threshold is 1 - the heartbeat window
# is the tolerance, see the note above - unless a probe sets
# failure_threshold, for checks that legitimately fail for a minute or two.
probe_alert:
type: signal
failure-threshold: 1
success-threshold: 2
send-on-resolved: true
minimum-reminder-interval: 6h
tasks:
- name: Build the probe endpoint list
ansible.builtin.set_fact:
probe_endpoints: "{{ probe_endpoints | default([]) + [{
'name': item.name,
'group': 'probe',
'token': gatus_push_tokens[item.host],
'heartbeat': item.heartbeat | default('11m'),
'alerts': [probe_alert | combine({'failure-threshold': item.failure_threshold | default(1)})]}] }}"
loop: "{{ probes }}"
- name: Register the probe endpoints
ansible.builtin.include_role:
name: gatus_endpoint
vars:
gatus_endpoint_name: probes
gatus_endpoint_external: "{{ probe_endpoints }}"

109
ansible/infra/409_remove_legacy_monitoring.yml
View file

@ -1,109 +0,0 @@
---
# Remove the Uptime-Kuma-era monitoring that 400/401/402 replaced.
#
# Deleting the playbooks that installed these is NOT enough: the units are on
# the hosts, enabled, and keep firing regardless of what the repo says. Two of
# them still push to https://uptime.contrapeso.xyz every 15 minutes. A playbook
# that is deleted without a cleanup leaves its output running forever, with
# nothing in the repo left to explain it.
#
# What replaced what, all verified against the deployed scripts before removal:
#
# disk-usage-monitor -> disk-usage-healthcheck (infra/400)
# The old one checked ONLY "/" at 80%. The replacement walks every real
# filesystem, excluding tmpfs/devtmpfs/squashfs/overlay, at 85%. Strictly
# more coverage, so nothing is lost.
#
# system-healthcheck -> liveness-healthcheck (infra/400)
# The old script computed uptime and pushed. That is exactly a liveness
# heartbeat and nothing more.
#
# nodito-cpu-temp-monitor -> cpu-temp-healthcheck (infra/400)
# zfs-health-monitor -> zfs-health-healthcheck (infra/400)
# The ZFS check logic was ported verbatim - same five conditions - so only
# the reporting transport changed.
#
# NOT removed, because they are not monitoring:
# zfs-monthly-scrub.{timer,service} the actual scrub (infra/nodito/32)
# pull-backups, check-backups the backup machinery (playbooks/backups)
#
# This play is idempotent and kept permanently rather than run once and deleted:
# on a host that never had these it does nothing, and it guarantees a rebuilt or
# restored machine cannot quietly bring them back.
- name: Remove the legacy Uptime Kuma monitoring units
hosts: managed
become: yes
vars:
legacy_units:
- disk-usage-monitor
- system-healthcheck
- nodito-cpu-temp-monitor
- zfs-health-monitor
legacy_dirs:
- /opt/disk-monitoring
- /opt/system-healthcheck
- /opt/nodito-monitoring
- /opt/zfs-monitoring
tasks:
- name: Find which legacy units exist here
ansible.builtin.stat:
path: "/etc/systemd/system/{{ item.0 }}.{{ item.1 }}"
register: legacy_unit_files
loop: "{{ legacy_units | product(['timer', 'service']) | list }}"
# Stop and disable BEFORE deleting the unit file: systemd cannot disable a
# unit whose file has already gone, which would leave a dangling symlink in
# multi-user.target.wants and a warning on every daemon-reload.
- name: Stop and disable the legacy units
ansible.builtin.systemd:
name: "{{ item.item.0 }}.{{ item.item.1 }}"
state: stopped
enabled: no
loop: "{{ legacy_unit_files.results }}"
loop_control:
label: "{{ item.item.0 }}.{{ item.item.1 }}"
when: item.stat.exists
failed_when: false
- name: Remove the legacy unit files
ansible.builtin.file:
path: "/etc/systemd/system/{{ item.item.0 }}.{{ item.item.1 }}"
state: absent
loop: "{{ legacy_unit_files.results }}"
loop_control:
label: "{{ item.item.0 }}.{{ item.item.1 }}"
when: item.stat.exists
- name: Reload systemd
ansible.builtin.systemd:
daemon_reload: yes
- name: Remove the legacy monitoring scripts and their logs
ansible.builtin.file:
path: "{{ item }}"
state: absent
loop: "{{ legacy_dirs }}"
# An orphan predating all of this: mode 0644, not executable, referenced by
# no unit and no cron entry, pushing to a Kuma monitor. Superseded by
# ups-status-healthcheck.
- name: Remove the orphaned hand-written UPS heartbeat
ansible.builtin.file:
path: /usr/local/bin/ups-heartbeat.sh
state: absent
- name: Confirm nothing still pushes to Uptime Kuma
ansible.builtin.shell: >-
grep -rl "uptime.contrapeso.xyz" /etc/systemd/system /usr/local/bin /opt 2>/dev/null || true
register: kuma_refs
changed_when: false
- name: Report any remaining references
ansible.builtin.debug:
msg: >-
{{ 'clean - nothing references Uptime Kuma'
if kuma_refs.stdout | trim | length == 0
else 'STILL REFERENCING KUMA: ' ~ kuma_refs.stdout_lines | join(', ') }}

330
ansible/infra/410_disk_usage_alerts.yml Normal file
View file

@ -0,0 +1,330 @@
- name: Deploy Disk Usage Monitoring
hosts: all
become: yes
vars_files:
- ../infra_vars.yml
- ../services_config.yml
- ../infra_secrets.yml
vars:
disk_usage_threshold_percent: 80
disk_check_interval_minutes: 15
monitored_mount_point: "/"
monitoring_script_dir: /opt/disk-monitoring
monitoring_script_path: "{{ monitoring_script_dir }}/disk_usage_monitor.sh"
log_file: "{{ monitoring_script_dir }}/disk_usage_monitor.log"
systemd_service_name: disk-usage-monitor
# Uptime Kuma configuration (auto-configured from services_config.yml and infra_secrets.yml)
uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}"
ntfy_topic: "{{ service_settings.ntfy.topic }}"
tasks:
- name: Validate Uptime Kuma configuration
assert:
that:
- uptime_kuma_api_url is defined
- uptime_kuma_api_url != ""
- uptime_kuma_username is defined
- uptime_kuma_username != ""
- uptime_kuma_password is defined
- uptime_kuma_password != ""
fail_msg: "uptime_kuma_api_url, uptime_kuma_username and uptime_kuma_password must be set"
- name: Get hostname for monitor identification
command: hostname
register: host_name
changed_when: false
- name: Set monitor name and group based on hostname and mount point
set_fact:
monitor_name: "disk-usage-{{ host_name.stdout }}-{{ monitored_mount_point | replace('/', 'root') }}"
monitor_friendly_name: "Disk Usage: {{ host_name.stdout }} ({{ monitored_mount_point }})"
uptime_kuma_monitor_group: "{{ host_name.stdout }} - infra"
- name: Create Uptime Kuma monitor setup script
copy:
dest: /tmp/setup_uptime_kuma_monitor.py
content: |
#!/usr/bin/env python3
import sys
import json
from uptime_kuma_api import UptimeKumaApi
def main():
api_url = sys.argv[1]
username = sys.argv[2]
password = sys.argv[3]
group_name = sys.argv[4]
monitor_name = sys.argv[5]
monitor_description = sys.argv[6]
interval = int(sys.argv[7])
ntfy_topic = sys.argv[8] if len(sys.argv) > 8 else "alerts"
api = UptimeKumaApi(api_url, timeout=60, wait_events=2.0)
api.login(username, password)
# Get all monitors
monitors = api.get_monitors()
# Get all notifications and find ntfy notification
notifications = api.get_notifications()
ntfy_notification = next((n for n in notifications if n.get('name') == f'ntfy ({ntfy_topic})'), None)
notification_id_list = {}
if ntfy_notification:
notification_id_list[ntfy_notification['id']] = True
# Find or create group
group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None)
if not group:
group_result = api.add_monitor(type='group', name=group_name)
# Refresh to get the full group object with id
monitors = api.get_monitors()
group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None)
# Find or create/update push monitor
existing_monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
monitor_data = {
'type': 'push',
'name': monitor_name,
'parent': group['id'],
'interval': interval,
'upsideDown': True,
'description': monitor_description,
'notificationIDList': notification_id_list
}
if existing_monitor:
monitor = api.edit_monitor(existing_monitor['id'], **monitor_data)
# Refresh to get the full monitor object with pushToken
monitors = api.get_monitors()
monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
else:
monitor_result = api.add_monitor(**monitor_data)
# Refresh to get the full monitor object with pushToken
monitors = api.get_monitors()
monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
# Output result as JSON
result = {
'monitor_id': monitor['id'],
'push_token': monitor['pushToken'],
'group_name': group_name,
'group_id': group['id'],
'monitor_name': monitor_name
}
print(json.dumps(result))
api.disconnect()
if __name__ == '__main__':
main()
mode: '0755'
delegate_to: localhost
become: no
- name: Run Uptime Kuma monitor setup script
command: >
{{ ansible_playbook_python }}
/tmp/setup_uptime_kuma_monitor.py
"{{ uptime_kuma_api_url }}"
"{{ uptime_kuma_username }}"
"{{ uptime_kuma_password }}"
"{{ uptime_kuma_monitor_group }}"
"{{ monitor_name }}"
"{{ monitor_friendly_name }} - Alerts when usage exceeds {{ disk_usage_threshold_percent }}%"
"{{ (disk_check_interval_minutes * 60) + 60 }}"
"{{ ntfy_topic }}"
register: monitor_setup_result
delegate_to: localhost
become: no
changed_when: false
- name: Parse monitor setup result
set_fact:
monitor_info_parsed: "{{ monitor_setup_result.stdout | from_json }}"
- name: Set push URL and monitor ID as facts
set_fact:
uptime_kuma_disk_usage_push_url: "{{ uptime_kuma_api_url }}/api/push/{{ monitor_info_parsed.push_token }}"
uptime_kuma_monitor_id: "{{ monitor_info_parsed.monitor_id }}"
- name: Install required packages for disk monitoring
package:
name:
- curl
state: present
- name: Create monitoring script directory
file:
path: "{{ monitoring_script_dir }}"
state: directory
owner: root
group: root
mode: '0755'
- name: Create disk usage monitoring script
copy:
dest: "{{ monitoring_script_path }}"
content: |
#!/bin/bash
# Disk Usage Monitoring Script
# Monitors disk usage and sends alerts to Uptime Kuma
# Mode: "No news is good news" - only sends alerts when disk usage is HIGH
LOG_FILE="{{ log_file }}"
USAGE_THRESHOLD="{{ disk_usage_threshold_percent }}"
UPTIME_KUMA_URL="{{ uptime_kuma_disk_usage_push_url }}"
MOUNT_POINT="{{ monitored_mount_point }}"
# Function to log messages
log_message() {
echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
}
# Function to get disk usage percentage
get_disk_usage() {
local mount_point="$1"
local usage=""
# Get disk usage percentage (without % sign)
usage=$(df -h "$mount_point" 2>/dev/null | awk 'NR==2 {gsub(/%/, "", $5); print $5}')
if [ -z "$usage" ]; then
log_message "ERROR: Could not read disk usage for $mount_point"
return 1
fi
echo "$usage"
}
# Function to get disk usage details
get_disk_details() {
local mount_point="$1"
df -h "$mount_point" 2>/dev/null | awk 'NR==2 {print "Used: "$3" / Total: "$2" ("$5" full)"}'
}
# Function to send alert to Uptime Kuma when disk usage exceeds threshold
# With upside-down mode enabled, sending status=up will trigger an alert
send_uptime_kuma_alert() {
local usage="$1"
local details="$2"
local message="DISK FULL WARNING: ${MOUNT_POINT} is ${usage}% full (Threshold: ${USAGE_THRESHOLD}%) - ${details}"
log_message "ALERT: $message"
# Send push notification to Uptime Kuma with status=up
# In upside-down mode, status=up is treated as down/alert
response=$(curl -s -w "\n%{http_code}" -G \
--data-urlencode "status=up" \
--data-urlencode "msg=$message" \
"$UPTIME_KUMA_URL" 2>&1)
http_code=$(echo "$response" | tail -n1)
if [ "$http_code" = "200" ] || [ "$http_code" = "201" ]; then
log_message "Alert sent successfully to Uptime Kuma (HTTP $http_code)"
else
log_message "ERROR: Failed to send alert to Uptime Kuma (HTTP $http_code)"
fi
}
# Main monitoring logic
main() {
log_message "Starting disk usage check for $MOUNT_POINT"
# Get current disk usage
current_usage=$(get_disk_usage "$MOUNT_POINT")
if [ $? -ne 0 ] || [ -z "$current_usage" ]; then
log_message "ERROR: Could not read disk usage"
exit 1
fi
# Get disk details
disk_details=$(get_disk_details "$MOUNT_POINT")
log_message "Current disk usage: ${current_usage}% - $disk_details"
# Check if usage exceeds threshold
if [ "$current_usage" -gt "$USAGE_THRESHOLD" ]; then
log_message "WARNING: Disk usage ${current_usage}% exceeds threshold ${USAGE_THRESHOLD}%"
send_uptime_kuma_alert "$current_usage" "$disk_details"
else
log_message "Disk usage is within normal range - no alert needed (no news is good news)"
fi
}
# Run main function
main
owner: root
group: root
mode: '0755'
- name: Create systemd service for disk usage monitoring
copy:
dest: "/etc/systemd/system/{{ systemd_service_name }}.service"
content: |
[Unit]
Description=Disk Usage Monitor
After=network.target
[Service]
Type=oneshot
ExecStart={{ monitoring_script_path }}
User=root
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
owner: root
group: root
mode: '0644'
- name: Create systemd timer for disk usage monitoring
copy:
dest: "/etc/systemd/system/{{ systemd_service_name }}.timer"
content: |
[Unit]
Description=Run Disk Usage Monitor every {{ disk_check_interval_minutes }} minute(s)
Requires={{ systemd_service_name }}.service
[Timer]
OnBootSec={{ disk_check_interval_minutes }}min
OnUnitActiveSec={{ disk_check_interval_minutes }}min
Persistent=true
[Install]
WantedBy=timers.target
owner: root
group: root
mode: '0644'
- name: Reload systemd daemon
systemd:
daemon_reload: yes
- name: Enable and start disk usage monitoring timer
systemd:
name: "{{ systemd_service_name }}.timer"
enabled: yes
state: started
- name: Test disk usage monitoring script
command: "{{ monitoring_script_path }}"
register: script_test
changed_when: false
- name: Verify script execution
assert:
that:
- script_test.rc == 0
fail_msg: "Disk usage monitoring script failed to execute properly"
- name: Clean up temporary Uptime Kuma setup script
file:
path: /tmp/setup_uptime_kuma_monitor.py
state: absent
delegate_to: localhost
become: no

312
ansible/infra/420_system_healthcheck.yml Normal file
View file

@ -0,0 +1,312 @@
- name: Deploy System Healthcheck Monitoring
hosts: all
become: yes
vars_files:
- ../infra_vars.yml
- ../services_config.yml
- ../infra_secrets.yml
vars:
healthcheck_interval_seconds: 60 # Send healthcheck every 60 seconds (1 minute)
healthcheck_timeout_seconds: 90 # Uptime Kuma should alert if no ping received within 90s
healthcheck_retries: 1 # Number of retries before alerting
monitoring_script_dir: /opt/system-healthcheck
monitoring_script_path: "{{ monitoring_script_dir }}/system_healthcheck.sh"
log_file: "{{ monitoring_script_dir }}/system_healthcheck.log"
systemd_service_name: system-healthcheck
# Uptime Kuma configuration (auto-configured from services_config.yml and infra_secrets.yml)
uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}"
ntfy_topic: "{{ service_settings.ntfy.topic }}"
tasks:
- name: Validate Uptime Kuma configuration
assert:
that:
- uptime_kuma_api_url is defined
- uptime_kuma_api_url != ""
- uptime_kuma_username is defined
- uptime_kuma_username != ""
- uptime_kuma_password is defined
- uptime_kuma_password != ""
fail_msg: "uptime_kuma_api_url, uptime_kuma_username and uptime_kuma_password must be set"
- name: Get hostname for monitor identification
command: hostname
register: host_name
changed_when: false
- name: Set monitor name and group based on hostname
set_fact:
monitor_name: "system-healthcheck-{{ host_name.stdout }}"
monitor_friendly_name: "System Healthcheck: {{ host_name.stdout }}"
uptime_kuma_monitor_group: "{{ host_name.stdout }} - infra"
- name: Create Uptime Kuma monitor setup script
copy:
dest: /tmp/setup_uptime_kuma_healthcheck_monitor.py
content: |
#!/usr/bin/env python3
import sys
import json
from uptime_kuma_api import UptimeKumaApi
def main():
api_url = sys.argv[1]
username = sys.argv[2]
password = sys.argv[3]
group_name = sys.argv[4]
monitor_name = sys.argv[5]
monitor_description = sys.argv[6]
interval = int(sys.argv[7])
retries = int(sys.argv[8])
ntfy_topic = sys.argv[9] if len(sys.argv) > 9 else "alerts"
api = UptimeKumaApi(api_url, timeout=60, wait_events=2.0)
api.login(username, password)
# Get all monitors
monitors = api.get_monitors()
# Get all notifications and find ntfy notification
notifications = api.get_notifications()
ntfy_notification = next((n for n in notifications if n.get('name') == f'ntfy ({ntfy_topic})'), None)
notification_id_list = {}
if ntfy_notification:
notification_id_list[ntfy_notification['id']] = True
# Find or create group
group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None)
if not group:
group_result = api.add_monitor(type='group', name=group_name)
# Refresh to get the full group object with id
monitors = api.get_monitors()
group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None)
# Find or create/update push monitor
existing_monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
monitor_data = {
'type': 'push',
'name': monitor_name,
'parent': group['id'],
'interval': interval,
'upsideDown': False, # Normal mode: receiving pings = healthy
'maxretries': retries,
'description': monitor_description,
'notificationIDList': notification_id_list
}
if existing_monitor:
monitor = api.edit_monitor(existing_monitor['id'], **monitor_data)
# Refresh to get the full monitor object with pushToken
monitors = api.get_monitors()
monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
else:
monitor_result = api.add_monitor(**monitor_data)
# Refresh to get the full monitor object with pushToken
monitors = api.get_monitors()
monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
# Output result as JSON
result = {
'monitor_id': monitor['id'],
'push_token': monitor['pushToken'],
'group_name': group_name,
'group_id': group['id'],
'monitor_name': monitor_name
}
print(json.dumps(result))
api.disconnect()
if __name__ == '__main__':
main()
mode: '0755'
delegate_to: localhost
become: no
- name: Run Uptime Kuma monitor setup script
command: >
{{ ansible_playbook_python }}
/tmp/setup_uptime_kuma_healthcheck_monitor.py
"{{ uptime_kuma_api_url }}"
"{{ uptime_kuma_username }}"
"{{ uptime_kuma_password }}"
"{{ uptime_kuma_monitor_group }}"
"{{ monitor_name }}"
"{{ monitor_friendly_name }} - Regular healthcheck ping every {{ healthcheck_interval_seconds }}s"
"{{ healthcheck_timeout_seconds }}"
"{{ healthcheck_retries }}"
"{{ ntfy_topic }}"
register: monitor_setup_result
delegate_to: localhost
become: no
changed_when: false
- name: Parse monitor setup result
set_fact:
monitor_info_parsed: "{{ monitor_setup_result.stdout | from_json }}"
- name: Set push URL and monitor ID as facts
set_fact:
uptime_kuma_healthcheck_push_url: "{{ uptime_kuma_api_url }}/api/push/{{ monitor_info_parsed.push_token }}"
uptime_kuma_monitor_id: "{{ monitor_info_parsed.monitor_id }}"
- name: Install required packages for healthcheck monitoring
package:
name:
- curl
state: present
- name: Create monitoring script directory
file:
path: "{{ monitoring_script_dir }}"
state: directory
owner: root
group: root
mode: '0755'
- name: Create system healthcheck script
copy:
dest: "{{ monitoring_script_path }}"
content: |
#!/bin/bash
# System Healthcheck Script
# Sends regular heartbeat pings to Uptime Kuma
# This ensures the system is running and able to communicate
LOG_FILE="{{ log_file }}"
UPTIME_KUMA_URL="{{ uptime_kuma_healthcheck_push_url }}"
HOSTNAME=$(hostname)
# Function to log messages
log_message() {
echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
}
# Function to send healthcheck ping to Uptime Kuma
send_healthcheck() {
local uptime_seconds=$(awk '{print int($1)}' /proc/uptime)
local uptime_days=$((uptime_seconds / 86400))
local uptime_hours=$(((uptime_seconds % 86400) / 3600))
local uptime_minutes=$(((uptime_seconds % 3600) / 60))
local message="System healthy - Uptime: ${uptime_days}d ${uptime_hours}h ${uptime_minutes}m"
log_message "Sending healthcheck ping: $message"
# Send push notification to Uptime Kuma with status=up
encoded_message=$(printf '%s\n' "$message" | sed 's/ /%20/g; s/(/%28/g; s/)/%29/g; s/:/%3A/g; s/\//%2F/g')
response=$(curl -s -w "\n%{http_code}" "$UPTIME_KUMA_URL?status=up&msg=$encoded_message" 2>&1)
http_code=$(echo "$response" | tail -n1)
if [ "$http_code" = "200" ] || [ "$http_code" = "201" ]; then
log_message "Healthcheck ping sent successfully (HTTP $http_code)"
else
log_message "ERROR: Failed to send healthcheck ping (HTTP $http_code)"
return 1
fi
}
# Main healthcheck logic
main() {
log_message "Starting system healthcheck for $HOSTNAME"
# Send healthcheck ping
if send_healthcheck; then
log_message "Healthcheck completed successfully"
else
log_message "ERROR: Healthcheck failed"
exit 1
fi
}
# Run main function
main
owner: root
group: root
mode: '0755'
- name: Create systemd service for system healthcheck
copy:
dest: "/etc/systemd/system/{{ systemd_service_name }}.service"
content: |
[Unit]
Description=System Healthcheck Monitor
After=network.target
[Service]
Type=oneshot
ExecStart={{ monitoring_script_path }}
User=root
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
owner: root
group: root
mode: '0644'
- name: Create systemd timer for system healthcheck
copy:
dest: "/etc/systemd/system/{{ systemd_service_name }}.timer"
content: |
[Unit]
Description=Run System Healthcheck every minute
Requires={{ systemd_service_name }}.service
[Timer]
OnBootSec=30sec
OnUnitActiveSec={{ healthcheck_interval_seconds }}sec
Persistent=true
[Install]
WantedBy=timers.target
owner: root
group: root
mode: '0644'
- name: Reload systemd daemon
systemd:
daemon_reload: yes
- name: Enable and start system healthcheck timer
systemd:
name: "{{ systemd_service_name }}.timer"
enabled: yes
state: started
- name: Test system healthcheck script
command: "{{ monitoring_script_path }}"
register: script_test
changed_when: false
- name: Verify script execution
assert:
that:
- script_test.rc == 0
fail_msg: "System healthcheck script failed to execute properly"
- name: Display monitor information
debug:
msg: |
✓ System healthcheck monitoring deployed successfully!
Monitor Name: {{ monitor_friendly_name }}
Monitor Group: {{ uptime_kuma_monitor_group }}
Healthcheck Interval: Every {{ healthcheck_interval_seconds }} seconds (1 minute)
Timeout: {{ healthcheck_timeout_seconds }} seconds (90s)
Retries: {{ healthcheck_retries }}
The system will send a heartbeat ping every minute.
Uptime Kuma will alert if no ping is received within 90 seconds (with 1 retry).
- name: Clean up temporary Uptime Kuma setup script
file:
path: /tmp/setup_uptime_kuma_healthcheck_monitor.py
state: absent
delegate_to: localhost
become: no

316
ansible/infra/430_cpu_temp_alerts.yml Normal file
View file

@ -0,0 +1,316 @@
- name: Deploy CPU Temperature Monitoring
hosts: nodito_host
become: yes
vars_files:
- ../infra_vars.yml
- ../services_config.yml
- ../infra_secrets.yml
vars:
temp_threshold_celsius: 80
temp_check_interval_minutes: 1
monitoring_script_dir: /opt/nodito-monitoring
monitoring_script_path: "{{ monitoring_script_dir }}/cpu_temp_monitor.sh"
log_file: "{{ monitoring_script_dir }}/cpu_temp_monitor.log"
systemd_service_name: nodito-cpu-temp-monitor
uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}"
ntfy_topic: "{{ service_settings.ntfy.topic }}"
tasks:
- name: Validate Uptime Kuma configuration
assert:
that:
- uptime_kuma_api_url is defined
- uptime_kuma_api_url != ""
- uptime_kuma_username is defined
- uptime_kuma_username != ""
- uptime_kuma_password is defined
- uptime_kuma_password != ""
fail_msg: "uptime_kuma_api_url, uptime_kuma_username and uptime_kuma_password must be set"
- name: Get hostname for monitor identification
command: hostname
register: host_name
changed_when: false
- name: Set monitor name and group based on hostname
set_fact:
monitor_name: "cpu-temp-{{ host_name.stdout }}"
monitor_friendly_name: "CPU Temperature: {{ host_name.stdout }}"
uptime_kuma_monitor_group: "{{ host_name.stdout }} - infra"
- name: Create Uptime Kuma CPU temperature monitor setup script
copy:
dest: /tmp/setup_uptime_kuma_cpu_temp_monitor.py
content: |
#!/usr/bin/env python3
import sys
import json
from uptime_kuma_api import UptimeKumaApi
def main():
api_url = sys.argv[1]
username = sys.argv[2]
password = sys.argv[3]
group_name = sys.argv[4]
monitor_name = sys.argv[5]
monitor_description = sys.argv[6]
interval = int(sys.argv[7])
ntfy_topic = sys.argv[8] if len(sys.argv) > 8 else "alerts"
api = UptimeKumaApi(api_url, timeout=60, wait_events=2.0)
api.login(username, password)
monitors = api.get_monitors()
notifications = api.get_notifications()
ntfy_notification = next((n for n in notifications if n.get('name') == f'ntfy ({ntfy_topic})'), None)
notification_id_list = {}
if ntfy_notification:
notification_id_list[ntfy_notification['id']] = True
group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None)
if not group:
api.add_monitor(type='group', name=group_name)
monitors = api.get_monitors()
group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None)
existing_monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
monitor_data = {
'type': 'push',
'name': monitor_name,
'parent': group['id'],
'interval': interval,
'upsideDown': True,
'description': monitor_description,
'notificationIDList': notification_id_list
}
if existing_monitor:
api.edit_monitor(existing_monitor['id'], **monitor_data)
else:
api.add_monitor(**monitor_data)
monitors = api.get_monitors()
monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
result = {
'monitor_id': monitor['id'],
'push_token': monitor['pushToken'],
'group_name': group_name,
'group_id': group['id'],
'monitor_name': monitor_name
}
print(json.dumps(result))
api.disconnect()
if __name__ == '__main__':
main()
mode: '0755'
delegate_to: localhost
become: no
- name: Run Uptime Kuma monitor setup script
command: >
{{ ansible_playbook_python }}
/tmp/setup_uptime_kuma_cpu_temp_monitor.py
"{{ uptime_kuma_api_url }}"
"{{ uptime_kuma_username }}"
"{{ uptime_kuma_password }}"
"{{ uptime_kuma_monitor_group }}"
"{{ monitor_name }}"
"{{ monitor_friendly_name }} - Alerts when temperature exceeds {{ temp_threshold_celsius }}°C"
"{{ (temp_check_interval_minutes * 60) + 60 }}"
"{{ ntfy_topic }}"
register: monitor_setup_result
delegate_to: localhost
become: no
changed_when: false
- name: Parse monitor setup result
set_fact:
monitor_info_parsed: "{{ monitor_setup_result.stdout | from_json }}"
- name: Set push URL and monitor ID as facts
set_fact:
uptime_kuma_cpu_temp_push_url: "{{ uptime_kuma_api_url }}/api/push/{{ monitor_info_parsed.push_token }}"
uptime_kuma_monitor_id: "{{ monitor_info_parsed.monitor_id }}"
- name: Install required packages for temperature monitoring
package:
name:
- lm-sensors
- curl
- jq
- bc
state: present
- name: Create monitoring script directory
file:
path: "{{ monitoring_script_dir }}"
state: directory
owner: root
group: root
mode: '0755'
- name: Create CPU temperature monitoring script
copy:
dest: "{{ monitoring_script_path }}"
content: |
#!/bin/bash
# CPU Temperature Monitoring Script
# Monitors CPU temperature and sends alerts to Uptime Kuma
LOG_FILE="{{ log_file }}"
TEMP_THRESHOLD="{{ temp_threshold_celsius }}"
UPTIME_KUMA_URL="{{ uptime_kuma_cpu_temp_push_url }}"
log_message() {
echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
}
get_cpu_temp() {
local temp=""
if command -v sensors >/dev/null 2>&1; then
temp=$(sensors 2>/dev/null | grep -E "Core 0|Package id 0|Tdie|Tctl" | head -1 | grep -oE '[0-9]+\.[0-9]+°C' | grep -oE '[0-9]+\.[0-9]+')
fi
if [ -z "$temp" ] && [ -f /sys/class/thermal/thermal_zone0/temp ]; then
temp=$(cat /sys/class/thermal/thermal_zone0/temp)
temp=$(echo "scale=1; $temp/1000" | bc -l 2>/dev/null || echo "$temp")
fi
if [ -z "$temp" ] && command -v acpi >/dev/null 2>&1; then
temp=$(acpi -t 2>/dev/null | grep -oE '[0-9]+\.[0-9]+' | head -1)
fi
echo "$temp"
}
send_uptime_kuma_alert() {
local temp="$1"
local message="CPU Temperature Alert: ${temp}°C (Threshold: ${TEMP_THRESHOLD}°C)"
log_message "ALERT: $message"
encoded_message=$(printf '%s\n' "$message" | sed 's/ /%20/g; s/°/%C2%B0/g; s/(/%28/g; s/)/%29/g; s/:/%3A/g')
response=$(curl -s -w "\n%{http_code}" "$UPTIME_KUMA_URL?status=up&msg=$encoded_message" 2>&1)
http_code=$(echo "$response" | tail -n1)
if [ "$http_code" = "200" ] || [ "$http_code" = "201" ]; then
log_message "Alert sent successfully to Uptime Kuma (HTTP $http_code)"
else
log_message "ERROR: Failed to send alert to Uptime Kuma (HTTP $http_code)"
fi
}
main() {
log_message "Starting CPU temperature check"
current_temp=$(get_cpu_temp)
if [ -z "$current_temp" ]; then
log_message "ERROR: Could not read CPU temperature"
exit 1
fi
log_message "Current CPU temperature: ${current_temp}°C"
if (( $(echo "$current_temp > $TEMP_THRESHOLD" | bc -l) )); then
log_message "WARNING: CPU temperature ${current_temp}°C exceeds threshold ${TEMP_THRESHOLD}°C"
send_uptime_kuma_alert "$current_temp"
else
log_message "CPU temperature is within normal range"
fi
}
main
owner: root
group: root
mode: '0755'
- name: Create systemd service for CPU temperature monitoring
copy:
dest: "/etc/systemd/system/{{ systemd_service_name }}.service"
content: |
[Unit]
Description=CPU Temperature Monitor
After=network.target
[Service]
Type=oneshot
ExecStart={{ monitoring_script_path }}
User=root
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
owner: root
group: root
mode: '0644'
- name: Create systemd timer for CPU temperature monitoring
copy:
dest: "/etc/systemd/system/{{ systemd_service_name }}.timer"
content: |
[Unit]
Description=Run CPU Temperature Monitor every {{ temp_check_interval_minutes }} minute(s)
Requires={{ systemd_service_name }}.service
[Timer]
OnBootSec={{ temp_check_interval_minutes }}min
OnUnitActiveSec={{ temp_check_interval_minutes }}min
Persistent=true
[Install]
WantedBy=timers.target
owner: root
group: root
mode: '0644'
- name: Reload systemd daemon
systemd:
daemon_reload: yes
- name: Enable and start CPU temperature monitoring timer
systemd:
name: "{{ systemd_service_name }}.timer"
enabled: yes
state: started
- name: Test CPU temperature monitoring script
command: "{{ monitoring_script_path }}"
register: script_test
changed_when: false
- name: Verify script execution
assert:
that:
- script_test.rc == 0
fail_msg: "CPU temperature monitoring script failed to execute properly"
- name: Display monitoring configuration
debug:
msg:
- "CPU Temperature Monitoring configured successfully"
- "Temperature threshold: {{ temp_threshold_celsius }}°C"
- "Check interval: {{ temp_check_interval_minutes }} minute(s)"
- "Monitor Name: {{ monitor_friendly_name }}"
- "Monitor Group: {{ uptime_kuma_monitor_group }}"
- "Uptime Kuma Push URL: {{ uptime_kuma_cpu_temp_push_url }}"
- "Monitoring script: {{ monitoring_script_path }}"
- "Systemd Service: {{ systemd_service_name }}.service"
- "Systemd Timer: {{ systemd_service_name }}.timer"
- name: Clean up temporary Uptime Kuma setup script
file:
path: /tmp/setup_uptime_kuma_cpu_temp_monitor.py
state: absent
delegate_to: localhost
become: no

4
ansible/infra/900_install_rsync.yml
View file

@ -1,5 +1,7 @@
- name: Install rsync - name: Install rsync
hosts: managed hosts: all
vars_files:
- ../infra_vars.yml
become: true become: true
tasks: tasks:

3
ansible/infra/910_docker_playbook.yml
View file

@ -1,5 +1,5 @@
- name: Install Docker and Docker Compose on Debian 12 - name: Install Docker and Docker Compose on Debian 12
hosts: managed hosts: all
become: yes become: yes
tasks: tasks:
@ -25,7 +25,6 @@
name: name:
- ca-certificates - ca-certificates
- curl - curl
- gnupg
state: present state: present
- name: Create directory for Docker GPG key - name: Create directory for Docker GPG key

6
ansible/infra/920_join_headscale_mesh.yml
View file

@ -1,10 +1,14 @@
- name: Join machine to headscale mesh network - name: Join machine to headscale mesh network
hosts: managed hosts: all
become: yes become: yes
vars_files:
- ../infra_vars.yml
- ../services_config.yml
vars: vars:
headscale_host_name: "spacey" headscale_host_name: "spacey"
headscale_subdomain: "{{ subdomains.headscale }}" headscale_subdomain: "{{ subdomains.headscale }}"
headscale_domain: "https://{{ headscale_subdomain }}.{{ root_domain }}" headscale_domain: "https://{{ headscale_subdomain }}.{{ root_domain }}"
headscale_namespace: "{{ service_settings.headscale.namespace }}"
tasks: tasks:
- name: Set facts for headscale server connection - name: Set facts for headscale server connection

4
ansible/infra/nodito/30_proxmox_bootstrap_playbook.yml
View file

@ -1,6 +1,8 @@
- name: Bootstrap Nodito SSH Key Access - name: Bootstrap Nodito SSH Key Access
hosts: hypervisor hosts: nodito_host
become: true become: true
vars_files:
- ../infra_vars.yml
tasks: tasks:
- name: Install sudo package - name: Install sudo package

4
ansible/infra/nodito/31_proxmox_community_repos_playbook.yml
View file

@ -1,6 +1,8 @@
- name: Switch Proxmox VE from Enterprise to Community Repositories - name: Switch Proxmox VE from Enterprise to Community Repositories
hosts: hypervisor hosts: nodito_host
become: true become: true
vars_files:
- ../infra_vars.yml
tasks: tasks:
- name: Check for deb822 sources format - name: Check for deb822 sources format

520
ansible/infra/nodito/32_zfs_pool_setup_playbook.yml
View file

@ -1,6 +1,9 @@
- name: Setup ZFS RAID 1 Pool for Proxmox Storage - name: Setup ZFS RAID 1 Pool for Proxmox Storage
hosts: hypervisor hosts: nodito_host
become: true become: true
vars_files:
- ../infra_vars.yml
- nodito_vars.yml
tasks: tasks:
- name: Verify Proxmox VE is running - name: Verify Proxmox VE is running
@ -136,18 +139,17 @@
Config file exists: {{ storage_cfg_file.stat.exists }} Config file exists: {{ storage_cfg_file.stat.exists }}
Storage check result: {{ storage_exists_check.rc }} Storage check result: {{ storage_exists_check.rc }}
Pool exists: {{ zfs_pool_exists.rc == 0 }} Pool exists: {{ zfs_pool_exists.rc == 0 }}
Will remove storage: {{ zfs_pool_exists.rc == 0 and storage_exists_check.rc == 0 }}
Will add storage: {{ zfs_pool_exists.rc == 0 and storage_exists_check.rc != 0 }} Will add storage: {{ zfs_pool_exists.rc == 0 and storage_exists_check.rc != 0 }}
# Registration is add-only on purpose. There used to be a "Remove existing - name: Remove existing storage if it exists
# storage if it exists" task here that ran `pvesm remove` whenever the command: pvesm remove {{ zfs_pool_name }}
# storage WAS present, paired with an add that only ran when it was ABSENT. register: pvesm_remove_result
# The two conditions are mutually exclusive, so a real run against a failed_when: false
# correctly-configured hypervisor removed the storage entry backing every VM when:
# and never put it back. It would also have dropped `mountpoint /var/lib/vz`, - zfs_pool_exists.rc == 0
# which the live entry has and which `pvesm add` below does not set. - storage_exists_check.rc == 0
#
# If the storage entry ever needs its options changed, edit
# /etc/pve/storage.cfg or use `pvesm set` - do not re-register it from here.
- name: Add ZFS pool storage to Proxmox using pvesm - name: Add ZFS pool storage to Proxmox using pvesm
command: > command: >
pvesm add zfspool {{ zfs_pool_name }} pvesm add zfspool {{ zfs_pool_name }}
@ -169,56 +171,498 @@
msg: "ZFS pool {{ zfs_pool_name }} is not in a healthy state" msg: "ZFS pool {{ zfs_pool_name }} is not in a healthy state"
when: "'ONLINE' not in final_zfs_status.stdout" when: "'ONLINE' not in final_zfs_status.stdout"
# ───────────────────────────────────────────────────────────────────────────── - name: Setup ZFS Pool Health Monitoring and Monthly Scrubs
# The monthly scrub. hosts: nodito
#
# The ZFS HEALTH CHECK that used to share this play is gone: it is now the
# zfs-health check in infra/400_host_monitoring.yml, which carries the same five
# conditions - pool state, device states, resilver in progress, read/write/
# checksum errors, and errors from the last scan - but reports to Gatus like
# every other host check instead of owning its own push plumbing.
#
# The scrub itself stays here, because it is not monitoring: it is the
# maintenance that gives the health check something true to report. A pool that
# is never scrubbed has no idea whether it is healthy.
# ─────────────────────────────────────────────────────────────────────────────
- name: Schedule the monthly ZFS scrub
hosts: hypervisor
become: true become: true
vars_files: vars_files:
- ../../infra_vars.yml - ../../infra_vars.yml
- ../../services_config.yml
- ../../infra_secrets.yml
- nodito_vars.yml
vars: vars:
zfs_check_interval_seconds: 86400 # 24 hours
zfs_check_timeout_seconds: 90000 # ~25 hours (interval + buffer)
zfs_check_retries: 1
zfs_monitoring_script_dir: /opt/zfs-monitoring
zfs_monitoring_script_path: "{{ zfs_monitoring_script_dir }}/zfs_health_monitor.sh"
zfs_log_file: "{{ zfs_monitoring_script_dir }}/zfs_health_monitor.log"
zfs_systemd_health_service_name: zfs-health-monitor
zfs_systemd_scrub_service_name: zfs-monthly-scrub zfs_systemd_scrub_service_name: zfs-monthly-scrub
uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}"
ntfy_topic: "{{ service_settings.ntfy.topic }}"
tasks: tasks:
- name: Validate Uptime Kuma configuration
assert:
that:
- uptime_kuma_api_url is defined
- uptime_kuma_api_url != ""
- uptime_kuma_username is defined
- uptime_kuma_username != ""
- uptime_kuma_password is defined
- uptime_kuma_password != ""
fail_msg: "uptime_kuma_api_url, uptime_kuma_username and uptime_kuma_password must be set"
- name: Get hostname for monitor identification
command: hostname
register: host_name
changed_when: false
- name: Set monitor name and group based on hostname
set_fact:
monitor_name: "zfs-health-{{ host_name.stdout }}"
monitor_friendly_name: "ZFS Pool Health: {{ host_name.stdout }}"
uptime_kuma_monitor_group: "{{ host_name.stdout }} - infra"
- name: Create Uptime Kuma ZFS health monitor setup script
copy:
dest: /tmp/setup_uptime_kuma_zfs_monitor.py
content: |
#!/usr/bin/env python3
import sys
import json
from uptime_kuma_api import UptimeKumaApi
def main():
api_url = sys.argv[1]
username = sys.argv[2]
password = sys.argv[3]
group_name = sys.argv[4]
monitor_name = sys.argv[5]
monitor_description = sys.argv[6]
interval = int(sys.argv[7])
retries = int(sys.argv[8])
ntfy_topic = sys.argv[9] if len(sys.argv) > 9 else "alerts"
api = UptimeKumaApi(api_url, timeout=120, wait_events=2.0)
api.login(username, password)
# Get all monitors
monitors = api.get_monitors()
# Get all notifications and find ntfy notification
notifications = api.get_notifications()
ntfy_notification = next((n for n in notifications if n.get('name') == f'ntfy ({ntfy_topic})'), None)
notification_id_list = {}
if ntfy_notification:
notification_id_list[ntfy_notification['id']] = True
# Find or create group
group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None)
if not group:
group_result = api.add_monitor(type='group', name=group_name)
# Refresh to get the full group object with id
monitors = api.get_monitors()
group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None)
# Find or create/update push monitor
existing_monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
monitor_data = {
'type': 'push',
'name': monitor_name,
'parent': group['id'],
'interval': interval,
'upsideDown': False, # Normal heartbeat mode: receiving pings = healthy
'maxretries': retries,
'description': monitor_description,
'notificationIDList': notification_id_list
}
if existing_monitor:
monitor = api.edit_monitor(existing_monitor['id'], **monitor_data)
# Refresh to get the full monitor object with pushToken
monitors = api.get_monitors()
monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
else:
monitor_result = api.add_monitor(**monitor_data)
# Refresh to get the full monitor object with pushToken
monitors = api.get_monitors()
monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
# Output result as JSON
result = {
'monitor_id': monitor['id'],
'push_token': monitor['pushToken'],
'group_name': group_name,
'group_id': group['id'],
'monitor_name': monitor_name
}
print(json.dumps(result))
api.disconnect()
if __name__ == '__main__':
main()
mode: '0755'
delegate_to: localhost
become: no
- name: Run Uptime Kuma ZFS monitor setup script
command: >
{{ ansible_playbook_python }}
/tmp/setup_uptime_kuma_zfs_monitor.py
"{{ uptime_kuma_api_url }}"
"{{ uptime_kuma_username }}"
"{{ uptime_kuma_password }}"
"{{ uptime_kuma_monitor_group }}"
"{{ monitor_name }}"
"{{ monitor_friendly_name }} - Daily health check for pool {{ zfs_pool_name }}"
"{{ zfs_check_timeout_seconds }}"
"{{ zfs_check_retries }}"
"{{ ntfy_topic }}"
register: monitor_setup_result
delegate_to: localhost
become: no
changed_when: false
- name: Parse monitor setup result
set_fact:
monitor_info_parsed: "{{ monitor_setup_result.stdout | from_json }}"
- name: Set push URL and monitor ID as facts
set_fact:
uptime_kuma_zfs_push_url: "{{ uptime_kuma_api_url }}/api/push/{{ monitor_info_parsed.push_token }}"
uptime_kuma_monitor_id: "{{ monitor_info_parsed.monitor_id }}"
- name: Install required packages for ZFS monitoring
package:
name:
- curl
- jq
state: present
- name: Create monitoring script directory
file:
path: "{{ zfs_monitoring_script_dir }}"
state: directory
owner: root
group: root
mode: '0755'
- name: Create ZFS health monitoring script
copy:
dest: "{{ zfs_monitoring_script_path }}"
content: |
#!/bin/bash
# ZFS Pool Health Monitoring Script
# Checks ZFS pool health using JSON output and sends heartbeat to Uptime Kuma if healthy
# If any issues detected, does NOT send heartbeat (triggers timeout alert)
LOG_FILE="{{ zfs_log_file }}"
UPTIME_KUMA_URL="{{ uptime_kuma_zfs_push_url }}"
POOL_NAME="{{ zfs_pool_name }}"
HOSTNAME=$(hostname)
# Function to log messages
log_message() {
echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
}
# Function to check pool health using JSON output
check_pool_health() {
local pool="$1"
local issues_found=0
# Get pool status as JSON
local pool_json
pool_json=$(zpool status -j "$pool" 2>&1)
if [ $? -ne 0 ]; then
log_message "ERROR: Failed to get pool status for $pool"
log_message " -> $pool_json"
return 1
fi
# Check 1: Pool state must be ONLINE
local pool_state
pool_state=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].state')
if [ "$pool_state" != "ONLINE" ]; then
log_message "ISSUE: Pool state is $pool_state (expected ONLINE)"
issues_found=1
else
log_message "OK: Pool state is ONLINE"
fi
# Check 2: Check all vdevs and devices for non-ONLINE states
local bad_states
bad_states=$(echo "$pool_json" | jq -r --arg pool "$pool" '
.pools[$pool].vdevs[] |
.. | objects |
select(.state? and .state != "ONLINE") |
"\(.name // "unknown"): \(.state)"
' 2>/dev/null)
if [ -n "$bad_states" ]; then
log_message "ISSUE: Found devices not in ONLINE state:"
echo "$bad_states" | while read -r line; do
log_message " -> $line"
done
issues_found=1
else
log_message "OK: All devices are ONLINE"
fi
# Check 3: Check for resilvering in progress
local scan_function scan_state
scan_function=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.function // "NONE"')
scan_state=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.state // "NONE"')
if [ "$scan_function" = "RESILVER" ] && [ "$scan_state" = "SCANNING" ]; then
local resilver_progress
resilver_progress=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.issued // "unknown"')
log_message "ISSUE: Pool is currently resilvering (disk reconstruction in progress) - ${resilver_progress} processed"
issues_found=1
fi
# Check 4: Check for read/write/checksum errors on all devices
# Note: ZFS JSON output has error counts as strings, so convert to numbers for comparison
local devices_with_errors
devices_with_errors=$(echo "$pool_json" | jq -r --arg pool "$pool" '
.pools[$pool].vdevs[] |
.. | objects |
select(.name? and ((.read_errors // "0" | tonumber) > 0 or (.write_errors // "0" | tonumber) > 0 or (.checksum_errors // "0" | tonumber) > 0)) |
"\(.name): read=\(.read_errors // 0) write=\(.write_errors // 0) cksum=\(.checksum_errors // 0)"
' 2>/dev/null)
if [ -n "$devices_with_errors" ]; then
log_message "ISSUE: Found devices with I/O errors:"
echo "$devices_with_errors" | while read -r line; do
log_message " -> $line"
done
issues_found=1
else
log_message "OK: No read/write/checksum errors detected"
fi
# Check 5: Check for scan errors (from last scrub/resilver)
local scan_errors
scan_errors=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.errors // "0"')
if [ "$scan_errors" != "0" ] && [ "$scan_errors" != "null" ] && [ -n "$scan_errors" ]; then
log_message "ISSUE: Last scan reported $scan_errors errors"
issues_found=1
else
log_message "OK: No scan errors"
fi
return $issues_found
}
# Function to get last scrub info for status message
get_scrub_info() {
local pool="$1"
local pool_json
pool_json=$(zpool status -j "$pool" 2>/dev/null)
local scan_func scan_state scan_start
scan_func=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.function // "NONE"')
scan_state=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.state // "NONE"')
scan_start=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.start_time // ""')
if [ "$scan_func" = "SCRUB" ] && [ "$scan_state" = "SCANNING" ]; then
echo "scrub in progress (started $scan_start)"
elif [ "$scan_func" = "SCRUB" ] && [ -n "$scan_start" ]; then
echo "last scrub: $scan_start"
else
echo "no scrub history"
fi
}
# Function to send heartbeat to Uptime Kuma
send_heartbeat() {
local message="$1"
log_message "Sending heartbeat to Uptime Kuma: $message"
# URL encode the message
local encoded_message
encoded_message=$(printf '%s\n' "$message" | sed 's/ /%20/g; s/(/%28/g; s/)/%29/g; s/:/%3A/g; s/\//%2F/g')
local response http_code
response=$(curl -s -w "\n%{http_code}" "$UPTIME_KUMA_URL?status=up&msg=$encoded_message" 2>&1)
http_code=$(echo "$response" | tail -n1)
if [ "$http_code" = "200" ] || [ "$http_code" = "201" ]; then
log_message "Heartbeat sent successfully (HTTP $http_code)"
return 0
else
log_message "ERROR: Failed to send heartbeat (HTTP $http_code)"
return 1
fi
}
# Main health check logic
main() {
log_message "=========================================="
log_message "Starting ZFS health check for pool: $POOL_NAME on $HOSTNAME"
# Run all health checks
if check_pool_health "$POOL_NAME"; then
# All checks passed - send heartbeat
local scrub_info
scrub_info=$(get_scrub_info "$POOL_NAME")
local message="Pool $POOL_NAME healthy ($scrub_info)"
send_heartbeat "$message"
log_message "Health check completed: ALL OK"
exit 0
else
# Issues found - do NOT send heartbeat (will trigger timeout alert)
log_message "Health check completed: ISSUES DETECTED - NOT sending heartbeat"
log_message "Uptime Kuma will alert after timeout due to missing heartbeat"
exit 1
fi
}
# Run main function
main
owner: root
group: root
mode: '0755'
- name: Create systemd service for ZFS health monitoring
copy:
dest: "/etc/systemd/system/{{ zfs_systemd_health_service_name }}.service"
content: |
[Unit]
Description=ZFS Pool Health Monitor
After=zfs.target network.target
[Service]
Type=oneshot
ExecStart={{ zfs_monitoring_script_path }}
User=root
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
owner: root
group: root
mode: '0644'
- name: Create systemd timer for daily ZFS health monitoring
copy:
dest: "/etc/systemd/system/{{ zfs_systemd_health_service_name }}.timer"
content: |
[Unit]
Description=Run ZFS Pool Health Monitor daily
Requires={{ zfs_systemd_health_service_name }}.service
[Timer]
OnBootSec=5min
OnUnitActiveSec={{ zfs_check_interval_seconds }}sec
Persistent=true
[Install]
WantedBy=timers.target
owner: root
group: root
mode: '0644'
- name: Create systemd service for ZFS monthly scrub - name: Create systemd service for ZFS monthly scrub
template: copy:
src: templates/zfs-monthly-scrub.service.j2
dest: "/etc/systemd/system/{{ zfs_systemd_scrub_service_name }}.service" dest: "/etc/systemd/system/{{ zfs_systemd_scrub_service_name }}.service"
content: |
[Unit]
Description=ZFS Monthly Scrub for {{ zfs_pool_name }}
After=zfs.target
[Service]
Type=oneshot
ExecStart=/sbin/zpool scrub {{ zfs_pool_name }}
User=root
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
owner: root owner: root
group: root group: root
mode: '0644' mode: '0644'
- name: Create systemd timer for monthly ZFS scrub - name: Create systemd timer for monthly ZFS scrub
template: copy:
src: templates/zfs-monthly-scrub.timer.j2
dest: "/etc/systemd/system/{{ zfs_systemd_scrub_service_name }}.timer" dest: "/etc/systemd/system/{{ zfs_systemd_scrub_service_name }}.timer"
content: |
[Unit]
Description=Run ZFS Scrub on last day of every month at 4:00 AM
Requires={{ zfs_systemd_scrub_service_name }}.service
[Timer]
OnCalendar=*-*~01 04:00:00
Persistent=true
[Install]
WantedBy=timers.target
owner: root owner: root
group: root group: root
mode: '0644' mode: '0644'
- name: Enable and start the monthly scrub timer - name: Reload systemd daemon
systemd:
daemon_reload: yes
- name: Enable and start ZFS health monitoring timer
systemd:
name: "{{ zfs_systemd_health_service_name }}.timer"
enabled: yes
state: started
- name: Enable and start ZFS monthly scrub timer
systemd: systemd:
name: "{{ zfs_systemd_scrub_service_name }}.timer" name: "{{ zfs_systemd_scrub_service_name }}.timer"
enabled: yes enabled: yes
state: started state: started
daemon_reload: yes
- name: Report the scrub schedule - name: Test ZFS health monitoring script
command: "{{ zfs_monitoring_script_path }}"
register: script_test
changed_when: false
- name: Verify script execution
assert:
that:
- script_test.rc == 0
fail_msg: "ZFS health monitoring script failed - check pool health"
- name: Display monitoring configuration
debug: debug:
msg: >- msg: |
Monthly scrub of {{ zfs_pool_name }}: ✓ ZFS Pool Health Monitoring deployed successfully!
last day of each month at 04:00.
Health is reported separately by the zfs-health check Monitor Name: {{ monitor_friendly_name }}
(infra/400_host_monitoring.yml). Monitor Group: {{ uptime_kuma_monitor_group }}
Pool Name: {{ zfs_pool_name }}
Health Check:
- Frequency: Every {{ zfs_check_interval_seconds }} seconds (24 hours)
- Timeout: {{ zfs_check_timeout_seconds }} seconds (~25 hours)
- Script: {{ zfs_monitoring_script_path }}
- Log: {{ zfs_log_file }}
- Service: {{ zfs_systemd_health_service_name }}.service
- Timer: {{ zfs_systemd_health_service_name }}.timer
Monthly Scrub:
- Schedule: Last day of month at 4:00 AM
- Service: {{ zfs_systemd_scrub_service_name }}.service
- Timer: {{ zfs_systemd_scrub_service_name }}.timer
Conditions monitored:
- Pool state (must be ONLINE)
- Device states (no DEGRADED/FAULTED/OFFLINE/UNAVAIL)
- Resilver status (alerts if resilvering)
- Read/Write/Checksum errors
- Scrub errors
- name: Clean up temporary Uptime Kuma setup script
file:
path: /tmp/setup_uptime_kuma_zfs_monitor.py
state: absent
delegate_to: localhost
become: no

5
ansible/infra/nodito/33_proxmox_debian_cloud_template.yml
View file

@ -1,6 +1,9 @@
- name: Create Proxmox template from Debian cloud image (no VM clone) - name: Create Proxmox template from Debian cloud image (no VM clone)
hosts: hypervisor hosts: nodito_host
become: true become: true
vars_files:
- ../../infra_vars.yml
- nodito_vars.yml
vars: vars:
# Defaults (override via vars_files or --extra-vars as needed) # Defaults (override via vars_files or --extra-vars as needed)

422
ansible/infra/nodito/34_nut_ups_setup_playbook.yml
View file

@ -1,33 +1,12 @@
- name: Setup NUT (Network UPS Tools) for CyberPower UPS - name: Setup NUT (Network UPS Tools) for CyberPower UPS
hosts: hypervisor hosts: nodito_host
become: true become: true
vars_files:
- ../../infra_vars.yml
- nodito_vars.yml
- nodito_secrets.yml
tasks: tasks:
# ------------------------------------------------------------------
# Safety catch
#
# /etc/nut/upsd.users and /etc/nut/upsmon.conf on nodito were written by
# hand in January 2026 and carry a working password. host_vars/nodito/vault.yml
# (formerly infra/nodito/nodito_secrets.yml) still holds the literal string
# CHANGE_ME_TO_SECURE_PASSWORD, so running this play would overwrite that
# working pair with a placeholder and restart NUT - leaving the hypervisor's
# UPS unmonitored and unable to trigger a clean shutdown on mains loss.
#
# Until the real password is put in the vault, stop here.
# ansible-vault edit host_vars/nodito/vault.yml
# ------------------------------------------------------------------
- name: Refuse to run with a placeholder UPS password
assert:
that:
- ups_password is defined
- ups_password | length > 0
- ups_password != "CHANGE_ME_TO_SECURE_PASSWORD"
fail_msg: >-
ups_password is unset or still the placeholder. Applying this play would
overwrite the working /etc/nut/upsd.users and /etc/nut/upsmon.conf on
nodito and restart NUT. Put the real password in the vault first:
ansible-vault edit host_vars/nodito/vault.yml
# ------------------------------------------------------------------ # ------------------------------------------------------------------
# Installation # Installation
# ------------------------------------------------------------------ # ------------------------------------------------------------------
@ -96,45 +75,90 @@
# Configuration files # Configuration files
# ------------------------------------------------------------------ # ------------------------------------------------------------------
- name: Configure NUT mode (standalone) - name: Configure NUT mode (standalone)
template: copy:
dest: /etc/nut/nut.conf dest: /etc/nut/nut.conf
src: templates/nut.conf.j2 content: |
# Managed by Ansible
MODE=standalone
owner: root owner: root
group: nut group: nut
mode: "0640" mode: "0640"
notify: Restart NUT services notify: Restart NUT services
- name: Configure UPS device - name: Configure UPS device
template: copy:
dest: /etc/nut/ups.conf dest: /etc/nut/ups.conf
src: templates/ups.conf.j2 content: |
# Managed by Ansible
[{{ ups_name }}]
driver = {{ ups_driver }}
port = {{ ups_port }}
desc = "{{ ups_desc }}"
offdelay = {{ ups_offdelay }}
ondelay = {{ ups_ondelay }}
owner: root owner: root
group: nut group: nut
mode: "0640" mode: "0640"
notify: Restart NUT services notify: Restart NUT services
- name: Configure upsd to listen on localhost - name: Configure upsd to listen on localhost
template: copy:
dest: /etc/nut/upsd.conf dest: /etc/nut/upsd.conf
src: templates/upsd.conf.j2 content: |
# Managed by Ansible
LISTEN 127.0.0.1 3493
owner: root owner: root
group: nut group: nut
mode: "0640" mode: "0640"
notify: Restart NUT services notify: Restart NUT services
- name: Configure upsd users - name: Configure upsd users
template: copy:
dest: /etc/nut/upsd.users dest: /etc/nut/upsd.users
src: templates/upsd.users.j2 content: |
# Managed by Ansible
[{{ ups_user }}]
password = {{ ups_password }}
upsmon master
owner: root owner: root
group: nut group: nut
mode: "0640" mode: "0640"
notify: Restart NUT services notify: Restart NUT services
- name: Configure upsmon - name: Configure upsmon
template: copy:
dest: /etc/nut/upsmon.conf dest: /etc/nut/upsmon.conf
src: templates/upsmon.conf.j2 content: |
# Managed by Ansible
MONITOR {{ ups_name }}@localhost 1 {{ ups_user }} {{ ups_password }} master
MINSUPPLIES 1
SHUTDOWNCMD "/sbin/shutdown -h +0"
POLLFREQ 5
POLLFREQALERT 5
HOSTSYNC 15
DEADTIME 15
POWERDOWNFLAG /etc/killpower
# Notifications
NOTIFYMSG ONLINE "UPS %s on line power"
NOTIFYMSG ONBATT "UPS %s on battery"
NOTIFYMSG LOWBATT "UPS %s battery is low"
NOTIFYMSG FSD "UPS %s: forced shutdown in progress"
NOTIFYMSG COMMOK "Communications with UPS %s established"
NOTIFYMSG COMMBAD "Communications with UPS %s lost"
NOTIFYMSG SHUTDOWN "Auto logout and shutdown proceeding"
NOTIFYMSG REPLBATT "UPS %s battery needs replacing"
# Log all events to syslog
NOTIFYFLAG ONLINE SYSLOG
NOTIFYFLAG ONBATT SYSLOG
NOTIFYFLAG LOWBATT SYSLOG
NOTIFYFLAG FSD SYSLOG
NOTIFYFLAG COMMOK SYSLOG
NOTIFYFLAG COMMBAD SYSLOG
NOTIFYFLAG SHUTDOWN SYSLOG
NOTIFYFLAG REPLBATT SYSLOG
owner: root owner: root
group: nut group: nut
mode: "0640" mode: "0640"
@ -225,11 +249,321 @@
- nut-server - nut-server
- nut-monitor - nut-monitor
# The UPS heartbeat play that used to live here is gone. What it deployed -
# /opt/ups-monitoring plus a ups-heartbeat timer - is now the ups-status check - name: Setup UPS Heartbeat Monitoring with Uptime Kuma
# in infra/400_host_monitoring.yml, which reports to Gatus like every other hosts: nodito
# host check instead of carrying its own push plumbing. become: true
# vars_files:
# This playbook is now purely NUT setup: the driver, upsd, upsmon and the - ../../infra_vars.yml
# shutdown behaviour. Monitoring whether the UPS is on mains is a separate - ../../services_config.yml
# concern and belongs with the other host checks. - ../../infra_secrets.yml
- nodito_vars.yml
- nodito_secrets.yml
vars:
ups_heartbeat_interval_seconds: 60
ups_heartbeat_timeout_seconds: 120
ups_heartbeat_retries: 1
ups_monitoring_script_dir: /opt/ups-monitoring
ups_monitoring_script_path: "{{ ups_monitoring_script_dir }}/ups_heartbeat.sh"
ups_log_file: "{{ ups_monitoring_script_dir }}/ups_heartbeat.log"
ups_systemd_service_name: ups-heartbeat
uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}"
ntfy_topic: "{{ service_settings.ntfy.topic }}"
tasks:
- name: Validate Uptime Kuma configuration
assert:
that:
- uptime_kuma_api_url is defined
- uptime_kuma_api_url != ""
- uptime_kuma_username is defined
- uptime_kuma_username != ""
- uptime_kuma_password is defined
- uptime_kuma_password != ""
fail_msg: "uptime_kuma_api_url, uptime_kuma_username and uptime_kuma_password must be set"
- name: Get hostname for monitor identification
command: hostname
register: host_name
changed_when: false
- name: Set monitor name and group based on hostname
set_fact:
monitor_name: "ups-{{ host_name.stdout }}"
monitor_friendly_name: "UPS Status: {{ host_name.stdout }}"
uptime_kuma_monitor_group: "{{ host_name.stdout }} - infra"
- name: Create Uptime Kuma UPS monitor setup script
copy:
dest: /tmp/setup_uptime_kuma_ups_monitor.py
content: |
#!/usr/bin/env python3
import sys
import json
from uptime_kuma_api import UptimeKumaApi
def main():
api_url = sys.argv[1]
username = sys.argv[2]
password = sys.argv[3]
group_name = sys.argv[4]
monitor_name = sys.argv[5]
monitor_description = sys.argv[6]
interval = int(sys.argv[7])
retries = int(sys.argv[8])
ntfy_topic = sys.argv[9] if len(sys.argv) > 9 else "alerts"
api = UptimeKumaApi(api_url, timeout=120, wait_events=2.0)
api.login(username, password)
monitors = api.get_monitors()
notifications = api.get_notifications()
ntfy_notification = next((n for n in notifications if n.get('name') == f'ntfy ({ntfy_topic})'), None)
notification_id_list = {}
if ntfy_notification:
notification_id_list[ntfy_notification['id']] = True
group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None)
if not group:
api.add_monitor(type='group', name=group_name)
monitors = api.get_monitors()
group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None)
existing_monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
monitor_data = {
'type': 'push',
'name': monitor_name,
'parent': group['id'],
'interval': interval,
'upsideDown': False, # Normal heartbeat mode: receiving pings = healthy
'maxretries': retries,
'description': monitor_description,
'notificationIDList': notification_id_list
}
if existing_monitor:
api.edit_monitor(existing_monitor['id'], **monitor_data)
monitors = api.get_monitors()
monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
else:
api.add_monitor(**monitor_data)
monitors = api.get_monitors()
monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
result = {
'monitor_id': monitor['id'],
'push_token': monitor['pushToken'],
'group_name': group_name,
'group_id': group['id'],
'monitor_name': monitor_name
}
print(json.dumps(result))
api.disconnect()
if __name__ == '__main__':
main()
mode: '0755'
delegate_to: localhost
become: no
- name: Run Uptime Kuma UPS monitor setup script
command: >
{{ ansible_playbook_python }}
/tmp/setup_uptime_kuma_ups_monitor.py
"{{ uptime_kuma_api_url }}"
"{{ uptime_kuma_username }}"
"{{ uptime_kuma_password }}"
"{{ uptime_kuma_monitor_group }}"
"{{ monitor_name }}"
"{{ monitor_friendly_name }} - Alerts when UPS goes on battery or loses communication"
"{{ ups_heartbeat_timeout_seconds }}"
"{{ ups_heartbeat_retries }}"
"{{ ntfy_topic }}"
register: monitor_setup_result
delegate_to: localhost
become: no
changed_when: false
- name: Parse monitor setup result
set_fact:
monitor_info_parsed: "{{ monitor_setup_result.stdout | from_json }}"
- name: Set push URL as fact
set_fact:
uptime_kuma_ups_push_url: "{{ uptime_kuma_api_url }}/api/push/{{ monitor_info_parsed.push_token }}"
- name: Install required packages for UPS monitoring
package:
name:
- curl
state: present
- name: Create monitoring script directory
file:
path: "{{ ups_monitoring_script_dir }}"
state: directory
owner: root
group: root
mode: '0755'
- name: Create UPS heartbeat monitoring script
copy:
dest: "{{ ups_monitoring_script_path }}"
content: |
#!/bin/bash
# UPS Heartbeat Monitoring Script
# Sends heartbeat to Uptime Kuma only when UPS is on mains power
# When on battery or communication lost, no heartbeat is sent (triggers timeout alert)
LOG_FILE="{{ ups_log_file }}"
UPTIME_KUMA_URL="{{ uptime_kuma_ups_push_url }}"
UPS_NAME="{{ ups_name }}"
log_message() {
echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
}
send_heartbeat() {
local message="$1"
local encoded_message
encoded_message=$(printf '%s\n' "$message" | sed 's/ /%20/g; s/(/%28/g; s/)/%29/g; s/:/%3A/g; s/\//%2F/g; s/%/%25/g')
local response http_code
response=$(curl -s -w "\n%{http_code}" "$UPTIME_KUMA_URL?status=up&msg=$encoded_message" 2>&1)
http_code=$(echo "$response" | tail -n1)
if [ "$http_code" = "200" ] || [ "$http_code" = "201" ]; then
log_message "Heartbeat sent: $message (HTTP $http_code)"
return 0
else
log_message "ERROR: Failed to send heartbeat (HTTP $http_code)"
return 1
fi
}
main() {
local status charge runtime load
status=$(upsc ${UPS_NAME}@localhost ups.status 2>/dev/null)
if [ -z "$status" ]; then
log_message "ERROR: Cannot communicate with UPS - NOT sending heartbeat"
exit 1
fi
charge=$(upsc ${UPS_NAME}@localhost battery.charge 2>/dev/null)
runtime=$(upsc ${UPS_NAME}@localhost battery.runtime 2>/dev/null)
load=$(upsc ${UPS_NAME}@localhost ups.load 2>/dev/null)
if [[ "$status" == *"OL"* ]]; then
local message="UPS on mains (charge=${charge}% runtime=${runtime}s load=${load}%)"
send_heartbeat "$message"
exit 0
else
log_message "UPS not on mains power (status=$status) - NOT sending heartbeat"
exit 1
fi
}
main
owner: root
group: root
mode: '0755'
- name: Create systemd service for UPS heartbeat
copy:
dest: "/etc/systemd/system/{{ ups_systemd_service_name }}.service"
content: |
[Unit]
Description=UPS Heartbeat Monitor
After=network.target nut-monitor.service
[Service]
Type=oneshot
ExecStart={{ ups_monitoring_script_path }}
User=root
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
owner: root
group: root
mode: '0644'
- name: Create systemd timer for UPS heartbeat
copy:
dest: "/etc/systemd/system/{{ ups_systemd_service_name }}.timer"
content: |
[Unit]
Description=Run UPS Heartbeat Monitor every {{ ups_heartbeat_interval_seconds }} seconds
Requires={{ ups_systemd_service_name }}.service
[Timer]
OnBootSec=1min
OnUnitActiveSec={{ ups_heartbeat_interval_seconds }}sec
Persistent=true
[Install]
WantedBy=timers.target
owner: root
group: root
mode: '0644'
- name: Reload systemd daemon
systemd:
daemon_reload: yes
- name: Enable and start UPS heartbeat timer
systemd:
name: "{{ ups_systemd_service_name }}.timer"
enabled: yes
state: started
- name: Test UPS heartbeat script
command: "{{ ups_monitoring_script_path }}"
register: script_test
changed_when: false
- name: Verify script execution
assert:
that:
- script_test.rc == 0
fail_msg: "UPS heartbeat script failed - check UPS status and communication"
- name: Display monitoring configuration
debug:
msg:
- "UPS Monitoring configured successfully"
- ""
- "NUT Configuration:"
- " UPS Name: {{ ups_name }}"
- " UPS Description: {{ ups_desc }}"
- " Off Delay: {{ ups_offdelay }}s (time after shutdown before UPS cuts power)"
- " On Delay: {{ ups_ondelay }}s (time after mains returns before UPS restores power)"
- ""
- "Uptime Kuma Monitoring:"
- " Monitor Name: {{ monitor_friendly_name }}"
- " Monitor Group: {{ uptime_kuma_monitor_group }}"
- " Push URL: {{ uptime_kuma_ups_push_url }}"
- " Heartbeat Interval: {{ ups_heartbeat_interval_seconds }}s"
- " Timeout: {{ ups_heartbeat_timeout_seconds }}s"
- ""
- "Scripts and Services:"
- " Script: {{ ups_monitoring_script_path }}"
- " Log: {{ ups_log_file }}"
- " Service: {{ ups_systemd_service_name }}.service"
- " Timer: {{ ups_systemd_service_name }}.timer"
- name: Clean up temporary Uptime Kuma setup script
file:
path: /tmp/setup_uptime_kuma_ups_monitor.py
state: absent
delegate_to: localhost
become: no

19
ansible/host_vars/nodito/main.yml → ansible/infra/nodito/nodito_vars.yml
View file

@ -14,12 +14,7 @@ systemd_service_name: nodito-cpu-temp-monitor
# ZFS Pool Configuration # ZFS Pool Configuration
zfs_pool_name: "proxmox-tank-1" zfs_pool_name: "proxmox-tank-1"
# Corrected 2026-09-13: this said WX11TN0Z, a disk that is no longer in the zfs_disk_1: "/dev/disk/by-id/ata-ST4000NT001-3M2101_WX11TN0Z" # First disk for RAID 1 mirror
# machine. The live mirror is WX120LHQ + WX11TN2P - a leg was evidently
# replaced and the repo never caught up. Pool creation is guarded by
# `when: zfs_pool_exists.rc != 0` so it was inert, but it would have been
# wrong on any disaster-recovery run.
zfs_disk_1: "/dev/disk/by-id/ata-ST4000NT001-3M2101_WX120LHQ" # First disk for RAID 1 mirror
zfs_disk_2: "/dev/disk/by-id/ata-ST4000NT001-3M2101_WX11TN2P" # Second disk for RAID 1 mirror zfs_disk_2: "/dev/disk/by-id/ata-ST4000NT001-3M2101_WX11TN2P" # Second disk for RAID 1 mirror
zfs_pool_mountpoint: "/var/lib/vz" zfs_pool_mountpoint: "/var/lib/vz"
@ -31,15 +26,3 @@ ups_port: auto
ups_user: counterweight ups_user: counterweight
ups_offdelay: 120 # Seconds after shutdown before UPS cuts outlet power ups_offdelay: 120 # Seconds after shutdown before UPS cuts outlet power
ups_ondelay: 30 # Seconds after mains returns before UPS restores outlet power ups_ondelay: 30 # Seconds after mains returns before UPS restores outlet power
# Systemd services deployed on this host, monitored every 5 minutes.
#
# The fact lives with the machine rather than in a central map, for the same
# reason the cross-host ports do: "what runs here" is a property of the host,
# and a central list is one more thing to forget to update when a service moves.
#
# Only units WE deploy belong here. Distro units (ssh, cron) have their own
# supervision and would be noise.
monitored_services:
- nut-server
- nut-monitor

2
ansible/infra/nodito/templates/nut.conf.j2
View file

@ -1,2 +0,0 @@
# Managed by Ansible
MODE=standalone

9
ansible/infra/nodito/templates/ups.conf.j2
View file

@ -1,9 +0,0 @@
# Managed by Ansible
maxretry = 3
[{{ ups_name }}]
driver = {{ ups_driver }}
port = {{ ups_port }}
desc = "{{ ups_desc }}"
offdelay = {{ ups_offdelay }}
ondelay = {{ ups_ondelay }}

2
ansible/infra/nodito/templates/upsd.conf.j2
View file

@ -1,2 +0,0 @@
# Managed by Ansible
LISTEN 127.0.0.1 3493

4
ansible/infra/nodito/templates/upsd.users.j2
View file

@ -1,4 +0,0 @@
# Managed by Ansible
[{{ ups_user }}]
password = {{ ups_password }}
upsmon master

34
ansible/infra/nodito/templates/upsmon.conf.j2
View file

@ -1,34 +0,0 @@
# Managed by Ansible
MONITOR {{ ups_name }}@localhost 1 {{ ups_user }} {{ ups_password }} master
MINSUPPLIES 1
SHUTDOWNCMD "/sbin/shutdown -h +0"
POLLFREQ 5
POLLFREQALERT 5
HOSTSYNC 15
DEADTIME 15
POWERDOWNFLAG "/etc/killpower"
OFFDURATION 30
RBWARNTIME 43200
NOCOMMWARNTIME 300
FINALDELAY 5
# Notifications
NOTIFYMSG ONLINE "UPS %s on line power"
NOTIFYMSG ONBATT "UPS %s on battery"
NOTIFYMSG LOWBATT "UPS %s battery is low"
NOTIFYMSG FSD "UPS %s: forced shutdown in progress"
NOTIFYMSG COMMOK "Communications with UPS %s established"
NOTIFYMSG COMMBAD "Communications with UPS %s lost"
NOTIFYMSG SHUTDOWN "Auto logout and shutdown proceeding"
NOTIFYMSG REPLBATT "UPS %s battery needs replacing"
# Log all events to syslog
NOTIFYFLAG ONLINE SYSLOG
NOTIFYFLAG ONBATT SYSLOG
NOTIFYFLAG LOWBATT SYSLOG
NOTIFYFLAG FSD SYSLOG
NOTIFYFLAG COMMOK SYSLOG
NOTIFYFLAG COMMBAD SYSLOG
NOTIFYFLAG SHUTDOWN SYSLOG
NOTIFYFLAG REPLBATT SYSLOG

13
ansible/infra/nodito/templates/zfs-monthly-scrub.service.j2
View file

@ -1,13 +0,0 @@
[Unit]
Description=ZFS Monthly Scrub for {{ zfs_pool_name }}
After=zfs.target
[Service]
Type=oneshot
ExecStart=/sbin/zpool scrub {{ zfs_pool_name }}
User=root
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target

10
ansible/infra/nodito/templates/zfs-monthly-scrub.timer.j2
View file

@ -1,10 +0,0 @@
[Unit]
Description=Run ZFS Scrub on last day of every month at 4:00 AM
Requires={{ zfs_systemd_scrub_service_name }}.service
[Timer]
OnCalendar=*-*~01 04:00:00
Persistent=true
[Install]
WantedBy=timers.target

28
ansible/infra_secrets.yml.example Normal file
View file

@ -0,0 +1,28 @@
# Uptime Kuma login credentials
# Used by the disk monitoring playbook to create monitors automatically
uptime_kuma_username: "admin"
uptime_kuma_password: "your_password_here"
# ntfy credentials
# Used for notification channel setup in Uptime Kuma
ntfy_username: "your_ntfy_username"
ntfy_password: "your_ntfy_password"
# headscale-ui credentials
# Used for HTTP basic authentication via Caddy
# Provide either:
# - headscale_ui_password: plain text password (will be hashed automatically)
# - headscale_ui_password_hash: pre-hashed bcrypt password (more secure, use caddy hash-password to generate)
headscale_ui_username: "admin"
headscale_ui_password: "your_secure_password_here"
# headscale_ui_password_hash: "$2a$14$..." # Optional: pre-hashed password
bitcoin_rpc_user: "bitcoinrpc"
bitcoin_rpc_password: "CHANGE_ME_TO_SECURE_PASSWORD"
# Mempool MariaDB credentials
# Used by: services/mempool/deploy_mempool_playbook.yml
mariadb_mempool_password: "CHANGE_ME_TO_SECURE_PASSWORD"

4
ansible/infra_vars.yml Normal file
View file

@ -0,0 +1,4 @@
new_user: counterweight
ssh_port: 22
allow_ssh_from: "any"
root_domain: contrapeso.xyz

88
ansible/inventory.ini
View file

@ -1,88 +0,0 @@
[vps]
vipy ansible_host=167.172.98.204 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
spacey ansible_host=46.101.250.26 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
[nodito_host]
nodito ansible_host=192.168.1.139 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
# Requires the tailnet to be up on the control node.
[nodito_vms]
knots_box_local ansible_host=knots-box lan_ip=192.168.1.135 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
fulcrum_box_local ansible_host=fulcrum-box lan_ip=192.168.1.140 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
mempool_box_local ansible_host=mempool-box lan_ip=192.168.1.142 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
memos_box_local ansible_host=memos-box lan_ip=192.168.1.130 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
forgejo_runner_local ansible_host=forgejo-runner-box lan_ip=192.168.1.132 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
arbret_staging_local ansible_host=arbret-staging-box lan_ip=192.168.1.145 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
small_backups_local ansible_host=small-backups-box lan_ip=192.168.1.131 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
monerod_box_local ansible_host=monerod-box lan_ip=192.168.1.146 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
# lan_ip read off nodito's ARP table for the VM's own MAC (bc:24:11:89:fc:1f),
# not guessed. Note .149 previously belonged to another machine - its stale host
# key was still in known_hosts and SSH refused the connection, which is the
# designed behaviour described in group_vars/nodito_vms.yml.
music_box_local ansible_host=music-box lan_ip=192.168.1.149 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
# lan_ip read from the Proxmox guest agent for VM 3300 (qm guest cmd
# network-get-interfaces), and the host key checked against the one the agent
# reads off the disk before it went into known_hosts.
jar_lid_staging_local ansible_host=jar-lid-archive-staging-box lan_ip=192.168.1.151 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua
# Local connection to laptop: this assumes you're running ansible commands from your personal laptop
[lapy]
localhost ansible_connection=local ansible_user=counterweight
[edge]
vipy
# Gatus and its Signal transport. Shares spacey with headscale since the
# 2026-10-17 rebuild; the separate `monitoring` droplet is gone.
[observability]
spacey
[vpn_control]
spacey
[hypervisor]
nodito
[bitcoin]
knots_box_local
[electrum]
fulcrum_box_local
[mempool]
mempool_box_local
[monero]
monerod_box_local
[memos]
memos_box_local
[music]
music_box_local
# Staging only for now; a production host joins this group with its own host_vars.
[jar_lid_archive]
jar_lid_staging_local
[ci_runner]
forgejo_runner_local
[control]
localhost
# Every machine Ansible may configure as a server.
# Deliberately EXCLUDES [control] (your laptop).
[managed:children]
vps
nodito_host
nodito_vms
# Hosts that run Caddy and therefore have /etc/caddy/sites-enabled.
[caddy:children]
edge
observability
vpn_control
[backup_store]
small_backups_local

16
ansible/inventory.ini.example Normal file
View file

@ -0,0 +1,16 @@
[vps]
vipy ansible_host=your.services.vps.ip ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/your-key
watchtower ansible_host=your.monitoring.vps.ip ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/your-key
spacey ansible_host=your.headscale.vps.ip ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/your-key
[nodito_host]
nodito ansible_host=your.proxmox.ip.here ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/your-key ansible_ssh_pass=your_root_password
[nodito_vms]
# Example node, replace with your VM names and addresses
# memos_box ansible_host=192.168.1.150 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/your-key
# Local connection to laptop: this assumes you're running ansible commands from your personal laptop
# Make sure to adjust the username
[lapy]
localhost ansible_connection=local ansible_user=your laptop user gpg_recipient=your_email@example.com gpg_key_id=your_gpg_key_id_here

115
ansible/playbooks/backups.yml
View file

@ -1,115 +0,0 @@
- name: Configure the offsite backup pull
hosts: backup_store
gather_facts: yes
tasks:
- name: Ensure the box pulls every source on a timer
ansible.builtin.include_role:
name: backup_store
vars:
# check-backups.sh reports one result per source plus one for the store
# itself, so it needs the collection URL and appends each key.
backup_store_check_push_base: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints"
backup_store_check_push_token: "{{ gatus_push_tokens[inventory_hostname] }}"
backup_store_sources:
# On vipy since 2026-10-17 (prd-arbret is gone). The arbret repo writes
# the dump to the backup_source contract - see arbret_handoff.md.
- name: arbret
source: "backup-pull@prd-vipy:/opt/backups/arbret/"
retention_days: 90
- name: headscale
source: "backup-pull@headscale.contrapeso.xyz:/opt/backups/headscale/"
retention_days: 90
- name: memos
source: "backup-pull@memos-box:/opt/backups/memos/"
retention_days: 90
- name: vaultwarden
source: "backup-pull@prd-vipy:/opt/backups/vaultwarden/"
retention_days: 90
- name: lnbits
source: "backup-pull@prd-vipy:/opt/backups/lnbits/"
retention_days: 90
- name: forgejo
source: "backup-pull@prd-vipy:/opt/backups/forgejo/"
retention_days: 14
# Staging: disposable data, so the long tail is as short as the local one.
- name: "{{ hostvars['jar_lid_staging_local'].jar_lid_archive_instance }}"
source: "backup-pull@{{ hostvars['jar_lid_staging_local'].ansible_host }}:/opt/backups/{{ hostvars['jar_lid_staging_local'].jar_lid_archive_instance }}/"
retention_days: "{{ hostvars['jar_lid_staging_local'].jar_lid_archive_backup_retention_days }}"
# ─────────────────────────────────────────────────────────────────────────────
# Register the backup checks with Gatus.
#
# Two groups on purpose, because they answer different questions and fail for
# different reasons:
#
# backup-dump did the SOURCE produce an artefact? Pushed by each dump right
# after it runs, so a broken dump is visible within minutes.
# backup-store did it ARRIVE, is it fresh, non-zero, plausibly sized, and is
# retention pruning? Pushed by check-backups.sh at 05:30.
#
# The store alone could catch almost everything, because the artefact filename
# carries the source's dump timestamp - a source whose timer died still pulls
# "ok" forever, but the timestamp gives it away. What the source side adds is
# LATENCY and DIAGNOSIS: the store only learns at the next 04:00 pull, and it
# cannot tell you whether the dump broke or the pull did.
#
# arbret's dump is written by the arbret repo, not by backup_source, and it
# pushes with its own `arbret` token (the `host` below is the token key).
# ─────────────────────────────────────────────────────────────────────────────
- name: Register the backup checks with Gatus
hosts: observability
become: yes
vars:
# Sources we deploy the dump for, and the host each one runs on.
dump_sources:
- {name: headscale, host: spacey}
- {name: memos, host: memos_box_local}
- {name: vaultwarden, host: vipy}
- {name: lnbits, host: vipy}
- {name: forgejo, host: vipy}
- {name: jar-lid-archive-staging, host: jar_lid_staging_local}
- {name: arbret, host: arbret}
store_sources: [arbret, headscale, memos, vaultwarden, lnbits, forgejo, jar-lid-archive-staging]
tasks:
# 26h, not 7h: the DUMP is genuinely daily, so the window cannot be tighter
# than a day plus slack. The store-side check catches the same fault within
# 6h by reading the artefact's dump timestamp out of the filename, so this is
# the slow backstop rather than the primary signal.
- name: Build the dump endpoint list
ansible.builtin.set_fact:
dump_endpoints: "{{ dump_endpoints | default([]) + [{
'name': item.name,
'group': 'backup-dump',
'token': gatus_push_tokens[item.host],
'heartbeat': '26h'}] }}"
loop: "{{ dump_sources }}"
- name: Build the store endpoint list
ansible.builtin.set_fact:
store_endpoints: "{{ store_endpoints | default([]) + [{
'name': item,
'group': 'backup-store',
'token': gatus_push_tokens['small_backups_local'],
'heartbeat': '7h'}] }}"
loop: "{{ store_sources }}"
- name: Register the backup endpoints
ansible.builtin.include_role:
name: gatus_endpoint
vars:
# Push endpoints: the heartbeat window is the tolerance, so alert on
# the first failure rather than waiting for three 7h windows to pass.
gatus_endpoint_default_alerts:
- type: signal
failure-threshold: 1
success-threshold: 2
send-on-resolved: true
minimum-reminder-interval: 12h
gatus_endpoint_name: backups
gatus_endpoint_external: "{{ dump_endpoints + store_endpoints + [{
'name': 'pull job',
'group': 'backup-store',
'token': gatus_push_tokens['small_backups_local'],
'heartbeat': '7h'}] }}"

13
ansible/requirements.yml
View file

@ -1,10 +1,11 @@
---
# Ansible Galaxy Collections Requirements # Ansible Galaxy Collections Requirements
# Install with: ansible-galaxy collection install -r requirements.yml # Install with: ansible-galaxy collection install -r requirements.yml
# No collections are currently required. collections:
# # Uptime Kuma Ansible Collection
# lucasheld.uptime_kuma was pinned here but never used — every monitor was created # Used by: infra/41_disk_usage_alerts.yml
# by hand-rolled Python instead. Removed 2026-09-11 along with Uptime Kuma itself. # Provides modules to manage Uptime Kuma monitors programmatically
# See archive/uptime_kuma/. - name: lucasheld.uptime_kuma
version: ">=1.0.0"
collections: []

126
ansible/roles/backup_source/README.md
View file

@ -1,126 +0,0 @@
# `backup_source`
Makes a host back **itself** up: dump to stdout, encrypt with `age`, write to a
local directory, prune, on a systemd timer. `small-backups-box` pulls the
directory later (see `backup_store`).
Modelled on `prd-arbret`, which has been doing exactly this correctly since
before the rest of the estate was migrated.
## Usage
```yaml
- ansible.builtin.include_role:
name: backup_source
vars:
backup_source_name: headscale
backup_source_description: "Headscale"
backup_source_dump_command: "tar -czf - -C / var/lib/headscale etc/headscale"
backup_source_stop_service: headscale
backup_source_retention_days: 7
```
Produces `/opt/backups/headscale/headscale_<YYYYmmdd_HHMMSS>.tar.gz.age`,
`headscale-backup.{service,timer}`, and `/usr/local/bin/headscale-backup.sh`.
## Why the source encrypts, not the destination
`age -r <recipient>` is asymmetric and the host holds only the **public** key, so
a compromised host cannot read its own backups — or anyone else's. The scripts
this replaces encrypted with GPG *on the laptop, after the data had already
crossed the network*, which protects the artefact at rest but not in transit.
The matching identity lives only on lapy and is escrowed. **Lose it and every
artefact everywhere becomes noise**, including arbret's.
## `backup_source_dump_command` writes to STDOUT
The role pipes it into `age`, so plaintext never touches the disk. Use `-C /`
with relative paths in `tar` rather than absolute ones: it avoids tar's "removing
leading /" and makes the restore target explicit.
## Services that are not systemd
`backup_source_stop_service` runs `systemctl stop/start`. For anything else,
give the pair explicitly — vaultwarden is a docker compose stack, so
`systemctl stop vaultwarden` silently does nothing:
```yaml
backup_source_stop_command: "docker compose -f /opt/vaultwarden/docker-compose.yml stop"
backup_source_start_command: "docker compose -f /opt/vaultwarden/docker-compose.yml start"
```
The same EXIT trap wraps both forms. The assert refuses a stop command without a
matching start command, because that combination fails in the one way you would
not notice: the service stops and never comes back.
## More than one thing to back up
`tar` takes several paths, so multiple files or directories are normally **one**
artefact — headscale captures `/var/lib/headscale` and `/etc/headscale` together,
lnbits captures its data directory and its `.env`.
Prefer one artefact. A backup should be a consistent snapshot, and two artefacts
written by two runs can drift — you can end up restoring an `.env` that does not
match the database it configures. Pulling a single file back out needs no
unpacking:
```bash
age -d -i <identity> <artefact> | tar -xzO opt/lnbits/lnbits/.env
```
If you genuinely need separate artefacts, call the role twice with different
`backup_source_name`s rather than extending it — but only one call may set
`backup_source_stop_service`, or the service is stopped twice per night.
The case this shape cannot express is a **database dump plus a file tree**
(`pg_dump` and a media directory, say): you cannot merge those into one stream
without staging plaintext on disk, which is exactly what this design avoids.
None of the current services need it — all are file trees, all stopped for the
dump. A future one that does should use two role calls.
## Everything here is sqlite, so everything stops
All five services are sqlite-backed, several in WAL mode (`-wal`/`-shm` files
present). A live copy of a WAL-mode database can be torn or stale, so each is
stopped for the duration. Measured downtime: under a second for headscale and
memos, ~6 s vaultwarden, ~11 s lnbits, and **2m36s for forgejo** — 2.7 G of repos
and database. That last one is the real cost of a consistent snapshot; if it
becomes unacceptable the answer is `sqlite3 .backup` plus an online repo copy,
not skipping the stop.
## The trap is the reason this role exists
When `backup_source_stop_service` is set, the script stops the unit and installs
an EXIT trap that starts it again. Without it, a failed dump leaves the service
down until the next timer fires — **every hand-written script this replaced had
that bug**, and it was only ever masked because their `systemctl stop` failed
first, before anything was stopped.
Verified on spacey: with the dump forced to fail, the log shows
`Stopping → Writing → Restarting`, the script exits 1 (so systemd marks the unit
failed rather than hiding it), and headscale is `active` afterwards.
If `systemctl stop` itself fails, `set -e` exits *before* the trap is installed —
which is correct, because nothing was stopped.
## `.partial`
The dump writes `<artifact>.partial` and only `mv`s it into place on success, so
a truncated file is never mistaken for a backup. A failure inside the pipeline
does leave one behind, and the prune glob cannot match it (it ends `.partial`,
not `.tar.gz.age`), so the script clears stale partials at the **start** of each
run. Tested by failing mid-pipeline: 1 partial left, 0 after the next run.
## `backup_source_stop_service` may be a bare name
`headscale` and `headscale.service` both work. The unit template normalises it,
because systemd rejects a bare name in `After=` with
`Failed to add dependency ... Invalid argument` — which it logs and then ignores,
so the unit appears to work while carrying no ordering at all.
## Retention is two-tier
`backup_source_retention_days` is **local** and short — these hosts are
disk-constrained. The long tail lives on `small-backups-box`, which decides its
own retention per source. Losing the local copy is expected and fine.

44
ansible/roles/backup_source/defaults/main.yml
View file

@ -1,44 +0,0 @@
---
# Required
backup_source_name: "" # "headscale" -> headscale_<ts>.tar.gz.age
backup_source_description: "" # "Headscale"
backup_source_dump_command: "" # must write the payload to STDOUT
# Placement
backup_source_dir: "/opt/backups/{{ backup_source_name }}"
backup_source_artifact_suffix: "tar.gz.age"
# Encryption. Asymmetric: the host holds only the public key and cannot decrypt
# what it produces.
backup_source_recipient: "{{ age_backup_recipient }}"
# The unprivileged account small-backups-box pulls as. It owns the dump
# directory and nothing else; it deliberately has no sudo.
backup_source_pull_user: backup-pull
backup_source_pull_key: "{{ backup_pull_public_key }}"
# Safety. Give either a systemd unit, or an explicit pair of commands for
# services that are not systemd-managed (vaultwarden is a docker compose stack).
# Whichever is used, a trap guarantees the restart.
backup_source_stop_service: "" # systemd unit stopped for the dump
backup_source_stop_command: "" # overrides stop_service when set
backup_source_start_command: "" # required alongside stop_command
# Retention here is LOCAL and short; small-backups-box keeps the long tail.
backup_source_retention_days: 7
# Schedule. The box pulls at 04:00, so dumps must land before that.
backup_source_on_calendar: "*-*-* 02:00:00"
# ── Reporting ────────────────────────────────────────────────────────────────
# Where to report that this dump ran and produced a plausible artefact.
# Gatus external endpoint:
# POST {url}?success=true|false&error=...
# Authorization: Bearer {token}
# Empty is valid and is not an error: the unit's exit code is still the answer,
# and the STORE will independently notice a stale dump within ~26h because the
# artefact filename carries this dump's timestamp. Reporting here only buys
# earlier detection and tells you it was the DUMP that broke rather than the
# pull.
backup_source_push_url: ""
backup_source_push_token: ""

4
ansible/roles/backup_source/handlers/main.yml
View file

@ -1,4 +0,0 @@
---
- name: Reload systemd for backup units
ansible.builtin.systemd:
daemon_reload: yes

110
ansible/roles/backup_source/tasks/main.yml
View file

@ -1,110 +0,0 @@
---
- name: Assert backup_source parameters are sane
ansible.builtin.assert:
that:
- backup_source_name | length > 0
- backup_source_description | length > 0
- backup_source_dump_command | length > 0
- backup_source_recipient | length > 0
- backup_source_recipient is match('^age1[0-9a-z]{58}$')
- not (backup_source_stop_command | length > 0 and backup_source_start_command | length == 0)
fail_msg: >-
backup_source: '{{ backup_source_name | default("<unnamed>") }}' needs a name,
description, dump command and a valid age recipient (age1... 62 chars).
backup_source_stop_command must be paired with backup_source_start_command.
quiet: true
# Declared here rather than assumed. Stage 1 installed it by hand; this is what
# makes a rebuilt host get it too.
# Cache refresh is best-effort on purpose. An unrelated third-party repo with a
# bad signing key (spacey had two: an expired Caddy subkey and a SHA1 nodesource
# key) makes `apt-get update` return warnings, which the apt module treats as a
# hard failure — and that must not stop backups being configured. Installing the
# package is NOT best-effort: if age is genuinely unavailable, the next task fails.
- name: Refresh the apt cache (best effort)
ansible.builtin.apt:
update_cache: yes
cache_valid_time: 3600
failed_when: false
changed_when: false
- name: Ensure age is installed
ansible.builtin.apt:
name:
- age
# curl is needed only when backup_source_push_url is set, but installing it
# unconditionally keeps the task idempotent and it is present on every
# Debian host here anyway.
- curl
state: present
# The pull account: unprivileged, no sudo, exists only so small-backups-box can
# read the dump directory. Trust points one way — the box can read backups, and
# can do nothing else on this host.
- name: "Ensure the {{ backup_source_pull_user }} account exists"
ansible.builtin.user:
name: "{{ backup_source_pull_user }}"
system: yes
shell: /bin/sh # rsync-over-ssh needs a shell; nologin breaks it
home: "/var/lib/{{ backup_source_pull_user }}"
create_home: yes
password: '!' # no password login, ever
when: backup_source_pull_user | length > 0
- name: "Authorise the backup box's key for {{ backup_source_pull_user }}"
ansible.posix.authorized_key:
user: "{{ backup_source_pull_user }}"
key: "{{ backup_source_pull_key }}"
key_options: "restrict" # no pty, no forwarding, no user rc
exclusive: yes
state: present
when: backup_source_pull_user | length > 0
# The shared container above the per-service directories. It must be traversable
# or the pull account cannot reach its own directory. The script's `mkdir -p`
# runs under `umask 077` and would otherwise create this 0700.
- name: "Ensure {{ backup_source_dir | dirname }} is traversable"
ansible.builtin.file:
path: "{{ backup_source_dir | dirname }}"
state: directory
owner: root
group: root
mode: '0755'
- name: "Ensure {{ backup_source_dir }} exists"
ansible.builtin.file:
path: "{{ backup_source_dir }}"
state: directory
owner: root
group: "{{ backup_source_pull_user | default('root', true) }}"
mode: '0750'
- name: "Install the {{ backup_source_name }} backup script"
ansible.builtin.template:
src: backup.sh.j2
dest: "/usr/local/bin/{{ backup_source_name }}-backup.sh"
owner: root
group: root
mode: '0750'
validate: "bash -n %s"
# The .service carries the push token in an Environment= line, so it is 0600.
# The .timer holds nothing secret and stays world-readable.
- name: "Install the {{ backup_source_name }}-backup systemd units"
ansible.builtin.template:
src: "backup.{{ item.unit }}.j2"
dest: "/etc/systemd/system/{{ backup_source_name }}-backup.{{ item.unit }}"
owner: root
group: root
mode: "{{ item.mode }}"
loop:
- {unit: service, mode: "0600"}
- {unit: timer, mode: "0644"}
notify: Reload systemd for backup units
- name: "Enable the {{ backup_source_name }}-backup timer"
ansible.builtin.systemd:
name: "{{ backup_source_name }}-backup.timer"
enabled: yes
state: started
daemon_reload: yes

16
ansible/roles/backup_source/templates/backup.service.j2
View file

@ -1,16 +0,0 @@
[Unit]
Description={{ backup_source_description }} backup
{% if backup_source_stop_service %}
{# systemd rejects a bare name here ("Failed to add dependency ... Invalid
argument"), so normalise to a full unit name. #}
After={{ backup_source_stop_service if '.' in backup_source_stop_service else backup_source_stop_service ~ '.service' }}
{% endif %}
[Service]
Type=oneshot
ExecStart=/usr/local/bin/{{ backup_source_name }}-backup.sh
Environment=BACKUP_PUSH_URL={{ backup_source_push_url }}
Environment=BACKUP_PUSH_TOKEN={{ backup_source_push_token }}
StandardOutput=journal
StandardError=journal
SyslogIdentifier={{ backup_source_name }}-backup

129
ansible/roles/backup_source/templates/backup.sh.j2
View file

@ -1,129 +0,0 @@
#!/usr/bin/env bash
# {{ backup_source_description }} backup — managed by Ansible (roles/backup_source)
#
# Dumps to stdout, encrypts with age, writes {{ backup_source_dir }}.
# The host holds only the age PUBLIC key, so it cannot read its own backups.
set -euo pipefail
umask 077
BACKUP_DIR="{{ backup_source_dir }}"
RETENTION_DAYS={{ backup_source_retention_days }}
RECIPIENT="{{ backup_source_recipient }}"
SUFFIX="{{ backup_source_artifact_suffix }}"
NAME="{{ backup_source_name }}"
{% if backup_source_stop_service or backup_source_stop_command %}
STOP_CMD={{ (backup_source_stop_command or ('systemctl stop ' ~ backup_source_stop_service)) | quote }}
START_CMD={{ (backup_source_start_command or ('systemctl start ' ~ backup_source_stop_service)) | quote }}
SERVICE="{{ backup_source_stop_service or backup_source_description }}" # label for the log only
{% endif %}
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
ARTIFACT="${BACKUP_DIR}/${NAME}_${TIMESTAMP}.${SUFFIX}"
die() { echo "FATAL: $*" >&2; exit 1; }
log() { echo "$(date '+%Y-%m-%d %H:%M:%S') $*"; }
# --- Pre-flight ---
[[ -n "$RECIPIENT" ]] || die "no age recipient configured"
command -v age >/dev/null || die "age is not installed"
# Mode must agree with what the role sets, or each undoes the other every run.
mkdir -p "$BACKUP_DIR"
{% if backup_source_pull_user %}
chown root:{{ backup_source_pull_user }} "$BACKUP_DIR"
chmod 750 "$BACKUP_DIR"
{% else %}
chmod 700 "$BACKUP_DIR"
{% endif %}
# A run that died mid-dump leaves a .partial. It is not a backup, and the prune
# glob below cannot match it (it ends .partial, not .${SUFFIX}), so clear them
# here or they accumulate forever.
rm -f "${BACKUP_DIR}/${NAME}_"*.partial
# --- Reporting -------------------------------------------------------------
# A dump that exits non-zero, or that produces a zero-byte artefact, is a failed
# backup even though the script "finished". Both are reported as failures.
PUSH_URL="${BACKUP_PUSH_URL:-}"
PUSH_TOKEN="${BACKUP_PUSH_TOKEN:-}"
report() {
local success="$1" message="$2"
[ -n "$PUSH_URL" ] || return 0
local encoded
encoded=$(printf '%s' "$message" | sed 's/%/%25/g; s/ /%20/g; s/&/%26/g; s/+/%2B/g; s/#/%23/g')
curl -s -o /dev/null --max-time 15 --retry 2 --retry-delay 3 -X POST \
-H "Authorization: Bearer ${PUSH_TOKEN}" \
"${PUSH_URL}?success=${success}&error=${encoded}" 2>/dev/null || true
}
# Reports on ANY exit path, so a dump that dies halfway still reports rather
# than going quiet. The size of the FINISHED artefact decides success, not
# merely reaching the end of the script.
#
# This is called FROM the single EXIT trap below - it must never register an
# EXIT trap of its own. `trap ... EXIT` REPLACES the existing handler rather
# than adding to it, so a second trap here silently discards the one that
# restarts the service, and a backup run leaves the service stopped. That is
# precisely the failure the restart trap exists to prevent.
report_outcome() {
local rc="$1"
if [ "$rc" -ne 0 ]; then
report "false" "${NAME} dump exited ${rc}"
elif [ ! -s "$ARTIFACT" ]; then
report "false" "${NAME} produced no artefact at ${ARTIFACT}"
else
report "true" "${NAME} $(du -h "$ARTIFACT" | cut -f1)"
fi
}
# --- One EXIT handler, doing both jobs -------------------------------------
# bash keeps exactly ONE EXIT trap: `trap ... EXIT` REPLACES the previous
# handler rather than adding to it. Registering a second one here would
# silently discard the service restart and leave the service stopped after
# every backup - which is the exact bug the restart exists to prevent, and it
# is invisible until someone notices the service is down.
on_exit() {
local rc=$?
{% if backup_source_stop_service or backup_source_stop_command %}
log "Restarting ${SERVICE}..."
eval "$START_CMD" || true
{% endif %}
report_outcome "$rc"
}
trap on_exit EXIT
{% if backup_source_stop_service or backup_source_stop_command %}
# --- Stop the service; the trap above guarantees it comes back -------------
# The trap is the point: without it a failed dump leaves the service down until
# the next timer fires. Every hand-written script this replaced had that bug.
# It is armed BEFORE the stop, so even a failure during the stop restarts.
log "Stopping ${SERVICE}..."
eval "$STOP_CMD"
{% endif %}
# --- Dump straight into age; plaintext never touches the disk ---
log "Writing ${ARTIFACT}..."
{{ backup_source_dump_command }} | age -r "$RECIPIENT" -o "${ARTIFACT}.partial"
{% if backup_source_pull_user %}
# Match the final ownership immediately, so even a partial left by a later
# failure is not an unreadable obstacle to the pull.
chown root:{{ backup_source_pull_user }} "${ARTIFACT}.partial"
chmod 640 "${ARTIFACT}.partial"
{% endif %}
mv "${ARTIFACT}.partial" "$ARTIFACT"
{% if backup_source_pull_user %}
# Readable by the pull account and nobody else. The contents are age-encrypted
# regardless, so this is depth rather than the actual protection.
chown root:{{ backup_source_pull_user }} "$ARTIFACT"
chmod 640 "$ARTIFACT"
{% else %}
chmod 600 "$ARTIFACT"
{% endif %}
log "Wrote ${ARTIFACT} ($(du -h "$ARTIFACT" | cut -f1))"
# --- Prune ---
log "Pruning local artefacts older than ${RETENTION_DAYS} days..."
find "$BACKUP_DIR" -maxdepth 1 -type f -name "${NAME}_*.${SUFFIX}" -mtime +"${RETENTION_DAYS}" -delete
log "Done."

11
ansible/roles/backup_source/templates/backup.timer.j2
View file

@ -1,11 +0,0 @@
[Unit]
Description={{ backup_source_description }} backup
[Timer]
OnCalendar={{ backup_source_on_calendar }}
# Persistent: a window missed while the host was down runs on next boot. cron on
# a laptop had no equivalent, which is how two backups went unnoticed for months.
Persistent=true
[Install]
WantedBy=timers.target

55
ansible/roles/backup_store/README.md
View file

@ -1,55 +0,0 @@
# `backup_store`
Pulls already-encrypted backup artefacts from every source host onto
`small-backups-box`, on a timer, and expires them per source.
Generalises the hand-written `pull-backups.sh` that had one hardcoded source
(`arbret`). That job's behaviour is preserved exactly: same source path, same
90 days, same destination directory.
## This host holds no key
Everything pulled here is ciphertext produced by `backup_source` on the source
host. The box cannot read any of it — the age identity lives only on lapy. That
is deliberate: the machine holding every backup should not also be able to open
them.
## One failing source must not stop the others
The script is `set -uo pipefail`, **not** `-e`. Each source runs in its own
function, failures are counted, and the script exits non-zero at the end so
systemd marks the unit failed. A dead host costs you that one source, not the
whole run.
This is the specific failure the whole plan exists to prevent: the laptop jobs
aborted on first error and then silently produced empty directories for nine
months.
## Trust points one way
The box authenticates with `~/.ssh/id_pull` to an unprivileged, dedicated
account on each source (`backup-pull`, or `arbret` on prd-arbret), authorised
with `restrict`. That account can read one directory and do nothing else — no
sudo, no pty, no forwarding. A compromised backup box cannot reach into
production.
## Addressing: names, never IPs
Sources are addressed by name. The job this replaced hardcoded spacey's IP; the
droplet was later rebuilt, the address was recycled to a stranger, and the
backup failed silently from 2025-12-01 while the directory listing still looked
healthy.
Two kinds of name are in play:
- **Tailnet members** (vipy, memos-box, …) → MagicDNS names. These require a
headscale ACL grant from `tag:small-backups-box` to the source's `:22`; without
it the box cannot even resolve the peer, let alone reach it.
- **spacey** is *not* a tailnet member — it is the headscale control server — so
its backup is pulled over the public internet via `headscale.contrapeso.xyz`,
which follows the host if the droplet is rebuilt.
## Retention here is the long tail
Sources keep a few days locally; this box keeps 90 (or whatever the source entry
says). Losing the source's local copy is expected.

32
ansible/roles/backup_store/defaults/main.yml
View file

@ -1,32 +0,0 @@
---
backup_store_dir: "{{ ansible_env.HOME }}/backups"
backup_store_ssh_key: "{{ ansible_env.HOME }}/.ssh/id_pull"
backup_store_on_calendar: "*-*-* 04:00:00"
# One entry per source. `retention_days` is the LONG tail; the source keeps its
# own short local retention.
# - name: headscale
# source: "backup-pull@headscale.contrapeso.xyz:/opt/backups/headscale/"
# retention_days: 90
backup_store_sources: []
# ── Reporting ────────────────────────────────────────────────────────────────
# check-backups.sh reports one result PER SOURCE plus one for the store itself,
# so the base URL is the endpoints collection and the script appends each key.
# Empty is valid: the script still prints its report and exits 0/1.
backup_store_check_push_base: ""
backup_store_check_push_token: ""
# Every six hours, offset past the 04:00 pull so the first run of the day sees a
# finished pull. The BACKUPS are daily, but this check is not - it reads the
# source's dump timestamp out of the artefact filename, so running it more often
# catches "the source stopped dumping" within hours rather than a day, and lets
# the Gatus heartbeat be 7h instead of 30h.
# 05:30 then every 6h. Note the syntax: a comma-separated list of full
# times ("05:30:00,11:30:00,...") is NOT valid and systemd rejects the unit
# with "bad unit file setting" - validated with `systemd-analyze calendar`.
backup_store_check_on_calendar: "*-*-* 05/6:30:00"
# An artefact older than this is stale. Sources dump daily at 02:00-02:30 and the
# pull is at 04:00, so 26h tolerates exactly one missed night before alarming.
backup_store_check_max_age_hours: 26

5
ansible/roles/backup_store/handlers/main.yml
View file

@ -1,5 +0,0 @@
---
- name: Reload systemd for pull-backups
ansible.builtin.systemd:
daemon_reload: yes
become: yes

89
ansible/roles/backup_store/tasks/main.yml
View file

@ -1,89 +0,0 @@
---
- name: Assert backup_store sources are sane
ansible.builtin.assert:
that:
- backup_store_sources | length > 0
- backup_store_sources | map(attribute='name') | list | length == backup_store_sources | length
- backup_store_sources | map(attribute='source') | list | length == backup_store_sources | length
- backup_store_sources | map(attribute='retention_days') | list | length == backup_store_sources | length
fail_msg: "backup_store: every source needs name, source and retention_days"
quiet: true
- name: Ensure rsync is installed
ansible.builtin.apt:
name: rsync
state: present
update_cache: yes
cache_valid_time: 3600
become: yes
- name: Ensure the backup store directory exists
ansible.builtin.file:
path: "{{ backup_store_dir }}"
state: directory
mode: '0700'
- name: Install the pull-backups script
ansible.builtin.template:
src: pull-backups.sh.j2
dest: /usr/local/bin/pull-backups.sh
owner: root
group: root
mode: '0755'
validate: "bash -n %s"
become: yes
# An assertion that last night actually worked. Generated from the same source
# list as the puller, so it can never drift out of sync with what is supposed to
# be arriving. Runs on a timer AND is useful by hand.
- name: Install the backup check script
ansible.builtin.template:
src: check-backups.sh.j2
dest: /usr/local/bin/check-backups.sh
owner: root
group: root
mode: '0755'
validate: "bash -n %s"
become: yes
# The .service carries the push token, so it is 0600; the .timer is not secret.
- name: Install the check-backups systemd units
ansible.builtin.template:
src: "check-backups.{{ item.unit }}.j2"
dest: "/etc/systemd/system/check-backups.{{ item.unit }}"
owner: root
group: root
mode: "{{ item.mode }}"
loop:
- {unit: service, mode: "0600"}
- {unit: timer, mode: "0644"}
become: yes
# restarted, not started: `started` is a no-op on an already-active timer, so a
# changed schedule would never be picked up.
- name: Enable the check-backups timer
ansible.builtin.systemd:
name: check-backups.timer
enabled: yes
state: restarted
daemon_reload: yes
become: yes
- name: Install the pull-backups systemd units
ansible.builtin.template:
src: "pull-backups.{{ item }}.j2"
dest: "/etc/systemd/system/pull-backups.{{ item }}"
owner: root
group: root
mode: '0644'
loop: [service, timer]
become: yes
notify: Reload systemd for pull-backups
- name: Enable the pull-backups timer
ansible.builtin.systemd:
name: pull-backups.timer
enabled: yes
state: started
daemon_reload: yes
become: yes

16
ansible/roles/backup_store/templates/check-backups.service.j2
View file

@ -1,16 +0,0 @@
[Unit]
Description=Verify the nightly backup pull actually worked
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
User={{ ansible_user_id }}
ExecStart=/usr/local/bin/check-backups.sh {{ backup_store_check_max_age_hours }}
Environment=BACKUP_CHECK_PUSH_BASE={{ backup_store_check_push_base }}
Environment=BACKUP_CHECK_PUSH_TOKEN={{ backup_store_check_push_token }}
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target

197
ansible/roles/backup_store/templates/check-backups.sh.j2
View file

@ -1,197 +0,0 @@
#!/usr/bin/env bash
# Assert the nightly backups actually worked.
#
# Run as {{ ansible_user_id }} on this host. Needs no sudo.
#
# What it CANNOT do: verify contents. The age identity lives only on lapy, so
# this host cannot decrypt anything it holds — by design. These are freshness,
# completeness and integrity checks. To verify content, decrypt on lapy:
# ssh {{ ansible_user_id }}@$(hostname) "cat ~/backups/<svc>/<artefact>" \
# | age -d -i ~/.age/counterweight_age | tar -tzf - | head
#
# Exit 0 = everything passed (warnings allowed), 1 = at least one FAIL.
set -uo pipefail
STORE="{{ backup_store_dir }}"
MAX_AGE_H="${1:-26}" # an artefact older than this is stale
NOW=$(date +%s)
fails=0; warns=0
# Colour only when attached to a terminal: this gets piped into files and, later,
# probably into a notification.
if [ -t 1 ]; then R=$'\033[31m'; Y=$'\033[33m'; G=$'\033[32m'; N=$'\033[0m'
else R=''; Y=''; G=''; N=''; fi
# Per-source verdicts, so each source can be reported independently. A single
# aggregate red light tells you backups are broken; it does not tell you which
# one, which is the thing you need at 3am.
declare -A SRC_FAIL SRC_MSG
CURRENT=""
red() { printf ' %sFAIL%s %s\n' "$R" "$N" "$*"; fails=$((fails+1));
[ -n "$CURRENT" ] && { SRC_FAIL[$CURRENT]=1; SRC_MSG[$CURRENT]="${SRC_MSG[$CURRENT]:-}${SRC_MSG[$CURRENT]:+; }$*"; }; }
yell() { printf ' %sWARN%s %s\n' "$Y" "$N" "$*"; warns=$((warns+1)); }
ok() { printf ' %sok%s %s\n' "$G" "$N" "$*";
[ -n "$CURRENT" ] && SRC_MSG[$CURRENT]="${SRC_MSG[$CURRENT]:-}${SRC_MSG[$CURRENT]:+; }$*"; }
# --- Reporting -------------------------------------------------------------
# Each source gets its own Gatus external endpoint, plus one for the store
# itself (the pull unit, the timer, and disk capacity). PUSH_BASE empty means
# report nowhere, which is valid: the exit code is still the whole answer.
PUSH_BASE="${BACKUP_CHECK_PUSH_BASE:-}"
PUSH_TOKEN="${BACKUP_CHECK_PUSH_TOKEN:-}"
report() {
local key="$1" success="$2" message="$3"
[ -n "$PUSH_BASE" ] || return 0
local encoded
encoded=$(printf '%s' "$message" | sed 's/%/%25/g; s/ /%20/g; s/&/%26/g; s/+/%2B/g; s/#/%23/g')
curl -s -o /dev/null --max-time 15 --retry 2 --retry-delay 3 -X POST \
-H "Authorization: Bearer ${PUSH_TOKEN}" \
"${PUSH_BASE}/${key}/external?success=${success}&error=${encoded}" 2>/dev/null || true
}
hours_since() { echo $(( (NOW - $1) / 3600 )); }
# Pull the dump timestamp out of <name>_YYYYmmdd_HHMMSS.<suffix>. This is when
# the SOURCE produced it, which is the thing that actually matters: a source
# whose timer died still pulls "ok" forever, because yesterday's artefact is
# still sitting there. Checking only the pull would miss exactly that.
dump_epoch() {
local base ts
base=$(basename "$1")
ts=$(echo "$base" | grep -oE '[0-9]{8}_[0-9]{6}' | head -1) || return 1
[ -n "$ts" ] || return 1
date -d "${ts:0:4}-${ts:4:2}-${ts:6:2} ${ts:9:2}:${ts:11:2}:${ts:13:2}" +%s 2>/dev/null
}
check_source() {
local name="$1" keep="$2" dir="$STORE/$1"
printf '\n%s\n' "== $name"
CURRENT="$name"
SRC_FAIL[$name]=0
SRC_MSG[$name]=""
[ -d "$dir" ] || { red "$name: no directory $dir"; return; }
local n; n=$(find "$dir" -maxdepth 1 -type f -name "${name}_*" | wc -l)
[ "$n" -gt 0 ] || { red "$name: no artefacts at all"; return; }
local partials; partials=$(find "$dir" -maxdepth 1 -name '*.partial' | wc -l)
[ "$partials" -eq 0 ] || red "$name: $partials .partial file(s) pulled — the pull should exclude these"
local newest; newest=$(ls -t "$dir"/${name}_* 2>/dev/null | head -1)
local prev; prev=$(ls -t "$dir"/${name}_* 2>/dev/null | sed -n 2p)
# 1. Did the SOURCE dump recently?
local de; de=$(dump_epoch "$newest")
if [ -z "${de:-}" ]; then
yell "$name: cannot parse a dump timestamp from $(basename "$newest")"
else
local dh; dh=$(hours_since "$de")
if [ "$dh" -lt 0 ]; then
# A future-dated artefact would otherwise stay "fresh" forever and the
# staleness check would never fire again — the exact silent failure this
# script exists to catch.
red "$name: newest dump is dated ${dh#-}h in the FUTURE — clock skew on the source?"
elif [ "$dh" -gt "$MAX_AGE_H" ]; then
red "$name: newest dump is ${dh}h old (>${MAX_AGE_H}h) — the source timer did not run"
else
ok "$name: dumped ${dh}h ago"
fi
fi
# 2. Did the PULL bring it over recently?
local ph; ph=$(hours_since "$(stat -c %Y "$newest")")
if [ "$ph" -gt "$MAX_AGE_H" ]; then
red "$name: newest artefact was pulled ${ph}h ago (>${MAX_AGE_H}h)"
else
ok "$name: pulled ${ph}h ago"
fi
# 3. Is it plausibly a real backup?
local sz; sz=$(stat -c %s "$newest")
if [ "$sz" -eq 0 ]; then
red "$name: newest artefact is ZERO bytes"
elif [ -n "$prev" ]; then
local psz; psz=$(stat -c %s "$prev")
if [ "$psz" -gt 0 ] && [ "$sz" -lt $(( psz / 2 )) ]; then
# Not automatically wrong: headscale legitimately shrank 297K -> 20K when
# a clean stop checkpointed its write-ahead log into the database.
yell "$name: $(numfmt --to=iec "$sz") is less than half the previous $(numfmt --to=iec "$psz") — check it decrypts to what you expect"
else
ok "$name: $(numfmt --to=iec "$sz") ($n artefacts)"
fi
else
ok "$name: $(numfmt --to=iec "$sz") (first artefact)"
fi
# 4. Is retention pruning? Allow generous slack for multiple dumps per day.
if [ "$n" -gt $(( keep * 3 + 10 )) ]; then
yell "$name: $n artefacts for a ${keep}-day retention — pruning may not be working"
fi
}
echo "Backup check on $(hostname) at $(date '+%Y-%m-%d %H:%M:%S %Z')"
echo "Artefacts older than ${MAX_AGE_H}h are treated as stale."
# --- the pull job itself ---
# Reserved key, reported as backup-store_pull-job. The store's own machinery is
# a different alarm from any one source being stale, and it is a LEADING
# indicator where the per-source checks are lagging ones: those only fire once
# an artefact is >26h stale, i.e. about a day after the fault. A disabled timer,
# a failed pull job or a filling disk are all visible here immediately, and they
# name the cause instead of showing six stale sources with no explanation.
CURRENT="__store"
SRC_FAIL[__store]=0
SRC_MSG[__store]=""
printf '\n%s\n' "== pull-backups.service"
result=$(systemctl show pull-backups.service -p Result --value 2>/dev/null)
status=$(systemctl show pull-backups.service -p ExecMainStatus --value 2>/dev/null)
when=$(systemctl show pull-backups.service -p ExecMainExitTimestamp --value 2>/dev/null)
[ "$result" = "success" ] && ok "last run result: success" || red "last run result: ${result:-unknown} (exit ${status:-?})"
if [ -n "$when" ]; then
wh=$(hours_since "$(date -d "$when" +%s)")
[ "$wh" -le "$MAX_AGE_H" ] && ok "last ran ${wh}h ago" || red "last ran ${wh}h ago (>${MAX_AGE_H}h) — did the timer fire?"
fi
systemctl is-enabled pull-backups.timer >/dev/null 2>&1 \
&& ok "timer enabled, next $(systemctl show pull-backups.timer -p NextElapseUSecRealtime --value 2>/dev/null)" \
|| red "pull-backups.timer is NOT enabled"
# --- each source ---
{% for src in backup_store_sources %}
check_source "{{ src.name }}" {{ src.retention_days }}
CURRENT=""
# The store key is what Gatus computes from group+name: sanitize("backup-store")
# + "_" + sanitize("{{ src.name }}").
report "backup-store_{{ src.name }}" \
"$([ "${SRC_FAIL[{{ src.name }}]:-1}" -eq 0 ] && echo true || echo false)" \
"${SRC_MSG[{{ src.name }}]:-no result}"
{% endfor %}
# --- capacity ---
CURRENT="__store"
printf '\n%s\n' "== disk"
use=$(df --output=pcent "$STORE" | tail -1 | tr -dc '0-9')
avail=$(df -h --output=avail "$STORE" | tail -1 | tr -d ' ')
if [ "$use" -ge 90 ]; then red "store is ${use}% full, ${avail} free"
elif [ "$use" -ge 75 ]; then yell "store is ${use}% full, ${avail} free"
else ok "store is ${use}% full, ${avail} free"; fi
CURRENT=""
report "backup-store_pull-job" \
"$([ "${SRC_FAIL[__store]:-1}" -eq 0 ] && echo true || echo false)" \
"${SRC_MSG[__store]:-no result}"
printf '\n%s\n' "-----"
if [ "$fails" -gt 0 ]; then
echo "RESULT: $fails failure(s), $warns warning(s)"
echo "Investigate with: journalctl -u pull-backups -n 50 --no-pager"
exit 1
fi
if [ "$warns" -gt 0 ]; then
echo "RESULT: all checks passed, $warns warning(s)"
else
echo "RESULT: all checks passed"
fi
exit 0

11
ansible/roles/backup_store/templates/check-backups.timer.j2
View file

@ -1,11 +0,0 @@
[Unit]
Description=Run the backup verification after the nightly pull
Requires=check-backups.service
[Timer]
OnCalendar={{ backup_store_check_on_calendar }}
# Run a missed occurrence on the next boot rather than skipping the day.
Persistent=true
[Install]
WantedBy=timers.target

10
ansible/roles/backup_store/templates/pull-backups.service.j2
View file

@ -1,10 +0,0 @@
[Unit]
Description=Pull encrypted backups from production
[Service]
Type=oneshot
User={{ ansible_user_id }}
ExecStart=/usr/local/bin/pull-backups.sh
StandardOutput=journal
StandardError=journal
SyslogIdentifier=pull-backups

48
ansible/roles/backup_store/templates/pull-backups.sh.j2
View file

@ -1,48 +0,0 @@
#!/usr/bin/env bash
# Pull encrypted backups from production — managed by Ansible (roles/backup_store)
#
# Everything here is already ciphertext: this host only moves and expires files,
# and holds no key that can read them.
set -uo pipefail # deliberately NOT -e; see the loop below
SSH_KEY="{{ backup_store_ssh_key }}"
STORE="{{ backup_store_dir }}"
log() { echo "$(date '+%Y-%m-%d %H:%M:%S') $*"; }
fail() { echo "$(date '+%Y-%m-%d %H:%M:%S') ERROR: $*" >&2; failures=$((failures + 1)); }
failures=0
# One source failing must not stop the others. The whole point of this box is
# that a single dead host cannot silently take the rest of the backups with it —
# which is exactly how the laptop-based jobs failed unnoticed for nine months.
{% for src in backup_store_sources %}
# --- {{ src.name }} ---
pull_{{ src.name | replace('-', '_') }}() {
local dir="${STORE}/{{ src.name }}"
mkdir -p "$dir"
log "Pulling {{ src.name }} from {{ src.source }}..."
# --exclude '*.partial': a dump that died mid-write leaves one behind, owned
# root:root 0600 because the chown only happens after a successful mv. Without
# this exclude the pull account cannot read it and rsync fails for the WHOLE
# source — so one failed dump would silently block every subsequent pull of
# that service. An incomplete artefact is never worth transferring anyway.
if rsync -az --timeout=120 --exclude '*.partial' \
-e "ssh -i $SSH_KEY -o StrictHostKeyChecking=accept-new -o ConnectTimeout=15" \
"{{ src.source }}" "$dir/"; then
log " {{ src.name }}: ok ($(find "$dir" -maxdepth 1 -type f | wc -l) artefacts, $(du -sh "$dir" | cut -f1))"
else
fail "{{ src.name }}: rsync failed"
return 1
fi
log " {{ src.name }}: pruning older than {{ src.retention_days }} days"
find "$dir" -maxdepth 1 -type f -name '{{ src.name }}_*' -mtime +{{ src.retention_days }} -delete
}
pull_{{ src.name | replace('-', '_') }} || true
{% endfor %}
if [ "$failures" -gt 0 ]; then
log "FAILED: $failures source(s) did not pull"
exit 1
fi
log "All sources pulled."

9
ansible/roles/backup_store/templates/pull-backups.timer.j2
View file

@ -1,9 +0,0 @@
[Unit]
Description=Daily offsite backup pull
[Timer]
OnCalendar={{ backup_store_on_calendar }}
Persistent=true
[Install]
WantedBy=timers.target

85
ansible/roles/bitcoin_knots/README.md
View file

@ -1,85 +0,0 @@
# `bitcoin_knots`
Builds Bitcoin Knots from source with PGP + SHA256 verification of the release
tarball, runs it as a full node on `knots-box`, and keeps a health check on a
systemd timer. The second play in the calling playbook publishes the P2P port
from the edge host via `socket_proxy`.
Converted from `deploy_bitcoin_knots_playbook.yml` (892 lines) under Plan 6. The
playbook is now 40 lines.
## The build is guarded; the chain is never touched
`build.yml` is 32 tasks, every one carrying
`when: not bitcoind_binary_exists.stat.exists`. On a host that already has the
binary the whole download / verify / 30-60 minute compile skips — **including the
two `state: absent` deletions**, which target `/opt/bitcoin-knots/source` and the
extracted build directory.
The chain lives elsewhere and nothing here touches it:
| | |
|---|---|
| `bitcoin_knots_dir` | `/opt/bitcoin-knots` — build tree, safe to delete |
| `bitcoin_data_dir` | `/var/lib/bitcoin` — config, logs, wallets |
| `bitcoin_large_data_dir` | `/mnt/knots_data` — **~875 GB of blockchain** |
The signature-verification tasks are the security control of this role. They are
copied verbatim; do not "simplify" them.
## ⚠ This node is half of the mining setup
`bitcoin.conf` carries a DATUM Gateway section that was hand-added on the node
and was **missing from the playbook's template**:
```ini
blockmaxsize=3985000
blockmaxweight=3985000
blocknotify=killall -USR1 datum_gateway
maxmempool=1000
blockreconstructionextratxn=1000000
```
`blocknotify` is how `datum_gateway` learns a new block landed. Running the old
playbook would have deleted all of it, and solo mining would have carried on
grinding against a stale template — a silent failure that costs money rather
than raising an error. The template now carries it behind
`bitcoin_datum_gateway_enabled`.
**bitcoin-knots and datum-gateway are one system, not two services.** Changing
either config means thinking about both.
## The restart handler, and why exactness matters now
The hand-written `Restart bitcoind` handler carried
`when: uptime_kuma_enabled | default(false)`, so it had been inert since the
decommissioning: `bitcoin.conf` and the systemd unit both notify it and neither
could restart anything. A config change applied to disk, reported success, and
never took effect.
It is ungated here — which raises the bar for the template. **Any** residual
difference between the template and the live file, down to a trailing newline,
means the task reports `changed` and restarts a Bitcoin node on every run. It
took four rounds of `--check --diff` to reach `changed=0`: the DATUM section, an
explanatory comment that was rendering into the deployed file (now a `{# #}`
Jinja comment), a `# Pruning (optional)` comment the live file had, and one
trailing blank line.
## `dbcache`
Computed as 90% of RAM unless `bitcoin_dbcache_mb_override` is set. The live node
was hand-tuned to **200 MB**; the calculation produces 3528. As with fulcrum,
`set_fact` outranks role defaults, so the *calculation* honours the override — a
value pinned only in `defaults/` is silently ignored.
## Monitoring: one variable, no product knowledge
The check tests bitcoind's RPC and records the answer in its exit code, which
systemd keeps: `systemctl is-failed bitcoin-knots-healthcheck.service`. Set
`healthcheck_push_url` to report anywhere that accepts an HTTP ping.
The timer had last fired **2026-08-09** while still reporting `active` and
`enabled` — the same `OnBootSec` + `OnUnitActiveSec` dead chain as fulcrum, where
nothing re-arms it if the service does not run in a given boot. The role runs the
check once after enabling, which both smoke-tests it and supplies the reference
the timer schedules from.

93
ansible/roles/bitcoin_knots/defaults/main.yml
View file

@ -1,93 +0,0 @@
# Bitcoin Knots Configuration Variables
# Version - REQUIRED: Specify exact version/tag to build
# The only version string. There used to be a second, v-prefixed copy
# (bitcoin_knots_version) that nothing read - two hand-maintained copies of one
# fact, with nothing keeping them in step.
bitcoin_knots_version_short: "29.2.knots20251110"
# Directories
bitcoin_knots_dir: /opt/bitcoin-knots
bitcoin_knots_source_dir: "{{ bitcoin_knots_dir }}/source"
bitcoin_data_dir: /var/lib/bitcoin # Standard location for config, logs, wallets
bitcoin_large_data_dir: /mnt/knots_data # Custom location for blockchain data (blocks, chainstate)
bitcoin_conf_dir: /etc/bitcoin
# Network
bitcoin_rpc_port: 8332
# The edge host's socket-proxy/Caddy play needs this too, and a role default is
# invisible outside this role. The authoritative value for the live deployment is
# in host_vars/knots_box_local/main.yml, which outranks this; the value here is the
# protocol standard, so the role still works standalone.
bitcoin_p2p_port: 8333
bitcoin_rpc_bind: "0.0.0.0"
# RPC capacity. When a block lands, Fulcrum, mempool and DATUM all fetch it at
# once while validation holds the node busy on a saturated HDD; with the
# defaults the queue overflowed and bitcoind REJECTED calls ("http work queue
# depth exceeded" - 93 times on 2026-10-18), which also made the health probe
# flap. A deeper queue lets those bursts wait instead of failing. Costs a few
# idle threads and a little memory.
bitcoin_rpcworkqueue: 64
bitcoin_rpcthreads: 8
# Build options
bitcoin_build_jobs: 4 # Parallel build jobs (-j flag), adjust based on CPU cores
bitcoin_build_prefix: /usr/local
# Configuration options
bitcoin_enable_txindex: true # Set to true if transaction index needed (REQUIRED for Electrum servers like Electrs/ElectrumX)
bitcoin_max_connections: 125
# dbcache will be calculated as 90% of host RAM automatically in playbook
# ZMQ Configuration
bitcoin_zmq_enabled: true
bitcoin_zmq_bind: "tcp://0.0.0.0"
bitcoin_zmq_port_rawblock: 28332
bitcoin_zmq_port_rawtx: 28333
bitcoin_zmq_port_hashblock: 28334
bitcoin_zmq_port_hashtx: 28335
# Service user
bitcoin_user: bitcoin
bitcoin_group: bitcoin
# --- Health check ----------------------------------------------------------
# Checks bitcoind RPC and records the answer in its exit code, which systemd
# keeps: `systemctl is-failed bitcoin-knots-healthcheck.service`.
#
# WHERE TO REPORT HEALTH — the one place to plug in monitoring. Empty means
# check, exit honestly, report nowhere. Any endpoint accepting an HTTP ping
# works; nothing here is specific to a monitoring product.
healthcheck_push_url: ""
# Bearer token for the Gatus external endpoint. Required whenever a push URL
# is set: Gatus rejects an unauthenticated push with 401.
healthcheck_push_token: ""
# --- Logging ----------------------------------------------------------------
# The live node logs to a file. Set to "" to use printtoconsole=1 (journald).
bitcoin_logfile: "{{ bitcoin_data_dir }}/debug.log"
# --- dbcache ----------------------------------------------------------------
# Computed as 90% of RAM unless this is set. As with fulcrum, note that
# set_fact outranks role defaults, so the CALCULATION has to honour this - a
# value pinned only in defaults/ is silently ignored.
#
# 1200, raised from a hand-tuned 200 on 2026-10-18. At 200 MB almost no UTXO
# lookup hit memory: the P2P thread alone read ~4 MB/s off the saturated HDD
# mirror validating relayed transactions, and block validation took a median
# 11 s, p90 103 s, max 344 s - long enough for RPC to stall and the health
# probe to fail. 1200 is what fits in the 4 GB VM (bitcoind ~1.5 GB RSS at
# 200, mempool ~3 MB in practice). Raising the VM to 6 GB would allow ~3000.
bitcoin_dbcache_mb_override: 1200
# --- DATUM Gateway ----------------------------------------------------------
# This node feeds block templates to datum_gateway on knots-box. These settings
# were hand-added to bitcoin.conf and were missing from the template, so a
# playbook run would have removed them and broken the mining setup.
bitcoin_datum_gateway_enabled: true
bitcoin_blockmaxsize: 3985000
bitcoin_blockmaxweight: 3985000
bitcoin_blocknotify: "killall -USR1 datum_gateway"
bitcoin_maxmempool: 1000
bitcoin_blockreconstructionextratxn: 1000000

14
ansible/roles/bitcoin_knots/handlers/main.yml
View file

@ -1,14 +0,0 @@
---
# Ungated on purpose. The hand-written handler carried
# when: uptime_kuma_enabled | default(false)
# so it has been inert since the decommissioning. Two tasks notify it —
# bitcoin.conf and the systemd unit — and neither could actually restart
# bitcoind. A configuration change to a Bitcoin node therefore applied to disk,
# reported success, and silently never took effect.
#
# Restarting bitcoind re-opens the chainstate; it does not reindex.
- name: Restart bitcoind
systemd:
name: bitcoind
state: restarted
daemon_reload: yes

222
ansible/roles/bitcoin_knots/tasks/build.yml
View file

@ -1,222 +0,0 @@
---
# Every task here is guarded by `when: not bitcoind_binary_exists.stat.exists`,
# so on a host that already has the binary the whole download / verify / build
# sequence skips — including the two `state: absent` deletions, which target
# /opt/bitcoin-knots/{source,bitcoin-<version>} and never the chain data in
# /mnt/knots_data.
- name: Check if bitcoind binary already exists
stat:
path: "{{ bitcoin_build_prefix }}/bin/bitcoind"
register: bitcoind_binary_exists
changed_when: false
- name: Install gnupg for signature verification
apt:
name: gnupg
state: present
when: not bitcoind_binary_exists.stat.exists
- name: Import Luke Dashjr's Bitcoin Knots signing key
command: gpg --keyserver hkps://keyserver.ubuntu.com --recv-keys 90C8019E36C2E964
register: key_import
changed_when: "'already in secret keyring' not in key_import.stdout and 'already in public keyring' not in key_import.stdout"
when: not bitcoind_binary_exists.stat.exists
failed_when: key_import.rc != 0
- name: Display imported key fingerprint
command: gpg --fingerprint 90C8019E36C2E964
register: key_fingerprint
changed_when: false
when: not bitcoind_binary_exists.stat.exists
- name: Download SHA256SUMS file
get_url:
url: "https://bitcoinknots.org/files/{{ bitcoin_version_major }}.x/{{ bitcoin_knots_version_short }}/SHA256SUMS"
dest: "/tmp/bitcoin-knots-{{ bitcoin_knots_version_short }}-SHA256SUMS"
mode: '0644'
when: not bitcoind_binary_exists.stat.exists
- name: Download SHA256SUMS.asc signature file
get_url:
url: "https://bitcoinknots.org/files/{{ bitcoin_version_major }}.x/{{ bitcoin_knots_version_short }}/SHA256SUMS.asc"
dest: "/tmp/bitcoin-knots-{{ bitcoin_knots_version_short }}-SHA256SUMS.asc"
mode: '0644'
when: not bitcoind_binary_exists.stat.exists
- name: Verify PGP signature on SHA256SUMS file
command: gpg --verify /tmp/bitcoin-knots-{{ bitcoin_knots_version_short }}-SHA256SUMS.asc /tmp/bitcoin-knots-{{ bitcoin_knots_version_short }}-SHA256SUMS
register: sha256sums_verification
changed_when: false
failed_when: false # Don't fail here - check for 'Good signature' in next task
when: not bitcoind_binary_exists.stat.exists
- name: Display SHA256SUMS verification result
debug:
msg: "{{ sha256sums_verification.stdout_lines + sha256sums_verification.stderr_lines }}"
when: not bitcoind_binary_exists.stat.exists
- name: Fail if SHA256SUMS signature verification failed
fail:
msg: "SHA256SUMS signature verification failed. Aborting build."
when: not bitcoind_binary_exists.stat.exists and ('Good signature' not in sha256sums_verification.stdout and 'Good signature' not in sha256sums_verification.stderr)
- name: Remove any existing tarball to force fresh download
file:
path: /tmp/bitcoin-{{ bitcoin_knots_version_short }}.tar.gz
state: absent
when: not bitcoind_binary_exists.stat.exists
- name: Download Bitcoin Knots source tarball
get_url:
url: "{{ bitcoin_source_tarball_url }}"
dest: "/tmp/bitcoin-{{ bitcoin_knots_version_short }}.tar.gz"
mode: '0644'
validate_certs: yes
force: yes
when: not bitcoind_binary_exists.stat.exists
- name: Calculate SHA256 checksum of downloaded tarball
command: sha256sum /tmp/bitcoin-{{ bitcoin_knots_version_short }}.tar.gz
register: tarball_checksum
changed_when: false
when: not bitcoind_binary_exists.stat.exists
- name: Extract expected checksum from SHA256SUMS file
shell: grep "bitcoin-{{ bitcoin_knots_version_short }}.tar.gz" /tmp/bitcoin-knots-{{ bitcoin_knots_version_short }}-SHA256SUMS | awk '{print $1}'
register: expected_checksum
changed_when: false
when: not bitcoind_binary_exists.stat.exists
failed_when: expected_checksum.stdout == ""
- name: Display checksum comparison
debug:
msg:
- "Expected: {{ expected_checksum.stdout | trim }}"
- "Actual: {{ tarball_checksum.stdout.split()[0] }}"
when: not bitcoind_binary_exists.stat.exists
- name: Verify tarball checksum matches SHA256SUMS
fail:
msg: "Tarball checksum mismatch! Expected {{ expected_checksum.stdout | trim }}, got {{ tarball_checksum.stdout.split()[0] }}"
when: not bitcoind_binary_exists.stat.exists and expected_checksum.stdout | trim != tarball_checksum.stdout.split()[0]
- name: Remove existing source directory if it exists (to force fresh extraction)
file:
path: "{{ bitcoin_knots_source_dir }}"
state: absent
when: not bitcoind_binary_exists.stat.exists
- name: Remove extracted directory if it exists (from previous runs)
file:
path: "{{ bitcoin_knots_dir }}/bitcoin-{{ bitcoin_knots_version_short }}"
state: absent
when: not bitcoind_binary_exists.stat.exists
- name: Extract verified source tarball
unarchive:
src: /tmp/bitcoin-{{ bitcoin_knots_version_short }}.tar.gz
dest: "{{ bitcoin_knots_dir }}"
remote_src: yes
when: not bitcoind_binary_exists.stat.exists
- name: Check if extracted directory exists
stat:
path: "{{ bitcoin_knots_dir }}/bitcoin-{{ bitcoin_knots_version_short }}"
register: extracted_dir_stat
changed_when: false
when: not bitcoind_binary_exists.stat.exists
- name: Rename extracted directory to expected name
command: mv "{{ bitcoin_knots_dir }}/bitcoin-{{ bitcoin_knots_version_short }}" "{{ bitcoin_knots_source_dir }}"
when: not bitcoind_binary_exists.stat.exists and extracted_dir_stat.stat.exists
- name: Check if CMakeLists.txt exists
stat:
path: "{{ bitcoin_knots_source_dir }}/CMakeLists.txt"
register: cmake_exists
changed_when: false
when: not bitcoind_binary_exists.stat.exists
- name: Create CMake build directory
file:
path: "{{ bitcoin_knots_source_dir }}/build"
state: directory
mode: '0755'
when: not bitcoind_binary_exists.stat.exists and cmake_exists.stat.exists | default(false)
- name: Configure Bitcoin Knots build with CMake
command: >
cmake
-DCMAKE_INSTALL_PREFIX={{ bitcoin_build_prefix }}
-DBUILD_BITCOIN_WALLET=OFF
-DCMAKE_BUILD_TYPE=Release
-DWITH_ZMQ=ON
..
args:
chdir: "{{ bitcoin_knots_source_dir }}/build"
when: not bitcoind_binary_exists.stat.exists and cmake_exists.stat.exists | default(false)
register: configure_result
changed_when: true
- name: Verify CMake enabled ZMQ
shell: |
set -e
cd "{{ bitcoin_knots_source_dir }}/build"
cmake -LAH .. | grep -iE 'ZMQ|WITH_ZMQ|ENABLE_ZMQ|USE_ZMQ'
when: not bitcoind_binary_exists.stat.exists and cmake_exists.stat.exists | default(false)
register: zmq_check
changed_when: false
- name: Fail if CMakeLists.txt not found
fail:
msg: "CMakeLists.txt not found in {{ bitcoin_knots_source_dir }}. Cannot build Bitcoin Knots."
when: not bitcoind_binary_exists.stat.exists and not (cmake_exists.stat.exists | default(false))
- name: Build Bitcoin Knots with CMake (this may take 30-60+ minutes)
command: cmake --build . -j{{ bitcoin_build_jobs }}
args:
chdir: "{{ bitcoin_knots_source_dir }}/build"
when: not bitcoind_binary_exists.stat.exists and cmake_exists.stat.exists | default(false)
async: 3600
poll: 0
register: build_result
changed_when: true
- name: Check build status
async_status:
jid: "{{ build_result.ansible_job_id }}"
register: build_job_result
until: build_job_result.finished
retries: 120
delay: 60
when: not bitcoind_binary_exists.stat.exists and build_result.ansible_job_id is defined
- name: Fail if build failed
fail:
msg: "Bitcoin Knots build failed: {{ build_job_result.msg }}"
when: not bitcoind_binary_exists.stat.exists and build_result.ansible_job_id is defined and build_job_result.failed | default(false)
- name: Install Bitcoin Knots binaries
command: cmake --install .
args:
chdir: "{{ bitcoin_knots_source_dir }}/build"
when: not bitcoind_binary_exists.stat.exists and cmake_exists.stat.exists | default(false)
changed_when: true
- name: Verify bitcoind binary exists
stat:
path: "{{ bitcoin_build_prefix }}/bin/bitcoind"
register: bitcoind_installed
changed_when: false
- name: Verify bitcoin-cli binary exists
stat:
path: "{{ bitcoin_build_prefix }}/bin/bitcoin-cli"
register: bitcoin_cli_installed
changed_when: false
- name: Fail if binaries not found
fail:
msg: "Bitcoin Knots binaries not found after installation"
when: not bitcoind_installed.stat.exists or not bitcoin_cli_installed.stat.exists

20
ansible/roles/bitcoin_knots/tasks/configure.yml
View file

@ -1,20 +0,0 @@
---
# Ownership copied verbatim from the playbook this replaces; verified
# mechanically against `git show HEAD:` rather than retyped from memory.
- name: Create bitcoin.conf configuration file
ansible.builtin.template:
src: bitcoin.conf.j2
dest: "{{ bitcoin_conf_dir }}/bitcoin.conf"
owner: "{{ bitcoin_user }}"
group: "{{ bitcoin_group }}"
mode: '0640'
notify: Restart bitcoind
- name: Create systemd service file for bitcoind
ansible.builtin.template:
src: bitcoind.service.j2
dest: /etc/systemd/system/bitcoind.service
owner: root
group: root
mode: '0644'
notify: Restart bitcoind

56
ansible/roles/bitcoin_knots/tasks/healthcheck.yml
View file

@ -1,56 +0,0 @@
---
# Everything here answers "is bitcoind healthy" and records the answer. The
# Uptime Kuma specifics that used to follow — an embedded Python script creating
# monitors over the API, a /tmp credentials file, push-URL extraction and a
# systemd Environment= rewrite — are gone. Where it reports is now one variable,
# healthcheck_push_url. See the role README.
- name: Install curl for health check script
apt:
name: curl
state: present
- name: Create Bitcoin Knots health check script
ansible.builtin.template:
src: healthcheck.sh.j2
dest: /usr/local/bin/bitcoin-knots-healthcheck-push.sh
owner: root
group: root
mode: '0755'
validate: "bash -n %s"
- name: Create systemd service for Bitcoin Knots health check
ansible.builtin.template:
src: healthcheck.service.j2
dest: /etc/systemd/system/bitcoin-knots-healthcheck.service
owner: root
group: root
mode: "0600"
- name: Create systemd timer for Bitcoin Knots health check
ansible.builtin.template:
src: healthcheck.timer.j2
dest: /etc/systemd/system/bitcoin-knots-healthcheck.timer
owner: root
group: root
mode: '0644'
- name: Reload systemd daemon for health check
systemd:
daemon_reload: yes
- name: Enable and restart the Bitcoin Knots health check timer
systemd:
name: bitcoin-knots-healthcheck.timer
enabled: yes
state: restarted
daemon_reload: yes
# Runs the check once, which is both a smoke test and the thing that actually
# arms the timer. This timer is OnBootSec + OnUnitActiveSec with no OnCalendar:
# OnBootSec elapses once, and OnUnitActiveSec needs the SERVICE to have run this
# boot to have anything to schedule from. Restarting the timer does not supply
# that reference; running the service does. The live timer had last fired on
# 2026-08-09 while still reporting `active` and `enabled`.
- name: Run the Bitcoin Knots health check once to arm the timer
command: systemctl start bitcoin-knots-healthcheck.service
changed_when: false

104
ansible/roles/bitcoin_knots/tasks/install.yml
View file

@ -1,104 +0,0 @@
---
- name: Calculate dbcache as a share of system RAM
set_fact:
bitcoin_dbcache_mb: "{{ (ansible_memtotal_mb | float * 0.9) | int }}"
when: bitcoin_dbcache_mb_override | string | length == 0
- name: Use the explicit dbcache override
set_fact:
bitcoin_dbcache_mb: "{{ bitcoin_dbcache_mb_override }}"
when: bitcoin_dbcache_mb_override | string | length > 0
changed_when: false
- name: Display calculated dbcache value
debug:
msg: "Setting dbcache to {{ bitcoin_dbcache_mb }} MB (90% of {{ ansible_memtotal_mb }} MB total RAM)"
- name: Install build dependencies
apt:
name:
- build-essential
- libtool
- autotools-dev
- automake
- pkg-config
- bsdmainutils
- python3
- python3-pip
- libevent-dev
- libboost-system-dev
- libboost-filesystem-dev
- libboost-test-dev
- libboost-thread-dev
- libboost-chrono-dev
- libboost-program-options-dev
- libboost-dev
- libssl-dev
- libdb-dev
- libminiupnpc-dev
- libzmq3-dev
- libnatpmp-dev
- libsqlite3-dev
- git
- curl
- wget
- cmake
state: present
update_cache: yes
- name: Create bitcoin group
group:
name: "{{ bitcoin_group }}"
system: yes
state: present
- name: Create bitcoin user
user:
name: "{{ bitcoin_user }}"
group: "{{ bitcoin_group }}"
system: yes
shell: /usr/sbin/nologin
home: "{{ bitcoin_data_dir }}"
create_home: yes
state: present
- name: Create bitcoin-knots directory
file:
path: "{{ bitcoin_knots_dir }}"
state: directory
owner: root
group: root
mode: '0755'
- name: Create bitcoin-knots source directory
file:
path: "{{ bitcoin_knots_source_dir }}"
state: directory
owner: root
group: root
mode: '0755'
- name: Create bitcoin data directory (for config, logs, wallets)
file:
path: "{{ bitcoin_data_dir }}"
state: directory
owner: "{{ bitcoin_user }}"
group: "{{ bitcoin_group }}"
mode: '0750'
- name: Create bitcoin large data directory (for blockchain)
file:
path: "{{ bitcoin_large_data_dir }}"
state: directory
owner: "{{ bitcoin_user }}"
group: "{{ bitcoin_group }}"
mode: '0750'
- name: Create bitcoin config directory
file:
path: "{{ bitcoin_conf_dir }}"
state: directory
owner: root
group: root
mode: '0755'

8
ansible/roles/bitcoin_knots/tasks/main.yml
View file

@ -1,8 +0,0 @@
---
# import_tasks, not include_tasks: static imports stay visible to --list-tasks,
# which is how this conversion was verified against the playbook it replaced.
- ansible.builtin.import_tasks: install.yml
- ansible.builtin.import_tasks: build.yml
- ansible.builtin.import_tasks: configure.yml
- ansible.builtin.import_tasks: service.yml
- ansible.builtin.import_tasks: healthcheck.yml

37
ansible/roles/bitcoin_knots/tasks/service.yml
View file

@ -1,37 +0,0 @@
---
- name: Reload systemd daemon
systemd:
daemon_reload: yes
- name: Enable and start bitcoind service
systemd:
name: bitcoind
enabled: yes
state: started
- name: Wait for bitcoind RPC to be available
uri:
url: "http://{{ bitcoin_rpc_bind }}:{{ bitcoin_rpc_port }}"
method: POST
body_format: json
body:
jsonrpc: "1.0"
id: "healthcheck"
method: "getblockchaininfo"
params: []
user: "{{ bitcoin_rpc_user }}"
password: "{{ bitcoin_rpc_password }}"
status_code: 200
timeout: 10
register: rpc_check
until: rpc_check.status == 200
retries: 30
delay: 5
ignore_errors: yes
- name: Display RPC connection status
debug:
msg: "Bitcoin Knots RPC is {{ 'available' if rpc_check.status == 200 else 'not yet available' }}"
# ═════════════════════════════════════════════════════════════════════════
# DEPRECATED — Uptime Kuma was decommissioned on 2026-09-11.

69
ansible/roles/bitcoin_knots/templates/bitcoin.conf.j2
View file

@ -1,69 +0,0 @@
# Bitcoin Knots Configuration
# Generated by Ansible
# Data directory (blockchain storage)
datadir={{ bitcoin_large_data_dir }}
# RPC Configuration
server=1
rpcuser={{ bitcoin_rpc_user }}
rpcpassword={{ bitcoin_rpc_password }}
rpcbind={{ bitcoin_rpc_bind }}
rpcport={{ bitcoin_rpc_port }}
rpcallowip=0.0.0.0/0
rpcworkqueue={{ bitcoin_rpcworkqueue }}
rpcthreads={{ bitcoin_rpcthreads }}
# Network Configuration
listen=1
port={{ bitcoin_p2p_port }}
maxconnections={{ bitcoin_max_connections }}
# Performance
dbcache={{ bitcoin_dbcache_mb }}
# Transaction Index (optional)
{% if bitcoin_enable_txindex %}
txindex=1
{% endif %}
{# The live node carries this comment and the template never produced it, so a
run would have silently deleted it. Harmless in itself, but matching it keeps
this task at `ok` - which means any future `changed` here is a real signal
rather than known noise. #}
# Pruning (optional)
# Logging
logtimestamps=1
{% if bitcoin_logfile %}
logfile={{ bitcoin_logfile }}
{% else %}
printtoconsole=1
{% endif %}
# ZMQ Configuration
{% if bitcoin_zmq_enabled | default(false) %}
zmqpubrawblock={{ bitcoin_zmq_bind }}:{{ bitcoin_zmq_port_rawblock }}
zmqpubrawtx={{ bitcoin_zmq_bind }}:{{ bitcoin_zmq_port_rawtx }}
zmqpubhashblock={{ bitcoin_zmq_bind }}:{{ bitcoin_zmq_port_hashblock }}
zmqpubhashtx={{ bitcoin_zmq_bind }}:{{ bitcoin_zmq_port_hashtx }}
{% endif %}
# Security
disablewallet=1
{% if bitcoin_datum_gateway_enabled %}
{# These were hand-added on the node and were NOT in this template, so running
the playbook would have stripped them. blocknotify is how datum_gateway
learns a new block landed; without it solo mining keeps grinding on a stale
template - a silent failure that costs money rather than raising an error.
Kept as a Jinja comment so the explanation stays in the repo and out of the
deployed config. #}
# Specific for DATUM gateway
blockmaxsize={{ bitcoin_blockmaxsize }}
blockmaxweight={{ bitcoin_blockmaxweight }}
blocknotify={{ bitcoin_blocknotify }}
maxmempool={{ bitcoin_maxmempool }}
blockreconstructionextratxn={{ bitcoin_blockreconstructionextratxn }}
{% endif %}

17
ansible/roles/bitcoin_knots/templates/bitcoind.service.j2
View file

@ -1,17 +0,0 @@
[Unit]
Description=Bitcoin Knots daemon
After=network.target
[Service]
Type=simple
User={{ bitcoin_user }}
Group={{ bitcoin_group }}
ExecStart={{ bitcoin_build_prefix }}/bin/bitcoind -conf={{ bitcoin_conf_dir }}/bitcoin.conf
Restart=always
RestartSec=10
TimeoutStopSec=600
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target

15
ansible/roles/bitcoin_knots/templates/healthcheck.service.j2
View file

@ -1,15 +0,0 @@
[Unit]
Description=Bitcoin Knots Health Check
After=network.target bitcoind.service
[Service]
Type=oneshot
User=root
ExecStart=/usr/local/bin/bitcoin-knots-healthcheck-push.sh
Environment=HEALTHCHECK_PUSH_URL={{ healthcheck_push_url }}
Environment=HEALTHCHECK_PUSH_TOKEN={{ healthcheck_push_token }}
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target

69
ansible/roles/bitcoin_knots/templates/healthcheck.sh.j2
View file

@ -1,69 +0,0 @@
#!/bin/bash
# Bitcoin Knots health check — managed by Ansible (roles/bitcoin_knots)
#
# The exit code is the answer and systemd keeps it:
# systemctl is-failed bitcoin-knots-healthcheck.service
# Reporting anywhere else is optional and generic.
#
#
RPC_HOST="{{ bitcoin_rpc_bind }}"
RPC_PORT={{ bitcoin_rpc_port }}
RPC_USER="{{ bitcoin_rpc_user }}"
RPC_PASSWORD="{{ bitcoin_rpc_password }}"
PUSH_URL="${HEALTHCHECK_PUSH_URL:-}"
PUSH_TOKEN="${HEALTHCHECK_PUSH_TOKEN:-}"
# Check if bitcoind RPC is responding
check_bitcoind() {
local response
response=$(curl -s --max-time 30 \
--user "${RPC_USER}:${RPC_PASSWORD}" \
--data-binary '{"jsonrpc":"1.0","id":"healthcheck","method":"getblockchaininfo","params":[]}' \
--header 'Content-Type: application/json' \
"http://${RPC_HOST}:${RPC_PORT}" 2>&1)
if [ $? -eq 0 ]; then
# Check if response contains a non-null error
# Successful responses have "error": null, failures have "error": {...}
if echo "$response" | grep -q '"error":null\|"error": null'; then
return 0
else
return 1
fi
else
return 1
fi
}
report() {
local status=$1
local msg=$2
# No push URL is normal, not an error: the exit code below is still a
# complete answer for anything reading unit state.
[ -n "$PUSH_URL" ] || return 0
# URL encode spaces in message
local encoded_msg="${msg// /%20}"
# Gatus external endpoint: a POST with a bearer token, NOT Uptime Kuma's
# GET with ?status=up. The callers still pass up/down, so the mapping is
# done here rather than at every call site.
local _ok=false
[ "${status}" = "up" ] && _ok=true
if ! curl -s --max-time 15 --retry 2 -o /dev/null -X POST \
-H "Authorization: Bearer ${PUSH_TOKEN}" \
"${PUSH_URL}?success=${_ok}&error=${encoded_msg}"; then
return 1
fi
}
# Main health check
if check_bitcoind; then
report "up" "OK"
exit 0
else
report "down" "bitcoind RPC not responding"
exit 1
fi

11
ansible/roles/bitcoin_knots/templates/healthcheck.timer.j2
View file

@ -1,11 +0,0 @@
[Unit]
Description=Bitcoin Knots Health Check Timer
Requires=bitcoind.service
[Timer]
OnBootSec=1min
OnUnitActiveSec=1min
Persistent=true
[Install]
WantedBy=timers.target

118
ansible/roles/caddy_site/README.md
View file

@ -1,118 +0,0 @@
# `caddy_site`
Writes one Caddy site file into `{{ caddy_sites_dir }}`, makes sure the main
Caddyfile imports that directory, validates the result, and reloads Caddy once.
Replaces the four-task block that was copy-pasted into 10 playbooks.
Runs on any host in the `[caddy]` group — `edge` (vipy), `monitoring`
(watchtower) and `vpn_control` (spacey).
## Usage
```yaml
- ansible.builtin.include_role:
name: caddy_site
vars:
caddy_site_name: forgejo # -> forgejo.conf
caddy_site_domain: "{{ forgejo_domain }}"
caddy_site_upstream: "localhost:{{ forgejo_port }}"
```
Use `include_role`, not a `roles:` block, so the call stays in task order next
to the tasks it depends on. Variables passed this way are scoped to the include
and do not leak into later calls — so **every call must pass everything it
needs**; nothing carries over.
## Shapes
Pick exactly one of `caddy_site_upstream`, `caddy_site_root`, `caddy_site_body`.
| Want | Set |
|---|---|
| `reverse_proxy host:port` | `caddy_site_upstream` |
| static `root *` + `file_server` | `caddy_site_root` |
| anything else | `caddy_site_body` (raw, indented 4 for you) |
`caddy_site_upstream` accepts two modifiers, which add a block to the
`reverse_proxy`:
- `caddy_site_headers_up: {"X-Forwarded-Host": "..."}`
- `caddy_site_resolvers: "100.100.100.100"` — Tailscale MagicDNS
and `caddy_site_basic_auth` wraps the site in a `basic_auth` block.
## `caddy_site_basic_auth` is a LIST, not a dict
```yaml
caddy_site_basic_auth:
- user: "{{ datum_dashboard_username }}"
hash: "{{ datum_dashboard_password_hash }}"
```
**Ansible does not template dictionary keys.** With `{ "{{ user }}": "hash" }`
the value is rendered and the key is not, so the literal string
`{{ datum_dashboard_username }}` lands in the config file. Found while building
this role; the `assert` refuses a mapping so it cannot happen again.
## Secrets and `--diff`
Rendered site files can carry credentials — `datum-gateway.conf` holds a bcrypt
hash — and `--diff` prints rendered content. The template task therefore sets
`diff: "{{ caddy_site_reveal | bool }}"`, default `false`, so `--diff` runs are
safe everywhere. Pass `-e caddy_site_reveal=true` to see what moved on a site
you know is not secret.
## Validation
`validate: "caddy validate --adapter caddyfile --config %s"` runs against the
rendered temp file before it is moved into place. Verified on vipy that a single
site fragment validates cleanly (rc=0, `Valid configuration`) and that a
malformed one is rejected (rc=1, with the syntax error and line number). A
failed validate leaves the live file untouched, so a broken config can no longer
reach a running Caddy.
What it cannot catch is a conflict with the global `/etc/caddy/Caddyfile`.
## The reload is a handler
`Reload caddy` fires **once, at the end of the play**, however many sites
notified it. The code this replaced ran `command: systemctl reload caddy`
immediately, mid-play. If a later task in the same play needs the new config to
be live, flush first:
```yaml
- ansible.builtin.meta: flush_handlers
```
## Known intentional difference
The `resolvers` block is commented `# Use Tailscale MagicDNS to resolve the
upstream hostname` in every case. `datum-gateway` previously said `# Resolve via
Tailscale MagicDNS`. Migrating it therefore rewrites one comment line, which
Caddy ignores. Every other site renders byte-identical to what its playbook
produced.
## Sites on the hosts that this role does NOT manage
Four vhosts exist in `/etc/caddy/sites-enabled/` that no playbook writes. They
were made by hand. The role only ever writes the one file it is told to, so it
leaves them alone — but nothing in the repo records them, and that is why they
are listed here. Checked 2026-09-11:
| File | Host | Serves | State |
|---|---|---|---|
| `uptime-kuma.conf` | watchtower | `localhost:3001` | **HTTP 302 — still live**, see below |
| `arbretstaging.conf` | vipy | `arbret-staging-box:80` via MagicDNS | HTTP 200 |
| `bitcoininfra.conf` | vipy | static `file_server` from `/var/www/bitcoin-services-home` | HTTP 200 |
| `scriberr.conf` | vipy | `scriberr-box:8080` via MagicDNS | HTTP 502 — upstream down |
**`uptime-kuma.conf` must not be deleted as dead config.** Uptime Kuma was
"decommissioned" in the repo — its playbooks archived and its credentials pulled
from the vault — but the container is **still running** on watchtower
(`louislam/uptime-kuma:latest`, created 2026-02-07, `restart=unless-stopped`)
and is still reachable at its public subdomain. Only the Ansible code was
retired; the service was not. See `archive/uptime_kuma/`.
`scriberr` returning 502 is the one that looks like genuine rot: it proxies to a
`scriberr-box` that is not answering, and `scriberr-box` is not in the inventory.

24
ansible/roles/caddy_site/defaults/main.yml
View file

@ -1,24 +0,0 @@
---
# Required
caddy_site_name: "" # file basename -> <name>.conf
caddy_site_domain: "" # site address line; may hold several, comma separated
# Pick exactly one shape
caddy_site_upstream: "" # "localhost:3000" -> reverse_proxy
caddy_site_root: "" # filesystem path -> root * + file_server
caddy_site_body: "" # raw escape hatch for one-off sites; wins over both
# reverse_proxy modifiers
caddy_site_resolvers: "" # "100.100.100.100" for Tailscale MagicDNS
caddy_site_headers_up: {} # {"X-Forwarded-Host": "wallet.example.com"}
# A LIST, not a dict: Ansible does not template dict *keys*, so a Jinja
# expression for the username silently passes through as literal text.
caddy_site_basic_auth: [] # [{user: "{{ x_user }}", hash: "{{ x_hash }}"}]
# Placement. This is now the only definition of caddy_sites_dir - services_config.yml
# used to carry an identical copy, which was removed as redundant.
caddy_sites_dir: /etc/caddy/sites-enabled
# Rendered site files can carry credentials (basic_auth hashes), so --diff is
# suppressed by default. Pass -e caddy_site_reveal=true to see what moved.
caddy_site_reveal: false

12
ansible/roles/caddy_site/handlers/main.yml
View file

@ -1,12 +0,0 @@
---
# Fires once at the end of the play, however many sites notified it.
# Anything later in the same play that needs the new config live must be
# preceded by `- ansible.builtin.meta: flush_handlers`.
# become is explicit because handlers do not inherit it from the task that
# notified them. headscale's play runs become: no and elevates per task, so
# without this the reload would run unprivileged and fail.
- name: Reload caddy
become: true
ansible.builtin.systemd:
name: caddy
state: reloaded

46
ansible/roles/caddy_site/tasks/main.yml
View file

@ -1,46 +0,0 @@
---
- name: Assert caddy_site parameters are sane
ansible.builtin.assert:
that:
- caddy_site_name | length > 0
- caddy_site_domain | length > 0
- (caddy_site_upstream | length > 0) or (caddy_site_root | length > 0) or (caddy_site_body | length > 0)
- caddy_site_basic_auth is not mapping
fail_msg: >-
caddy_site: '{{ caddy_site_name | default("<unnamed>") }}' needs a name, a domain and
one of caddy_site_upstream / caddy_site_root / caddy_site_body.
caddy_site_basic_auth must be a LIST of {user, hash} — Ansible does not template dict keys.
quiet: true
- name: Ensure Caddy sites-enabled directory exists
ansible.builtin.file:
path: "{{ caddy_sites_dir }}"
state: directory
owner: root
group: root
mode: '0755'
- name: Ensure Caddyfile imports sites-enabled
ansible.builtin.lineinfile:
path: /etc/caddy/Caddyfile
line: 'import sites-enabled/*'
insertafter: EOF
state: present
create: yes
mode: '0644'
backup: yes
# `validate` runs `caddy validate` against the rendered temp file before it is
# moved into place: verified on vipy that a single site fragment validates
# cleanly (rc=0, "Valid configuration") and that a malformed one is rejected
# (rc=1). A failed validate leaves the live file untouched.
- name: "Write Caddy site '{{ caddy_site_name }}'"
ansible.builtin.template:
src: site.conf.j2
dest: "{{ caddy_sites_dir }}/{{ caddy_site_name }}.conf"
owner: root
group: root
mode: '0644'
validate: "caddy validate --adapter caddyfile --config %s"
diff: "{{ caddy_site_reveal | bool }}"
notify: Reload caddy

34
ansible/roles/caddy_site/templates/site.conf.j2
View file

@ -1,34 +0,0 @@
{{ caddy_site_domain }} {
{% if caddy_site_body %}
{{ caddy_site_body | trim | indent(4, first=True) }}
{% else %}
{% if caddy_site_basic_auth %}
basic_auth {
{% for cred in caddy_site_basic_auth %}
{{ cred.user }} {{ cred.hash }}
{% endfor %}
}
{% endif %}
{% if caddy_site_root %}
root * {{ caddy_site_root }}
file_server
{% endif %}
{% if caddy_site_upstream %}
{% if caddy_site_headers_up or caddy_site_resolvers %}
reverse_proxy {{ caddy_site_upstream }} {
{% for key, value in caddy_site_headers_up.items() %}
header_up {{ key }} {{ value }}
{% endfor %}
{% if caddy_site_resolvers %}
# Use Tailscale MagicDNS to resolve the upstream hostname
transport http {
resolvers {{ caddy_site_resolvers }}
}
{% endif %}
}
{% else %}
reverse_proxy {{ caddy_site_upstream }}
{% endif %}
{% endif %}
{% endif %}
}

73
ansible/roles/datum_gateway/README.md
View file

@ -1,73 +0,0 @@
# `datum_gateway`
Builds and runs [DATUM Gateway](https://github.com/OCEAN-xyz/datum_gateway), the
solo/pooled mining gateway, on `knots-box`. The calling playbook adds two more
plays on the edge host: the dashboard via `caddy_site`, and the public Stratum
port via `socket_proxy`.
Converted from `deploy_datum_gateway_playbook.yml` (802 lines) under Plan 6. The
playbook is now 68 lines and keeps all three plays.
## ⚠ This is half of a system
The Bitcoin Knots node on the same host feeds this gateway through
`blocknotify=killall -USR1 datum_gateway` in `bitcoin.conf` — see
`roles/bitcoin_knots/README.md`, where that line was found to be missing from the
template entirely. **Changing either config means thinking about both.**
Interrupting Stratum costs mining shares. Check before any run that restarts it:
```bash
ss -tn state established '( sport = :23334 )'
```
## Two pieces of drift where the node was right
The repo and the node had diverged on values that matter, and the deployment
would have applied the repo's:
| | node (correct) | repo said |
|---|---|---|
| `datum_mining_address` | `bc1qvrj3g84…` | `bc1qdse9dsg…` |
| `pool_pass_workers` / `_full_users` | `false` | `true` |
The address is the one that would have hurt: **it is where block rewards are
paid**, and unlike fulcrum and bitcoin-knots the `Restart datum-gateway` handler
here was *never* gated, so the change would have applied immediately rather than
sitting inert. Both corrected in the vault and defaults, with notes.
Verify semantics rather than text when touching `config.json` — render it and
compare parsed JSON, because the live file is single-line and the template is
pretty-printed, so a textual diff is all noise:
```python
json.load(open('live.json')) == json.load(open('rendered.json'))
```
## `config.json` holds real secrets — diff is suppressed
The file carries `bitcoind.rpcpassword` and `api.admin_password`. `--diff`
prints rendered content, so the task sets `diff: false` by default; pass
`-e datum_reveal_config=true` to opt in.
Note `pool_pass_workers` / `pool_pass_full_users` are **booleans**, not
passwords, despite the names — they control DATUM's pool-password passthrough.
`mining.pool_address` is a Bitcoin address and public by nature.
## Expect `changed` on the compile every run
`Configure cmake build` and `Compile datum_gateway` are bare `command:` tasks
with no `changed_when`, so they always report changed and always re-run. The
build is reproducible — `Install datum_gateway binary` sees identical content and
does not replace it, so the installed binary keeps its original timestamp — but
the compile itself is wasted work on every run. That is the idempotent floor, not
drift.
## Monitoring: one variable, no product knowledge
The check tests the gateway API and records the answer in its exit code, which
systemd keeps: `systemctl is-failed datum-gateway-healthcheck.service`. Set
`healthcheck_push_url` to report anywhere accepting an HTTP ping.
Unlike the other services here, only the health-check *timer* handler was gated
by `uptime_kuma_enabled`; the main deployment restart worked throughout.

61
ansible/roles/datum_gateway/defaults/main.yml
View file

@ -1,61 +0,0 @@
# DATUM Gateway Configuration Variables
# https://github.com/OCEAN-xyz/datum_gateway
# Version - pin to a specific tag
datum_gateway_version: "v0.4.1beta"
# Directories
datum_gateway_dir: /opt/datum-gateway
datum_gateway_source_dir: "{{ datum_gateway_dir }}/source"
datum_gateway_config_dir: /etc/datum-gateway
datum_gateway_log_dir: /var/log/datum-gateway
# Binary
datum_gateway_bin_path: /usr/local/bin/datum_gateway
# Ports
# The edge host's socket-proxy/Caddy play needs this too, and a role default is
# invisible outside this role. The authoritative value for the live deployment is
# in host_vars/knots_box_local/main.yml, which outranks this; the value here is the
# protocol standard, so the role still works standalone.
datum_gateway_stratum_port: 23334
datum_gateway_api_port: 7152
# Stratum settings
datum_vardiff_min: 524288 # Minimum share difficulty (must be power of 2; OCEAN floor overrides if higher)
# Service user
datum_gateway_user: datum
datum_gateway_group: datum
# Build options
datum_gateway_build_jobs: 4
# Bitcoin node connection
# The gateway runs on the same host as Bitcoin Knots so localhost RPC works.
# datum_bitcoin_rpc_url should include http:// and port.
datum_bitcoin_rpc_url: "http://127.0.0.1:8332"
# Note: bitcoin_rpc_user and bitcoin_rpc_password come from group_vars/all/vault.yml
# Mining config
datum_coinbase_tag_primary: "DATUM"
datum_coinbase_tag_secondary: "BY ORDER OF BIP110"
# Both false on the node; the vars file said true. Corrected 2026-09-13 to
# match reality, on the same basis as datum_mining_address: the running node
# is authoritative. These control DATUM's pool-password passthrough.
datum_pool_pass_workers: false
datum_pool_pass_full_users: false
datum_pooled_mining_only: true
# --- Health check -----------------------------------------------------------
# Checks the DATUM Gateway API and records the answer in its exit code, which
# systemd keeps: `systemctl is-failed datum-gateway-healthcheck.service`.
#
# WHERE TO REPORT HEALTH — the one place to plug in monitoring. Empty means
# check, exit honestly, report nowhere.
healthcheck_push_url: ""
# Bearer token for the Gatus external endpoint. Required whenever a push URL
# is set: Gatus rejects an unauthenticated push with 401.
healthcheck_push_token: ""

15
ansible/roles/datum_gateway/handlers/main.yml
View file

@ -1,15 +0,0 @@
---
- name: Restart datum-gateway
systemd:
name: datum-gateway
state: restarted
daemon_reload: yes
# Ungated. This one carried `when: uptime_kuma_enabled | default(false)` while
# the main Restart datum-gateway handler above did not — so on this service the
# deployment restart worked and only the health-check timer restart was dead.
- name: Restart datum-gateway health check timer
systemd:
name: datum-gateway-healthcheck.timer
state: restarted
daemon_reload: yes

25
ansible/roles/datum_gateway/tasks/configure.yml
View file

@ -1,25 +0,0 @@
---
# Ownership copied verbatim from the playbook this replaces and verified
# mechanically against `git show HEAD:`.
- name: Write DATUM Gateway config.json
ansible.builtin.template:
src: config.json.j2
dest: "{{ datum_gateway_config_dir }}/config.json"
owner: "{{ datum_gateway_user }}"
group: "{{ datum_gateway_group }}"
mode: '0640'
# config.json carries the bitcoind RPC password, the API admin password and
# the pool passwords. `--diff` prints rendered content, so running with --diff
# put all of them on the terminal and into any log capturing it. Suppressed by
# default; pass -e datum_reveal_config=true when you genuinely need the diff.
diff: "{{ datum_reveal_config | default(false) | bool }}"
notify: Restart datum-gateway
- name: Create datum-gateway systemd service
ansible.builtin.template:
src: datum-gateway.service.j2
dest: /etc/systemd/system/datum-gateway.service
owner: root
group: root
mode: '0644'
notify: Restart datum-gateway

50
ansible/roles/datum_gateway/tasks/healthcheck.yml
View file

@ -1,50 +0,0 @@
---
# Everything here answers "is DATUM Gateway healthy" and records the answer. The
# Uptime Kuma specifics that used to follow — an embedded Python script creating
# monitors over the API, a /tmp credentials file, a push-URL file read back and
# parsed, and a systemd Environment= rewrite — are gone. Where it reports is now
# one variable, healthcheck_push_url.
- name: Create DATUM Gateway health check script
ansible.builtin.template:
src: healthcheck.sh.j2
dest: /usr/local/bin/datum-gateway-healthcheck-push.sh
owner: root
group: root
mode: '0755'
validate: "bash -n %s"
- name: Create datum-gateway health check systemd service
ansible.builtin.template:
src: healthcheck.service.j2
dest: /etc/systemd/system/datum-gateway-healthcheck.service
owner: root
group: root
mode: "0600"
notify: Restart datum-gateway health check timer
- name: Create datum-gateway health check systemd timer
ansible.builtin.template:
src: healthcheck.timer.j2
dest: /etc/systemd/system/datum-gateway-healthcheck.timer
owner: root
group: root
mode: '0644'
notify: Restart datum-gateway health check timer
- name: Reload systemd daemon after health check units
systemd:
daemon_reload: yes
# Ungated: enabling a timer is deployment, not monitoring.
- name: Enable and restart the datum-gateway health check timer
systemd:
name: datum-gateway-healthcheck.timer
enabled: yes
state: restarted
daemon_reload: yes
# Arms the timer and smoke-tests the check. See roles/bitcoin_knots/README.md for
# why restarting the timer alone is not enough with OnBootSec + OnUnitActiveSec.
- name: Run the DATUM Gateway health check once to arm the timer
command: systemctl start datum-gateway-healthcheck.service
changed_when: false

74
ansible/roles/datum_gateway/tasks/install.yml
View file

@ -1,74 +0,0 @@
---
- name: Install DATUM Gateway build dependencies
apt:
name:
- cmake
- build-essential
- git
- libjansson-dev
- libmicrohttpd-dev
- libsodium-dev
- libcurl4-openssl-dev
# Runtime-only (netcat for health check)
- netcat-openbsd
state: present
update_cache: yes
# ===========================================
# System User and Directories
# ===========================================
- name: Create datum system user
user:
name: "{{ datum_gateway_user }}"
system: yes
shell: /usr/sbin/nologin
home: "{{ datum_gateway_dir }}"
create_home: no
comment: "DATUM Gateway"
- name: Create DATUM Gateway directories
file:
path: "{{ item.path }}"
state: directory
owner: "{{ item.owner }}"
group: "{{ datum_gateway_group }}"
mode: "{{ item.mode }}"
loop:
- { path: "{{ datum_gateway_dir }}", owner: root, mode: "0755" }
- { path: "{{ datum_gateway_source_dir }}", owner: root, mode: "0755" }
- { path: "{{ datum_gateway_config_dir }}", owner: "{{ datum_gateway_user }}", mode: "0750" }
- { path: "{{ datum_gateway_log_dir }}", owner: "{{ datum_gateway_user }}", mode: "0750" }
# ===========================================
# Build from Source
# ===========================================
- name: Clone DATUM Gateway repository at {{ datum_gateway_version }}
git:
repo: https://github.com/OCEAN-xyz/datum_gateway.git
dest: "{{ datum_gateway_source_dir }}"
version: "{{ datum_gateway_version }}"
force: yes
register: git_clone
- name: Configure cmake build
command: cmake . -DCMAKE_BUILD_TYPE=Release
args:
chdir: "{{ datum_gateway_source_dir }}"
- name: Compile datum_gateway
command: make -j{{ datum_gateway_build_jobs }}
args:
chdir: "{{ datum_gateway_source_dir }}"
- name: Install datum_gateway binary
copy:
src: "{{ datum_gateway_source_dir }}/datum_gateway"
dest: "{{ datum_gateway_bin_path }}"
remote_src: yes
owner: root
group: root
mode: "0755"
notify: Restart datum-gateway
# ===========================================
# Configuration

6
ansible/roles/datum_gateway/tasks/main.yml
View file

@ -1,6 +0,0 @@
---
# import_tasks, not include_tasks: static imports stay visible to --list-tasks.
- ansible.builtin.import_tasks: install.yml
- ansible.builtin.import_tasks: configure.yml
- ansible.builtin.import_tasks: service.yml
- ansible.builtin.import_tasks: healthcheck.yml

16
ansible/roles/datum_gateway/tasks/service.yml
View file

@ -1,16 +0,0 @@
---
- name: Reload systemd daemon
systemd:
daemon_reload: yes
- name: Enable and start datum-gateway
systemd:
name: datum-gateway
enabled: yes
state: started
# ===========================================
# Health Check Script + Systemd Timer
# ===========================================
# ═════════════════════════════════════════════════════════════════════════
# DEPRECATED — Uptime Kuma was decommissioned on 2026-09-11.

35
ansible/roles/datum_gateway/templates/config.json.j2
View file

@ -1,35 +0,0 @@
{
"bitcoind": {
"rpcuser": "{{ bitcoin_rpc_user }}",
"rpcpassword": "{{ bitcoin_rpc_password }}",
"rpcurl": "{{ datum_bitcoin_rpc_url }}",
"notify_fallback": true
},
"stratum": {
"listen_port": {{ datum_gateway_stratum_port }},
"vardiff_min": {{ datum_vardiff_min }}
},
"mining": {
"pool_address": "{{ datum_mining_address }}",
"coinbase_tag_primary": "{{ datum_coinbase_tag_primary }}",
"coinbase_tag_secondary": "{{ datum_coinbase_tag_secondary }}"
},
"api": {
"admin_password": "{{ datum_gateway_admin_password }}",
"listen_port": {{ datum_gateway_api_port }},
"modify_conf": false
},
"logger": {
"log_to_console": true,
"log_to_file": true,
"log_file": "{{ datum_gateway_log_dir }}/datum_gateway.log",
"log_rotate_daily": true,
"log_level_console": 2,
"log_level_file": 1
},
"datum": {
"pool_pass_workers": {{ datum_pool_pass_workers | lower }},
"pool_pass_full_users": {{ datum_pool_pass_full_users | lower }},
"pooled_mining_only": {{ datum_pooled_mining_only | lower }}
}
}

22
ansible/roles/datum_gateway/templates/datum-gateway.service.j2
View file

@ -1,22 +0,0 @@
[Unit]
Description=DATUM Gateway - Bitcoin Mining Gateway
Documentation=https://github.com/OCEAN-xyz/datum_gateway
After=network.target bitcoind.service
Wants=bitcoind.service
[Service]
User={{ datum_gateway_user }}
Group={{ datum_gateway_group }}
Type=simple
ExecStart={{ datum_gateway_bin_path }} --config {{ datum_gateway_config_dir }}/config.json
Restart=on-failure
RestartSec=10
StandardOutput=journal
StandardError=journal
# Prevent config from being read by other users
ReadWritePaths={{ datum_gateway_log_dir }}
ReadOnlyPaths={{ datum_gateway_config_dir }}
[Install]
WantedBy=multi-user.target

15
ansible/roles/datum_gateway/templates/healthcheck.service.j2
View file

@ -1,15 +0,0 @@
[Unit]
Description=DATUM Gateway Health Check
After=network.target datum-gateway.service
[Service]
Type=oneshot
User=root
ExecStart=/usr/local/bin/datum-gateway-healthcheck-push.sh
Environment=HEALTHCHECK_PUSH_URL={{ healthcheck_push_url }}
Environment=HEALTHCHECK_PUSH_TOKEN={{ healthcheck_push_token }}
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target

Some files were not shown because too many files have changed in this diff Show more