diff --git a/.gitignore b/.gitignore index 471cd6f..6c0a084 100644 --- a/.gitignore +++ b/.gitignore @@ -1,23 +1,21 @@ # OpenTofu / Terraform .terraform/ .tofu/ +.terraform.lock.hcl +.tofu.lock.hcl terraform.tfstate terraform.tfstate.* crash.log +*.tfvars +*.tfvars.json -# Provider credentials only. Non-secret infra config (the vms map) is committed -# as *.auto.tfvars, and *.lock.hcl is committed on purpose so provider hashes -# are pinned. -terraform.tfvars -terraform.tfvars.json -*secrets.auto.tfvars - +test-inventory.ini +inventory.ini venv/* .env -# Secrets are ansible-vault encrypted and ARE committed. -# Anything matching *_secrets.plain.yml is a working decryption — never commit those. -*_secrets.plain.yml - -# Vault password — never commit -ansible/.vault_pass \ No newline at end of file +# Secrets and sensitive files +*_secrets.yml +*_secrets.yaml +secrets/ +.secrets/ diff --git a/01_infra_setup.md b/01_infra_setup.md index a8ba9c0..6ef0978 100644 --- a/01_infra_setup.md +++ b/01_infra_setup.md @@ -162,12 +162,6 @@ Note that, by applying these playbooks, both the root user and the `counterweigh ```bash cp ansible/infra_secrets.yml.example ansible/infra_secrets.yml ``` - > **DEPRECATED (2026-09-11).** Uptime Kuma has been decommissioned. The server - > deployment was removed from this repo; what it monitored and how it was set up is - > preserved in [`archive/uptime_kuma/`](archive/uptime_kuma/). The monitoring blocks in - > the playbooks are kept but inert (`uptime_kuma_enabled: false`) so the check logic - > survives for whatever replaces it. The credentials below no longer exist in the vault. - * Edit `ansible/infra_secrets.yml` and add your Uptime Kuma credentials: ```yaml uptime_kuma_username: "admin" diff --git a/02_vps_core_services_setup.md b/02_vps_core_services_setup.md index 76eefd5..1423d19 100644 --- a/02_vps_core_services_setup.md +++ b/02_vps_core_services_setup.md @@ -49,12 +49,6 @@ Checklist: ## Uptime Kuma -> **DEPRECATED (2026-09-11).** Uptime Kuma has been decommissioned. The server -> deployment was removed from this repo; what it monitored and how it was set up is -> preserved in [`archive/uptime_kuma/`](archive/uptime_kuma/). The monitoring blocks in -> the playbooks are kept but inert (`uptime_kuma_enabled: false`) so the check logic -> survives for whatever replaces it. The credentials below no longer exist in the vault. - Uptime Kuma gets used to monitor the availability of services, keep track of their uptime and notify issues. ### Deploy @@ -204,58 +198,9 @@ Checklist: - [ ] SSH cloning works after setting up your SSH pub key -## Phoenixd - -phoenixd is the server version of the Phoenix wallet: a self-custodial Lightning node that leans on the ACINQ node for liquidity and channel management. It is the Lightning backend that LNBits talks to, so deploy it first. - -It is not exposed to the internet. Its HTTP API is bound to `127.0.0.1` and only local consumers (LNBits, running on the same host) use it. There is no subdomain and no Caddy config for it. - -### Deploy - -* Review `ansible/services/phoenixd/phoenixd_vars.yml`. The things worth a look: - * `phoenixd_version`: pinned release tag. - * `phoenixd_auto_liquidity`: how much inbound liquidity phoenixd buys when it needs some (`off`, `2m`, `5m`, `10m`). This costs sats, it is not free. - * `phoenixd_http_bind_port`: the loopback API port LNBits will point at. -* No new secrets are needed: phoenixd generates its own API password on first start. -* Run the deployment playbook: `ansible-playbook -i inventory.ini services/phoenixd/deploy_phoenixd_playbook.yml`. -* The playbook will: - * Install `phoenixd` and `phoenix-cli` into `/usr/local/bin` - * Create the `phoenix` system user and a `0700` data dir at `/opt/phoenixd/.phoenix` - * Create and start the `phoenixd` systemd service - * Create a health check timer that runs `phoenix-cli getinfo` every minute - * Register a push monitor named `Phoenixd` in Uptime Kuma under the `services` group - -### Back up the seed - -**Do this immediately after the first deploy.** The seed is the only thing that recovers the funds. - -* Run the backup playbook: `ansible-playbook -i inventory.ini services/phoenixd/setup_backup_phoenixd_to_lapy.yml`. -* This gpg encrypts `seed.dat` and `phoenix.conf` to Lapy daily and keeps 14 days. It does not stop the node and it does not back up the channel database on purpose: phoenixd keeps channel state with its peer, and restoring a stale channel db to a live node can force close channels and cost you a penalty. -* Also write the 12 words down offline, once: `sudo cat /opt/phoenixd/.phoenix/seed.dat`. - -### Operate - -* Status: `sudo systemctl status phoenixd`, logs: `sudo journalctl -u phoenixd -f`. -* CLI: `sudo PHOENIX_DATADIR=/opt/phoenixd/.phoenix phoenix-cli getinfo` (also `getbalance`, `listchannels`, `createinvoice`). If you changed `phoenixd_http_bind_port`, add `--http-bind-port ` before the subcommand. -* API password, needed to wire anything up to the node: `sudo grep '^http-password=' /opt/phoenixd/.phoenix/phoenix.conf`. - -### Restoring on a fresh host - -* Deploy phoenixd but stop the service before it generates a new seed, or just drop the file in before the first run: - * `sudo mkdir -p /opt/phoenixd/.phoenix` - * `echo "your twelve words here" | sudo tee /opt/phoenixd/.phoenix/seed.dat` - * `sudo chown -R phoenix:phoenix /opt/phoenixd/.phoenix && sudo chmod 600 /opt/phoenixd/.phoenix/seed.dat` -* Then run the deployment playbook. Never run two nodes on the same seed at once. - -Checklist: -- [ ] `phoenix-cli getinfo` returns a node id -- [ ] The `Phoenixd` monitor in Uptime Kuma is green -- [ ] The seed is backed up to Lapy *and* written down offline - - ## LNBits -LNBits is a Lightning Network wallet and accounts system. It uses the phoenixd node deployed above as its Lightning backend. +LNBits is a Lightning Network wallet and accounts system. ### Deploy @@ -269,7 +214,6 @@ LNBits is a Lightning Network wallet and accounts system. It uses the phoenixd n * From that point on, you can configure through the Web UI. * Some advice around specifics of LNbits: * The default setup uses a FakeWallet backend for testing. Configure a real Lightning backend as needed by modifying the `.env` file located or using the superuser UI. - * To use the phoenixd node deployed above, set the backend to `PhoenixdWallet` with endpoint `http://127.0.0.1:9740` and the API password from `/opt/phoenixd/.phoenix/phoenix.conf`. The deployment playbook does not do this for you, since flipping the wallet backend on a live instance is not something to do behind your back. * For security, disable the new users registration. ### Set up backups to Lapy @@ -285,6 +229,22 @@ LNBits is a Lightning Network wallet and accounts system. It uses the phoenixd n * Start it up again. +## ntfy-emergency-app + +ntfy-emergency-app is a simple web application that allows trusted people to send emergency messages via ntfy notifications. Perfect for situations where you need to be alerted immediately but don't want to enable notifications on your regular messaging apps. + +### Deploy + +* Decide what subdomain you want to serve the emergency app on and update `ansible/services_config.yml` under `ntfy_emergency_app`. + * Note that you will have to add a DNS entry to point to the VPS public IP. +* Configure the ntfy settings in `ntfy_emergency_app_vars.yml`: + * `ntfy_emergency_app_topic`: The ntfy topic to send messages to (default: "emergency") + * `ntfy_emergency_app_ui_message`: Custom message displayed in the web interface +* Ensure `infra_secrets.yml` contains `ntfy_username` and `ntfy_password` with the credentials the app should use. +* Make sure docker is available on the host. +* Run the deployment playbook: `ansible-playbook -i inventory.ini services/ntfy-emergency-app/deploy_ntfy_emergency_app_playbook.yml`. + + ## Headscale Headscale is a self-hosted Tailscale control server that allows you to create your own Tailscale network. diff --git a/03_vm_disk_enlargement.md b/03_vm_disk_enlargement.md deleted file mode 100644 index 260a3ea..0000000 --- a/03_vm_disk_enlargement.md +++ /dev/null @@ -1,69 +0,0 @@ -# 03 VM Disk Enlargement - -How to enlarge the disk of an existing Proxmox VM that was provisioned via the tofu project. - -## Overview - -Since the tofu VM resource uses `ignore_changes = all` in its lifecycle block, disk resizes are performed manually through the Proxmox UI and then expanded inside the guest OS. You should still update the tofu vars to keep them in sync with reality. - -## Step 1: Update disk size in tofu vars - -* Edit `tofu/nodito/terraform.tfvars` and bump the `disk_size_gb` value for the VM you want to resize. -* This won't trigger any actual change (because of `ignore_changes = all`), but keeps the declared state consistent so future readers know the real disk size. - -## Step 2: Resize the disk in Proxmox - -* Open the Proxmox web UI and navigate to the VM. -* Go to the **Hardware** tab. -* Select the **Hard Disk** entry (scsi0). -* Click **Disk Action** > **Resize**. -* Enter the amount of additional space you want to add (this is the increment, not the total). For example, to go from 10 GB to 30 GB, enter `20`. -* Click **Resize disk**. - -The VM does **not** need to be stopped for this operation. Proxmox will grow the underlying block device while the VM is running. - -## Step 3: Expand the partition and filesystem inside the VM - -SSH into the VM and run these steps as root (or with `sudo`). - -### Check the current layout - -```bash -lsblk -``` - -You should see that the disk (`sda`) now shows the new total size, but the root partition (`sda1`) still has the old size. - -### Grow the partition - -Install `cloud-guest-utils` if `growpart` is not available: - -```bash -apt install -y cloud-guest-utils -``` - -Then grow the root partition to fill all available space: - -```bash -growpart /dev/sda 1 -``` - -> Note the space between the device and the partition number. - -### Resize the filesystem - -For ext4 (the default for Debian cloud images): - -```bash -resize2fs /dev/sda1 -``` - -This works online; no reboot is required. - -### Verify - -```bash -df -h / -``` - -The root filesystem should now reflect the new size. diff --git a/README.md b/README.md index 89ef092..f343cbc 100644 --- a/README.md +++ b/README.md @@ -6,12 +6,6 @@ My repo documenting my personal infra, along with artifacts, scripts, etc. Go through the different numbered markdowns in the repo root to do the different parts. -## How to edit secrets - -`ansible-vault edit ansible/your_file_with_secrets.yml` - -Assumes that you've set `ansible/.vault_pass` with `chmod 600`. - ## Overview ### Services @@ -22,7 +16,7 @@ Assumes that you've set `ansible/.vault_pass` with `chmod 600`. + Plan install + File based config + Crossbackup to Desky via rsync -* ~~Uptime Kuma~~ — decommissioned 2026-09-11, see `archive/uptime_kuma/` +* Uptime Kuma + Deployed on Vipy + Crossbackup to Desky via rsync * Vaultwarden diff --git a/ansible/ansible.cfg b/ansible/ansible.cfg deleted file mode 100644 index aa10181..0000000 --- a/ansible/ansible.cfg +++ /dev/null @@ -1,14 +0,0 @@ -[defaults] -inventory = inventory.ini -roles_path = roles -collections_path = collections -interpreter_python = auto_silent -stdout_callback = yaml -retry_files_enabled = False -host_key_checking = True -forks = 10 -vault_password_file = .vault_pass - -[ssh_connection] -pipelining = True -ssh_args = -o ControlMaster=auto -o ControlPersist=300s diff --git a/ansible/group_vars/all/main.yml b/ansible/group_vars/all/main.yml deleted file mode 100644 index e9ab735..0000000 --- a/ansible/group_vars/all/main.yml +++ /dev/null @@ -1,84 +0,0 @@ -new_user: counterweight -ssh_port: 22 -allow_ssh_from: "any" -root_domain: contrapeso.xyz - -# Uptime Kuma was decommissioned on 2026-09-11. The monitoring blocks in the -# playbooks are kept deliberately — the check logic is meant to be rewired to -# whatever replaces it. This flag keeps them inert until then. See archive/uptime_kuma/. - -# age recipient for all backup artefacts -age_backup_recipient: "age192wwdaseqej2ggwyp884gtm05c396anp7chr0vr8m47g50fahpyqr9fsza" - -# Public key small-backups-box pulls with -# Authorised on each source host for an unprivileged, dedicated user only -backup_pull_public_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOfIixKMhA9z+Nvyx6ToZIniC8aEgyiInRiboaTTemgX offsite-backup-pull" - - -# ───────────────────────────────────────────────────────────────────────────── -# Subdomains. Global because the edge host proxies for services that live on -# other machines, so no single inventory group covers the readers. Combine with -# root_domain above to build an FQDN. -# -# Moved here from services_config.yml, which 30 plays had to remember to name in -# vars_files: - a file everyone must opt into is a file someone will forget. -# ───────────────────────────────────────────────────────────────────────────── -subdomains: - # Monitoring - gatus: status - - # VPN infrastructure (spacey) - headscale: headscale - - # Core services (vipy) - vaultwarden: vault - forgejo: forgejo - lnbits: wallet - - # Secondary services (vipy) - personal_blog: pablohere - - # Memos (memos-box) - memos: memos - - # Mempool block explorer (mempool-box, proxied via vipy) - mempool: mempool - - # DATUM Gateway dashboard (knots-box, proxied via vipy) - datum_gateway: datum - - # Monero restricted RPC (monerod-box, proxied via vipy). Not a dashboard: - # this is the JSON-RPC endpoint Feather points at. Caddy terminates TLS and - # monerod's own Digest auth does the authentication. - monero: monero - - # Monero block explorer (monerod-box, proxied via vipy). Fully public and - # unauthenticated, like any block explorer - the data is the public chain. - monero_explorer: exmrplore - - # Navidrome music server (music-box, proxied via vipy). No basic auth: - # Navidrome has its own login, and the Subsonic API the mobile clients speak - # has nowhere to carry a second credential. Same call made for monero above. - music: music - - # Jar Lid Archive staging (jar-lid-archive-staging-box, proxied via vipy). - # Public with no basic auth: the one-off photo links are for people without - # an account, and the app has its own login for everything else. - jar_lid_archive_staging: jars-staging - -# Read by plays across several groups, so global rather than group_vars/. -headscale_namespace: counter-net - -# ───────────────────────────────────────────────────────────────────────────── -# Domains whose registration expiry is monitored (infra/402_public_monitoring). -# -# Registration renewal is a manual act at the registrar, and losing a domain is -# not recoverable in the way losing a host is - so these are checked daily and -# alarm with two weeks of runway. -# -# root_domain is the estate's own domain; the rest are domains we own that are -# served from it or from a host in the inventory. -# ───────────────────────────────────────────────────────────────────────────── -monitored_domains: - - "{{ root_domain }}" - - arbret.com diff --git a/ansible/group_vars/all/vault.yml b/ansible/group_vars/all/vault.yml deleted file mode 100644 index a3bdbd7..0000000 --- a/ansible/group_vars/all/vault.yml +++ /dev/null @@ -1,229 +0,0 @@ -$ANSIBLE_VAULT;1.1;AES256 -62636131396363616337633131353962356639613766396163383633353937303334306661656662 -3734393736396462323634396236386339623336663033310a366563623930616331616333343337 -35613165633465653630626139393930333263323238353530663065613734646334663833323638 -3364613961656433360a633136376664356262633338356163363631656665623562653034376338 -33376661633533653766636135616261633133313338383238313335363461373064646261646262 -36383030393235653566666537373232666538616435356332373530323536386535666532656461 -32363638613131616137383133613966353232323537373938306232643738353163306336613238 -36396539623730333232653338393636333464353536623337323935613461396136313131326133 -35646561323165323431633039366435623965396363353535343165656631626166393234643639 -37396139376531653839646233313837373136613337326439373062393030393130623638633334 -30343266323938333333633064376632663737343332306533653232653837376637656166393935 -32353231333532326436383532646336356162333738333037336337363133396465363937343330 -38333335316663333232663966376234323261353036363262353362643061396533646139633539 -35343065653763393032346333373734376133613039396161343438666566613530653463623239 -63643762663933653333363665383634373762303462313064323336353862363530343831643336 -64653030386438326633356138316338353838316562643138353065623262373639326638646365 -38653637633731326364653834393132336637663135366230316437316365363366343530316236 -31616466383462626266316330303532346536653435643234646534373064313564343661383139 -33363864386266313563643264306537303066366463633062346237653963383131633436303234 -32613462616261623632343566653836623539666539643865656366313264616635616638386237 -65396135303739616635356332353336303861373965323834303261396162393235663963383963 -38376136616537396162333566326465323761353030393961623262336165373931633333353330 -30313137326661363137366265376132333063326130356334616537613330373664336462666137 -39366336356332613836343039623330343231613530373463316435323034653732333362336362 -65313763393863303935353430333430333632633036363932396238303464353465363032613536 -66663766356262313135653964636437396665323461386337303731326166386331363134373133 -65636436323336393034666638646632643336363031306163643266323333633031653734376662 -61313734666436343062373037336262633333356462303562346536663862393431373334323363 -32616663383135613933303430323030656130333264343231393164336536326464623964313131 -38366139393732376665373537376563656332626530636363613433666237653835306433363031 -62323734393933363064636464616132383636353965366365386261303530646233336266326366 -63663332656233393232376235373836306432373434316532356264613733643530373230656135 -32626463613663653336303064306563323133383834313936316366623931396537396165333563 -31633264663030643164353138653565363533373336333833386330313335636230353030613061 -39333462663965333565363730313731623538306436303337653361663035386130333665623463 -63643536653831393361336665616361653764356163363832656463363038343330636239356136 -33316238343538306538306564326337396633666166333863646462326164376365303836323530 -33636230376635323166343631373530623437366461393638383532656564303233383865626332 -65343063666462656236346634396464386437616332303437333932663966333162623463353733 -66646563313363633361393862616563353232656136376138306239383561386636333231373436 -30333336363831666530363132613862306334656133666231633063313263373336373562353238 -35323166366163343430323439646164646161623264663966643334653237366335666464373131 -37303663643638316334336665303562363266373130656230356336303531323161393365383430 -33366165613236363138636237363766366366363539643361323664613461383965616333346661 -30653436373030333232393366326435653061383963316632613163346165633335643731633961 -34396261663939636237383439613461303735393233653164363837333235303132626436626132 -65383661353261383437366235316533343664383033333966333531353131666530303439353464 -39396337373331663634333839656362653761306130363136353861653033303430323938663339 -64376636393935313633336464333930326432383164376139323533356438393761376462323764 -35313339363032373463613335666438306139396634386466323039663735643563333664636364 -63623636333837366165373637613431613962343966613934663065383333353761336331343466 -31393466393661336539383161613963313661313366316437393835303564393335336630366434 -34383434613637356232383863366465613133303964393033646239303932663563333535333931 -32326630336233663131613063363533333133336463366466313362333331633430353433336131 -39346639306630326336306537663262386264383938643362633762306364643435303533633635 -33333364333466393038353434363334633965336533313135396436336432393333653832363531 -38356633633632346263663935326264396632626263326663303861306363626134313761396233 -31396435653463303530633739323930396438366166366137386566346338366566393961643865 -33613365663863326331616163326666346633663136663330623331373439343330643261623665 -63656439653935653637353134623865633666343136396361363835363861366162323234363737 -62333135613539643261363462373534623631616438663035396162646161373665383364656438 -61653337316537336337313231326639386665636263383566303164353535643238386563366661 -63633037386536656332623331316361613064343833623766366665643232636265363731346538 -30353037393365353333393261666532303035393665356165633263663066623464313935343536 -61636462613738623861386534353332613734643534383430613564616235313432303566633963 -61616662633638336261326365373735343264393432323965316330373430656635323937346239 -61363433303235613630666331626361633938376564353831623664323334626233383133656235 -66643732613432376561636436306236663734316136643735376564383362356262666131373038 -66393931623637333237646332356531326533613932393232306461623235376664383334366562 -30356365613331646530616431306330363063623234396265633936343832663036333364646337 -32376138316533643839323766613838323265653632643430356234643665303666386664323031 -66343764643038666237343536393737623862663363666464396535336335616136373930336433 -37303862343834306266366335623036653738393234626434343337646338643631313662666634 -64383034333936643164323736303639383237623235356639383165646433303935343062396633 -32633436396530396330336238313466316334386531383962336165336461633836643963333164 -61663434326463623737333530353965366361616639396238386565393533363339333235643761 -64663230343964313435663335333039616165623466376638616534356638396538346234663436 -38636233643265353561356666306165616462373639306135393866386433366536373938353764 -64613637616431623963393534316631616439383161373135393161313332653134643237353037 -64313966636566343766326132376336623065363937303633363634356438396161303437656636 -34353562316663666430396630343665646165663061633235383232323363636637383465343538 -37303733653166316564666163333963656265623264656666613539666166636466333866316634 -32383466623962356338353334613166363932633330633836353037336139383732613766653030 -32373064363766303436643331363732326665646334316663323562376561353063663837346537 -39386531653762393263363261353136313961626663643865616266613834333664376664643761 -31393661393634383030643462643062333635366135313564666336613965623730313138646365 -62396565623636656335343737303732386161643537663236393831336336363764643762613164 -66396262363265386164346665383961333332346139363337396639343731346131373932653332 -62386564343639343930306164376432653961386165353333646134306231383334316366613337 -62326339366130653333643333616636653937323130396563636666373832383066343666376465 -37373932326538666664313534633661373035333765323733643064373865356338376130626137 -32373763373734383539313839666665653536356335373933626361646139303235656132343033 -30346435396637313234346436303964393863303735323864613038643136343237376138346334 -31333637623339343263646533383434393862626134303463663832613836373565373164326435 -62663461366639353537376664313463666437336264323165333734386362313430653839353437 -66373130613338333034333663366564616562363632336236363237653536663963326230633136 -30383635356362616263653637356234613537363364313437636662623038643537666431653533 -35383165366537663839633638663938383836616366313831643630306435616431643464393562 -30613634303266383535373965643534623338623466633365643839396564613236663631343439 -38346164653764383333393432343565326537313639336630366163653937343566303938653665 -32316262653630623638373964623430633466333835366439666530356535636630643063393562 -37643966346437396463383133306466336365303137313932376536666462313262343131623161 -64376261633964623761623666636134313038396564633436383839613730396139636530366333 -36346338313330353437343464383232666338613337336464653663613662383230643939623136 -63383163663062643436326166313432306166646264623933363935663862373737383434636234 -36336464353934383034373166373733343133353165623661643333663332303266643538666530 -35653063366463333961303761633763373266636639356239653636653061303865636332636632 -35633333623236313566376131333830353734303665383433366239353562666664643662323065 -35363164393138636135663462333131313632626535653239323463653634623339393061663861 -61626531383964373730396536663434306531383437333335316166663864653561643939643837 -30396137393665316239633330343161643562353735316330346565343366616536613330306664 -32343537363466386664353638623034616631346366633862353538386566616137653162303466 -38326531666130373864303235636261616565386264663335663862616265666366333739326631 -38323766663864353331396334613735613436363263653938646661623735393038393661613266 -37313932313434313735653330666639623730666462396433356364393931313632383634353334 -66346539663836636139393839653237623932333537653666333832626333396235383731373237 -33616634336662396538323436613166616339636235663933313437353737376539663438343164 -34383138363833616364393537333331396566643562346464373134313433333931343939336261 -32653231656265326662653639376365643234383265363466393864363562633066376664373261 -62343233633034633261613839333461336639643136656166326534383366343239323262303630 -36336265313233386361366139623663393638613538313265366462356463633862336364623962 -31336363653964303466626239356435666562386535663832373333353062343034646432383832 -35383831383331393962376665613634303634333864613066323530346134303934386564313666 -36656162343965306461306132626336353435643334303531633736613633646536353433626437 -63373033323434616539363035626632653663613838653232303039356633636264643432323236 -35313461616231613563393436633933363935306236316466313166636564323139393137323534 -34613432306237356663356434656331626161663138663665613663313263366162623561616234 -63663865623765643236343035353739633266623366393139333261353539313462306132326463 -61636665353861663838396236643563353134663831643061313234653830643864386131366562 -65643866633632353932313733363163373237333635623736666139393766393634626430333039 -63353536383138313165366362633230383137626364326333353037353736633664663738343766 -35653536626531393832346634666131333730643465656136653433373134306461383231613963 -63393366663137653635383830613239333337626432666436666363343637346331303731626137 -32326366643439333966616165636632396366383364313832356636643032306131396534653339 -39383732633132646464376233383139653865393061623763313865343964613031643965653939 -36633963633033313530666661363331353836666137386536396533306165646131363235303336 -32663537316164653935663034313833633135333837636331303363653864313231353734633461 -31383035376639393335346531626266323730363934393062366263313136343665303036393933 -62643965653932633334366363633763353862333335643265386533383062346266636639653064 -31613464393866323363393731363766363063303234323861363166373430383266663234336332 -61386561363562393736343939356434386464626134353634303931396662313665336531373235 -36366463623562393139383033303630663062323564626135613837623335633763613435353964 -30323161666264646630623532326431616135396133323838653736356338616339656638386464 -31363331303164373031326161353666613337653536656330653865653062316636656237376562 -34386431346330636464616136336531333331663865656266333964316261613265306434653038 -35333935626461353033323733326634373036663133326336373665383233343636396466313566 -37396462363764326361363561323633633537353165396362366166643362636639646536633065 -34643633383264663137363264663964626335643938333933306232363466346430666230353561 -37643836623262343664323664646161353230393435393033306536393531323166363739643432 -32396236363839313131306266383866323861343537373738326265326637636531626431373335 -65386564333961346633663633313737396565383431373265656131666638626462653335393330 -33356362613535313063316330306330656263353463313538643232663136636136333539363663 -30363066653633306537626664323339326565633134373831636633623930343635396362393561 -35303237616335653263643132386237386639373939363536356238656163343133613065376531 -66386361643465633938353438333730323933366465643533323738653762353437353136373134 -36636533313665633362323361373839633839653338633336373561386663343361313733393031 -34626566386635346439386564643261373363323466633839316564336137333636353364613062 -37343038373933316665376539396263393765613034306139393962346563353065316638613738 -66393739663736383265383735393331663739306462393939626661656534366665333132623831 -34623962393535353665343537643938353033343634646664323535393762366639376132626362 -34373230663937336161333934333935343564336237313837316363623665313564333134393133 -64653562313036303137316436633065643062303335643266366361316163306433376565363034 -63633830633937343530653537663465336563373435376366623537383165663033623539323333 -66313238363266653065623532616331623635643038353032303266373565646136646538623530 -31613038646331313738373634353737613435303964316161323938316261303237653362383233 -64616133623432353139323835366238633562336161626564633530623035336232346465386332 -64343036366430646161633461306661663564343132346564346633303934663539643239323536 -61386238663037373534336535396230613565303566626631323138643365656132333635613532 -34613633313633366638376662366464333664636538373133373939613565356637383863613035 -30353261643161633734386331623765326664616636333965363161343661613937356636363864 -38633132326237623964643734613165613133633832396633393432303535623237663864373135 -34643964636565353635663633623933643164303734353766643665313933343565373530333833 -36636165323030336562313764633039313165323630313665353366313135653636313338663233 -66663431346234333364666661643165653963616230376130663062616366383238666561326364 -36636530333633653763363435373534616435326533383030386164646130646164323262663330 -32323163623534613036316134366463616639633934396363653539343830643239396633333064 -31323264663866353530343033323066393332323161363031626136336531666566343332663230 -62653635313763313461376236663432306130383264313763356531646637363432396365613766 -37323935623031656438643330616563343632376230666236356534346466323533376631346564 -63333536313132393164356337643465373737613939666666363361303636333436306632626166 -63396637313235396236646237383162366531303133343962333639326139613231663737313532 -37373732383734646330326662636330646135666462343133663336666462643264313565313933 -31373035626564396665323235616138326539336133636232633034383262396466383138626264 -61663166356162616465353565646166636238376464383033303636623334666236396637633635 -30643861623632663632616130316630333234666439393030303961666239613530333937656563 -30356363313733393663376433346138383966616465343934663734666437653863663133643437 -64613236633736333237383834323338313431326634333730656536323138626438306231333035 -35373936646264363337613134396337613039366363323638656437656564623831666461373064 -66633232623666383738613834663938626438653430376534306164646363306663643362623137 -61376433636336346266393532363131613938356262303564393636613833643738646230323938 -32353763326631656561666533366236656134343638393435666437646231633463316232666563 -63653635393334336266653265326239626531323035653930323438393235633962386438306133 -34333537326530383363343139363461646339383064396637663465393961306139643761333538 -39346362663830373535386665386533643435666438656638373861376537383538366331323461 -66363138326432363432343335636434656139386161313162643838343566356264313734613630 -64366162393932313131306533633666393564613835363664306639343161643339316537346338 -64303764366136303737666464393463343261383538333563313164343666383132333761306463 -62656138633939383239343533356136383735663834323763623437386434313333306230326237 -38373666376332646635366233646237373431336531366432323130353034666265303236363937 -36356536653465326139306635633032326535373734333664656339386266363365303139623463 -38396664613639613737343062326538653835373861393939656636366337366134393236346463 -33306462633639626463326230656338636664613730373731323531636337333062646233306139 -61303661313764623565383265353061616638663636336635356364383933313363663330306638 -39613637363935626363633635333533323735623963346334303661353030313066633636396361 -65303738383131363237653866636637346464336334646362326538303232333661623038366332 -36323736333538373931616638303030613765326165663066333131373734373766343066626131 -32626462653538376431373635363835643036376230623061633963393065323262353931613165 -39633136656666653231353036353034333634646266386533316232303462663161613239383931 -64623034333137376134373235626165356466323931333131376238306536633861366430616362 -63636233653339376665346635656239306161663538656166363563326564643661316563346666 -38666161366430663234396237316534323263363436613435643436376535643430323432333639 -32373436353936623966626263653037366263646232336665626533343366366530393535663733 -32616639386433303032376565626535393466356136316662626364376666373934613166303233 -39336136666437323766336430653433393134656230663961666239306639323039396432653936 -65363736343637636538646130616362373965376437333762383138393134353332663734326565 -31303136613739643965623437643663653339666638363861343164663538313234313330663738 -37646362636130663431633762323634613964666166323962346234643765336639613934393938 -64323239343832326564333731663961626661643936643233336662303335663861653331326161 -64396564316465353163633066383562643636363562376136363733646439356530643432336139 -63366265353435323565333161386563636134306261653366623036396266356137333364343930 -30313465313834623530663436623333663262646331653832656266353834323463666332643130 -32656539393838393766643962396133333038343166336166336539333434363433656265313339 -31613238643339656130396461306134646666386430313636376633333465356230333433303666 -37313534343037336432373639303534633235623264363064653635333837656132643665396132 -66336463313139336163623637383164643237636134653837663062303238633237306562616161 -62613536653036333431346131363864393730333432383632333034646361373731316431343662 -39656264323965653738316264636361343361303466326336663438643430366235643165323632 -34313331353530386266 diff --git a/ansible/group_vars/nodito_vms.yml b/ansible/group_vars/nodito_vms.yml deleted file mode 100644 index 1072b51..0000000 --- a/ansible/group_vars/nodito_vms.yml +++ /dev/null @@ -1,24 +0,0 @@ ---- -# Reach the VMs over Tailscale, and fall back to the LAN if the tailnet is down. -# -# ansible_host is a MagicDNS name. If tailscaled is not running on the control -# node that name does not resolve, the probe fails, and the LAN address recorded -# as lan_ip in inventory.ini takes over. -# -# Why the probe-then-connect shape rather than a plain `nc -w5 %h %p`: -# netcat-openbsd's -w is an IDLE timeout as well as a connect timeout, so a -# single `nc -w5` silently tears down the SSH session after five quiet seconds. -# That produced intermittent "Data could not be sent to remote host" failures on -# exactly the long, quiet operations (apt) where a dropped connection costs most. -# `nc -z` probes, then `exec nc` carries the session with no timeout at all. -# -# Safe against the LAN addresses drifting again (which is how fulcrum/mempool -# came to be transposed): known_hosts is keyed to the MagicDNS NAME, so if -# lan_ip ever points at a different machine the host key will not match and ssh -# aborts. Verified by pointing fulcrum-box at mempool-box's address: -# "Host key verification failed." -# -# lan_ip is a convenience, not an identity. If it goes stale the fallback stops -# working; it will never connect you to the wrong box. -ansible_ssh_common_args: >- - -o ProxyCommand="sh -c 'nc -z -w5 %h %p 2>/dev/null && exec nc %h %p || exec nc {{ lan_ip }} %p'" diff --git a/ansible/host_vars/forgejo_runner_local/main.yml b/ansible/host_vars/forgejo_runner_local/main.yml deleted file mode 100644 index 1fae71b..0000000 --- a/ansible/host_vars/forgejo_runner_local/main.yml +++ /dev/null @@ -1,17 +0,0 @@ ---- - -# Systemd services deployed on this host, monitored every 5 minutes. -# -# The fact lives with the machine rather than in a central map, for the same -# reason the cross-host ports do: "what runs here" is a property of the host, -# and a central list is one more thing to forget to update when a service moves. -# -# Only units WE deploy belong here. Distro units (ssh, cron) have their own -# supervision and would be noise. -monitored_services: - - forgejo-runner - # The isolated Docker daemon that Actions jobs build on. Listed separately - # from forgejo-runner because they fail separately and the interesting case - # is the asymmetric one: dind down, runner up, every job failing while the - # runner's own unit stays green. - - forgejo-dind diff --git a/ansible/host_vars/fulcrum_box_local/main.yml b/ansible/host_vars/fulcrum_box_local/main.yml deleted file mode 100644 index aa189b0..0000000 --- a/ansible/host_vars/fulcrum_box_local/main.yml +++ /dev/null @@ -1,17 +0,0 @@ -# fulcrum-box: the Electrum server. -# -# Read by the fulcrum role here and by the socket-proxy play on the edge host, -# which publishes the port. See host_vars/knots_box_local/main.yml for why this -# lives in host_vars rather than in the role's defaults. -fulcrum_ssl_port: 50002 - -# Systemd services deployed on this host, monitored every 5 minutes. -# -# The fact lives with the machine rather than in a central map, for the same -# reason the cross-host ports do: "what runs here" is a property of the host, -# and a central list is one more thing to forget to update when a service moves. -# -# Only units WE deploy belong here. Distro units (ssh, cron) have their own -# supervision and would be noise. -monitored_services: - - fulcrum diff --git a/ansible/host_vars/jar_lid_staging_local/main.yml b/ansible/host_vars/jar_lid_staging_local/main.yml deleted file mode 100644 index 7885123..0000000 --- a/ansible/host_vars/jar_lid_staging_local/main.yml +++ /dev/null @@ -1,28 +0,0 @@ ---- -# Jar Lid Archive STAGING. Secrets are in vault.yml beside this file. -# -# These live here rather than in the role defaults because three plays read -# them: the deploy on this box, the Caddy vhost on the edge host (port, domain) -# and the backup (retention). A role default cannot reach a second play. - -# Names this deployment everywhere it must be unique across hosts: the Caddy site -# file, the Gatus probe (probe_), the backup artefacts and the -# backup-dump/backup-store endpoints. -jar_lid_archive_instance: jar-lid-archive-staging - -# Fixed by the image's CMD (gunicorn --bind 0.0.0.0:8000); changing this moves -# what Caddy and the probe call, not where gunicorn listens. -jar_lid_archive_port: 8000 -jar_lid_archive_domain: "{{ subdomains.jar_lid_archive_staging }}.{{ root_domain }}" - -# Staging data is disposable: three days locally and on small-backups-box. -jar_lid_archive_backup_retention_days: 3 - -jar_lid_archive_seed_users: - - {username: alice, email: alice@example.com, password: "{{ jar_lid_archive_seed_password_alice }}"} - - {username: bob, email: bob@example.com, password: "{{ jar_lid_archive_seed_password_bob }}"} - - {username: admin, email: admin@example.com, password: "{{ jar_lid_archive_seed_password_admin }}", is_superuser: true} - -# No systemd units of ours run here - the stack is docker compose - so infra/401 -# has nothing to watch. The probe in the deploy playbook calls /health/, which -# checks the database and the bucket as well as gunicorn. diff --git a/ansible/host_vars/jar_lid_staging_local/vault.yml b/ansible/host_vars/jar_lid_staging_local/vault.yml deleted file mode 100644 index 0de8609..0000000 --- a/ansible/host_vars/jar_lid_staging_local/vault.yml +++ /dev/null @@ -1,34 +0,0 @@ -$ANSIBLE_VAULT;1.1;AES256 -37383535306133363031316535656461613038653562373839363432623163356234303061643566 -6533333535613064653336313664643063326562666639640a373530386539316363346332663137 -31303230323065623733663666383039346337343239646465393166663239643836356639373765 -3564313065363266360a643239333330616463616533623435656164386264303634356134303831 -36356131323637363734316161383065653663623732386638363165633265356362386538643039 -64636464323730316638316430643339356662353736366332663738656566663137366166393735 -34633430363931323330393234336661383532653366633735313365363562396137356438353330 -65313737313131326361326631363864396536343633623963303632623465363964373964393430 -61323161663939346435386435616364653336646165356232346335353235356339656336306138 -65613636353636623562613362633761333637613066663838396638613161663532393030303035 -35316464356539666631636265646433353537396564366538643962373364383730626164316265 -63393034663835346362636564316135326430633963393139646365333666663630336462323361 -31353830633335366635366435643864313665366563396134313938666537663232343162383366 -34636231636261663739383366363962366438323962393732393864623238343961363035333866 -61386630396334626564306536336561333265383464623730386263373132613438396332356631 -30386435333739653637633032313966626437303634613838653735373666343565306663376230 -61303734623865353835626464613033346437623234626136353664303434386262346634356463 -34396433353431393935386136333331643539633138636135353339323930386634393639666665 -36643462653061363062383039313936303062393364353366623965313530303031306336396336 -65643663343163636338393666386134366337633064376530303463393737396165663930383562 -64316363663636383764656139366433393766613262613433623637393538633130616232373136 -31376636633666626365376533356262393065656539613266326465643663383439373661393631 -30636136363066653936653837376634356333353036633165316662383337636435323162316430 -30316331353361396132653233623034376539323064363135306262376638353664643163653334 -37643638323038663531376336313064383238366430663363353733356530393661646138303230 -30633862626166653362346632626261636531373130353034656532386230616330643134383631 -63396138326336626431656130306531626539323934386332306361396239376466623866613364 -36666632316331326361373136326465323365653034363337313135656262623465306439646362 -30636639663163643430363165316231366232623332393635623461306339313762323265623732 -30633464343235323566356434376535313238663533633630356163336532303765633366316166 -38633462303762323930393337306265393033653061616337646163333335333737666332666632 -35613132646636333335313263343136323337616662623465653939336637653134383766336265 -35633638663636353537333238643236623033636163333164663062353863643164 diff --git a/ansible/host_vars/knots_box_local/main.yml b/ansible/host_vars/knots_box_local/main.yml deleted file mode 100644 index 3866541..0000000 --- a/ansible/host_vars/knots_box_local/main.yml +++ /dev/null @@ -1,26 +0,0 @@ -# knots-box: Bitcoin Knots and the DATUM Gateway. -# -# These ports are read twice: by the role that deploys the service here, and by -# the socket-proxy / Caddy plays that run on the EDGE host and publish them. -# A role default is invisible to that second play, which is why these live in -# host_vars rather than roles//defaults/ - the edge play reads them as -# hostvars['knots_box_local']., and the role picks them up automatically -# because host_vars outranks role defaults. -# -# They used to live in services_config.yml, a file 30 plays had to remember to -# name in vars_files: and that four role defaults silently depended on. -bitcoin_p2p_port: 8333 -datum_gateway_api_port: 7152 -datum_gateway_stratum_port: 23334 - -# Systemd services deployed on this host, monitored every 5 minutes. -# -# The fact lives with the machine rather than in a central map, for the same -# reason the cross-host ports do: "what runs here" is a property of the host, -# and a central list is one more thing to forget to update when a service moves. -# -# Only units WE deploy belong here. Distro units (ssh, cron) have their own -# supervision and would be noise. -monitored_services: - - bitcoind - - datum-gateway diff --git a/ansible/host_vars/memos_box_local/main.yml b/ansible/host_vars/memos_box_local/main.yml deleted file mode 100644 index f1fc19b..0000000 --- a/ansible/host_vars/memos_box_local/main.yml +++ /dev/null @@ -1,12 +0,0 @@ ---- - -# Systemd services deployed on this host, monitored every 5 minutes. -# -# The fact lives with the machine rather than in a central map, for the same -# reason the cross-host ports do: "what runs here" is a property of the host, -# and a central list is one more thing to forget to update when a service moves. -# -# Only units WE deploy belong here. Distro units (ssh, cron) have their own -# supervision and would be noise. -monitored_services: - - memos diff --git a/ansible/host_vars/mempool_box_local/main.yml b/ansible/host_vars/mempool_box_local/main.yml deleted file mode 100644 index f46a1a5..0000000 --- a/ansible/host_vars/mempool_box_local/main.yml +++ /dev/null @@ -1,6 +0,0 @@ -# mempool-box: the Mempool block explorer. -# -# Read by the mempool role here and by the Caddy play on the edge host, which -# proxies to it. See host_vars/knots_box_local/main.yml for why this lives in -# host_vars rather than in the role's defaults. -mempool_frontend_port: 8080 diff --git a/ansible/host_vars/monerod_box_local/main.yml b/ansible/host_vars/monerod_box_local/main.yml deleted file mode 100644 index d9bfef4..0000000 --- a/ansible/host_vars/monerod_box_local/main.yml +++ /dev/null @@ -1,31 +0,0 @@ -# monerod-box: the Monero node. -# -# These ports are read twice: by the role that deploys the service here, and by -# the Caddy play that runs on the EDGE host and publishes the restricted RPC. -# A role default is invisible to that second play, which is why they live in -# host_vars - the edge play reads them as -# hostvars['monerod_box_local']., and the role picks them up -# automatically because host_vars outranks role defaults. -monero_p2p_port: 18080 -monero_restricted_rpc_port: 18089 - -# The explorer's HTTP port, read by the Caddy play on the edge host. -xmrblocks_port: 8081 - -# The 500 GB chain disk, attached in Proxmox as scsi1. Named by that slot, which -# is fixed when the disk is attached and survives reboots and storage moves - -# not /dev/sdb, which it stopped being on 2026-10-04. Only read when the disk has -# no monero-data filesystem yet; see roles/monero/defaults/main.yml. -monero_data_device: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1 - -# Systemd services deployed on this host, monitored every 5 minutes. -# -# The fact lives with the machine rather than in a central map, for the same -# reason the cross-host ports do: "what runs here" is a property of the host, -# and a central list is one more thing to forget to update when a service moves. -# -# Only units WE deploy belong here. Distro units (ssh, cron) have their own -# supervision and would be noise. -monitored_services: - - monerod - - xmrblocks diff --git a/ansible/host_vars/music_box_local/main.yml b/ansible/host_vars/music_box_local/main.yml deleted file mode 100644 index 9af0beb..0000000 --- a/ansible/host_vars/music_box_local/main.yml +++ /dev/null @@ -1,11 +0,0 @@ ---- -# Navidrome's HTTP port. Read TWICE: by roles/navidrome on this box, and by the -# Caddy play on the edge host, which reads it as -# hostvars['music_box_local'].navidrome_port. A role default cannot reach a -# second play, which is why this lives here and not in defaults/main.yml. -navidrome_port: 4533 - -# Systemd units monitored every 5 minutes by infra/401_service_monitoring.yml. -# beets is not here: it is an interactive CLI, not a service. -monitored_services: - - navidrome diff --git a/ansible/host_vars/nodito/vault.yml b/ansible/host_vars/nodito/vault.yml deleted file mode 100644 index e64085e..0000000 --- a/ansible/host_vars/nodito/vault.yml +++ /dev/null @@ -1,11 +0,0 @@ -$ANSIBLE_VAULT;1.1;AES256 -30333035323663393939343061323234336164396465623665346165393534646366333332376463 -3364373463333664363334373964323838336531353364310a636636373539623464336630666164 -61376532616339376562373238383436306664313564663266303534346461666466383965323538 -3163313239626663310a613033336332653165333537313366636361663036383031376561613761 -31313563373062333033323037653939663762343161656264633436343361663737626366663732 -39366666626338323436383134646263643538333564313566346336323563663534653161396136 -39333565393538366238643563323630346166643461643063393631643665363566623631373762 -36643866646637306231653837363838656163613766636265383139333838396535626335343163 -32613935353330636263616333666230323436663935326133636362343836323535623237646235 -6266316363366335323162663039366137633865396237373632 diff --git a/ansible/host_vars/spacey/main.yml b/ansible/host_vars/spacey/main.yml deleted file mode 100644 index b5249e6..0000000 --- a/ansible/host_vars/spacey/main.yml +++ /dev/null @@ -1,20 +0,0 @@ ---- - -# Systemd services deployed on this host, monitored every 5 minutes. -# -# The fact lives with the machine rather than in a central map, for the same -# reason the cross-host ports do: "what runs here" is a property of the host, -# and a central list is one more thing to forget to update when a service moves. -# -# Only units WE deploy belong here. Distro units (ssh, cron) have their own -# supervision and would be noise. -monitored_services: - - headscale - - caddy - -# Gatus moved onto this host in the 2026-10-17 rebuild (the `monitoring` droplet -# is gone). Headscale already listens on 0.0.0.0:8080, so Gatus takes 8081. -# Host vars rather than role defaults: the Caddy vhost play in -# services/gatus/deploy_gatus_playbook.yml does not load the role and reads -# this same variable. -gatus_port: 8081 diff --git a/ansible/host_vars/vipy/main.yml b/ansible/host_vars/vipy/main.yml deleted file mode 100644 index 93415da..0000000 --- a/ansible/host_vars/vipy/main.yml +++ /dev/null @@ -1,20 +0,0 @@ ---- - -# Systemd services deployed on this host, monitored every 5 minutes. -# -# The fact lives with the machine rather than in a central map, for the same -# reason the cross-host ports do: "what runs here" is a property of the host, -# and a central list is one more thing to forget to update when a service moves. -# -# Only units WE deploy belong here. Distro units (ssh, cron) have their own -# supervision and would be noise. -monitored_services: - - forgejo - - lnbits - - caddy - - phoenixd - # Deployed by the arbret repo, not this one (see arbret_handoff.md); listed - # here because "what runs on this host" is still a property of the host. - - arbret-server - - arbret-worker - - postgresql diff --git a/ansible/infra/01_user_and_access_setup_playbook.yml b/ansible/infra/01_user_and_access_setup_playbook.yml index 0e2c914..13e5149 100644 --- a/ansible/infra/01_user_and_access_setup_playbook.yml +++ b/ansible/infra/01_user_and_access_setup_playbook.yml @@ -1,5 +1,7 @@ - name: Secure Debian - hosts: managed + hosts: all + vars_files: + - ../infra_vars.yml become: true tasks: diff --git a/ansible/infra/02_firewall_and_fail2ban_playbook.yml b/ansible/infra/02_firewall_and_fail2ban_playbook.yml index 309db56..e83cbcb 100644 --- a/ansible/infra/02_firewall_and_fail2ban_playbook.yml +++ b/ansible/infra/02_firewall_and_fail2ban_playbook.yml @@ -1,5 +1,7 @@ - name: Secure Debian - hosts: managed + hosts: all + vars_files: + - ../infra_vars.yml become: true tasks: diff --git a/ansible/infra/400_host_monitoring.yml b/ansible/infra/400_host_monitoring.yml deleted file mode 100644 index 3f6b119..0000000 --- a/ansible/infra/400_host_monitoring.yml +++ /dev/null @@ -1,216 +0,0 @@ ---- -# Host-level monitoring for the whole estate, reported to Gatus. -# -# Every check here PUSHES. Gatus never reaches out, which matters because nodito -# and its VMs sit behind NAT, and because four of the five checks are internal -# state with no pollable surface at all - disk usage, CPU temperature, ZFS pool -# health and UPS mains status cannot be observed from outside the machine. -# -# Liveness is a push too, and that is a choice rather than a limitation. A -# heartbeat proves the host is running AND can reach the internet; an ICMP probe -# from one vantage point only proves it answers pings from there. And because -# Gatus alerts when a heartbeat window expires, a check that stops running -# raises the alarm by itself - a dead timer looks exactly like a dead host, -# which is the correct reading. -# -# Each host has ONE bearer token, shared across its own checks: a token can only -# write results for that host's endpoints, so a compromised host can lie about -# itself, which it could do anyway. -# -# The push URL must use Gatus's own key format (config/key/key.go): -# key = sanitize(group) + "_" + sanitize(name) -# where sanitize lowercases and replaces / _ . , space # + & with "-". So -# knots_box_local becomes knots-box-local in the URL but stays readable in the -# name. host_key below is the Jinja equivalent; do not hand-write these. - -# ───────────────────────────────────────────────────────────────────────────── -# Register everything with Gatus. -# -# This play runs FIRST on purpose. Gatus reloads its config within 30s, and the -# host plays below take minutes, so every endpoint exists before its first push -# arrives. Registering afterwards would 404 every first report. -# -# Heartbeat windows are several times the check interval, so one missed run - a -# slow apt run, a reboot - does not raise an alarm, but a check that has -# genuinely stopped does. -# ───────────────────────────────────────────────────────────────────────────── -# ───────────────────────────────────────────────────────────────────────────── -# Alerting thresholds, and why they differ by check type. -# -# `failure-threshold` counts CONSECUTIVE failures, but "consecutive" means a -# different amount of wall-clock time per check: -# -# push/heartbeat endpoints a failure is produced once per heartbeat window -# pulled endpoints a failure is produced once per interval -# -# So the default of 3 would mean 33 minutes on an 11m heartbeat and over a day -# on a 7h one - and the heartbeat window ALREADY encodes the tolerance. An 11m -# window on a 5-minute push is precisely "one missed push forgiven"; stacking a -# threshold of 3 on top triples a tolerance that was already chosen. -# -# Hence: push endpoints alert on the FIRST heartbeat failure. Pulled endpoints -# have no built-in tolerance, so the threshold is where it belongs for them. -# ───────────────────────────────────────────────────────────────────────────── -- name: Register the host checks with Gatus - hosts: observability - become: yes - vars: - monitored: "{{ groups['managed'] | sort }}" - - tasks: - - name: Build the liveness endpoint list - ansible.builtin.set_fact: - liveness_endpoints: "{{ liveness_endpoints | default([]) + [{ - 'name': item, - 'group': 'liveness', - 'token': gatus_push_tokens[item], - 'heartbeat': '11m'}] }}" - loop: "{{ monitored }}" - - - name: Build the disk endpoint list - ansible.builtin.set_fact: - disk_endpoints: "{{ disk_endpoints | default([]) + [{ - 'name': item, - 'group': 'disk', - 'token': gatus_push_tokens[item], - 'heartbeat': '7h'}] }}" - loop: "{{ monitored }}" - - - name: Register liveness endpoints - ansible.builtin.include_role: - name: gatus_endpoint - vars: - gatus_endpoint_default_alerts: - - type: signal - # 1, not 3: the heartbeat window is the tolerance. See the note above. - failure-threshold: 1 - success-threshold: 2 - send-on-resolved: true - minimum-reminder-interval: 6h - gatus_endpoint_name: liveness - gatus_endpoint_external: "{{ liveness_endpoints }}" - - - name: Register disk endpoints - ansible.builtin.include_role: - name: gatus_endpoint - vars: - gatus_endpoint_default_alerts: - - type: signal - # 1, not 3: the heartbeat window is the tolerance. See the note above. - failure-threshold: 1 - success-threshold: 2 - send-on-resolved: true - minimum-reminder-interval: 6h - gatus_endpoint_name: disk - gatus_endpoint_external: "{{ disk_endpoints }}" - - - name: Register the hypervisor endpoints - ansible.builtin.include_role: - name: gatus_endpoint - vars: - gatus_endpoint_default_alerts: - - type: signal - # 1, not 3: the heartbeat window is the tolerance. See the note above. - failure-threshold: 1 - success-threshold: 2 - send-on-resolved: true - minimum-reminder-interval: 6h - gatus_endpoint_name: hypervisor - gatus_endpoint_external: - - name: cpu - group: hypervisor - token: "{{ gatus_push_tokens['nodito'] }}" - heartbeat: "11m" - - name: zfs - group: hypervisor - token: "{{ gatus_push_tokens['nodito'] }}" - heartbeat: "7h" - - name: ups - group: hypervisor - token: "{{ gatus_push_tokens['nodito'] }}" - heartbeat: "11m" - -- name: Deploy host liveness and disk checks - hosts: managed - become: yes - vars: - gatus_api: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints" - host_key: "{{ inventory_hostname | lower | regex_replace('[/_.,# +&]', '-') }}" - host_token: "{{ gatus_push_tokens[inventory_hostname] }}" - - tasks: - - name: Is the host up? - ansible.builtin.include_role: - name: healthcheck - vars: - healthcheck_name: liveness - healthcheck_description: "Liveness heartbeat for {{ inventory_hostname }}" - healthcheck_check: liveness - healthcheck_interval: "5min" - healthcheck_boot_delay: "1min" - healthcheck_push_url: "{{ gatus_api }}/liveness_{{ host_key }}/external" - healthcheck_push_token: "{{ host_token }}" - - - name: Is the disk packed? - ansible.builtin.include_role: - name: healthcheck - vars: - healthcheck_name: disk-usage - healthcheck_description: "Disk usage for {{ inventory_hostname }}" - healthcheck_check: disk-usage - # Every 6h rather than daily. Disk usage itself moves slowly, but the - # heartbeat can only be as tight as the push frequency - a daily push - # forces a >24h window, and a stuck check then hides for a day and a - # half. Six-hourly buys a 7h window. RandomizedDelaySec spreads the - # hosts so twelve boxes do not all report in the same second. - healthcheck_on_calendar: "*-*-* 00/6:00:00" - healthcheck_randomized_delay: "900" - healthcheck_boot_delay: "5min" - healthcheck_push_url: "{{ gatus_api }}/disk_{{ host_key }}/external" - healthcheck_push_token: "{{ host_token }}" - -- name: Deploy the hypervisor-only checks - hosts: hypervisor - become: yes - vars: - gatus_api: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints" - host_token: "{{ gatus_push_tokens[inventory_hostname] }}" - - tasks: - - name: Is the CPU hot? - ansible.builtin.include_role: - name: healthcheck - vars: - healthcheck_name: cpu-temp - healthcheck_description: "CPU temperature for {{ inventory_hostname }}" - healthcheck_check: cpu-temp - healthcheck_packages: [curl, lm-sensors] - healthcheck_interval: "5min" - healthcheck_push_url: "{{ gatus_api }}/hypervisor_cpu/external" - healthcheck_push_token: "{{ host_token }}" - - - name: Is ZFS broken? - ansible.builtin.include_role: - name: healthcheck - vars: - healthcheck_name: zfs-health - healthcheck_description: "ZFS pool health for {{ zfs_pool_name }}" - healthcheck_check: zfs-health - healthcheck_packages: [curl, jq] - healthcheck_zfs_pool: "{{ zfs_pool_name }}" - healthcheck_on_calendar: "*-*-* 00/6:20:00" - healthcheck_boot_delay: "10min" - healthcheck_push_url: "{{ gatus_api }}/hypervisor_zfs/external" - healthcheck_push_token: "{{ host_token }}" - - - name: Is the UPS online? - ansible.builtin.include_role: - name: healthcheck - vars: - healthcheck_name: ups-status - healthcheck_description: "UPS mains status for {{ ups_name }}" - healthcheck_check: ups-status - healthcheck_ups_name: "{{ ups_name }}" - healthcheck_interval: "5min" - healthcheck_push_url: "{{ gatus_api }}/hypervisor_ups/external" - healthcheck_push_token: "{{ host_token }}" diff --git a/ansible/infra/401_service_monitoring.yml b/ansible/infra/401_service_monitoring.yml deleted file mode 100644 index 088cc7b..0000000 --- a/ansible/infra/401_service_monitoring.yml +++ /dev/null @@ -1,124 +0,0 @@ ---- -# Is each systemd-deployed service actually running? -# -# Every 5 minutes, with an 11-minute Gatus heartbeat - one missed run before -# it alarms, so a reboot or a slow check does not page anyone, but a host that -# stops reporting does. -# -# This closes the gap that let a real bug run unnoticed: a backup script left -# forgejo, lnbits, headscale and memos stopped, and NOTHING caught it. The dumps -# exited 0, the artefacts were correct, the deploy said failed=0, and liveness -# only proves the HOST is up - not that anything on it is serving. -# -# One endpoint PER UNIT, not per host. A host running four services needs four -# endpoints, or a single red light says "something on vipy is down" without -# saying which - and that is the question you actually have at 3am. But only ONE -# timer per host: the check iterates that host's units and pushes a result for -# each, the same way check-backups.sh reports per source. Four units on vipy -# would otherwise mean four scripts, four services and four timers. -# -# Which units each host runs is in host_vars//main.yml as -# monitored_services, because "what runs here" is a property of the machine. -# -# Keys are host-qualified because unit names collide - caddy runs on four -# machines. Gatus computes sanitize(group)_sanitize(name), so group "services" -# and name "vipy/caddy" give services_vipy-caddy. - -# ───────────────────────────────────────────────────────────────────────────── -# Register one endpoint per unit. Runs first: Gatus reloads within 30s, and the -# host play above takes minutes, so every endpoint exists before its first push. -# ───────────────────────────────────────────────────────────────────────────── -# ───────────────────────────────────────────────────────────────────────────── -# Alerting thresholds, and why they differ by check type. -# -# `failure-threshold` counts CONSECUTIVE failures, but "consecutive" means a -# different amount of wall-clock time per check: -# -# push/heartbeat endpoints a failure is produced once per heartbeat window -# pulled endpoints a failure is produced once per interval -# -# So the default of 3 would mean 33 minutes on an 11m heartbeat and over a day -# on a 7h one - and the heartbeat window ALREADY encodes the tolerance. An 11m -# window on a 5-minute push is precisely "one missed push forgiven"; stacking a -# threshold of 3 on top triples a tolerance that was already chosen. -# -# Hence: push endpoints alert on the FIRST heartbeat failure. Pulled endpoints -# have no built-in tolerance, so the threshold is where it belongs for them. -# -# EXCEPT here, where the threshold is 2. The heartbeat only forgives a MISSING -# push. These checks also push an explicit `false` whenever a unit is caught -# mid-restart - a deploy, an unattended upgrade, a backup stopping the service -# for its dump - and at 1 every one of those pages, then resolves ten minutes -# later. 2 means "down on two consecutive 5-minute runs", i.e. 5-10 minutes. -# A host that stops reporting altogether still alarms on the heartbeat alone. -# ───────────────────────────────────────────────────────────────────────────── -- name: Register the service checks with Gatus - hosts: observability - become: yes - - tasks: - # Two plain steps rather than one clever expression: first collect which - # units each host declares, then flatten that into endpoints. - - name: Collect the units each host declares - ansible.builtin.set_fact: - host_units: "{{ host_units | default([]) + [{'host': item, 'units': hostvars[item].monitored_services}] }}" - loop: "{{ groups['managed'] | sort }}" - when: hostvars[item].monitored_services | default([]) | length > 0 - - - name: Build one endpoint per unit - ansible.builtin.set_fact: - service_endpoints: "{{ service_endpoints | default([]) + [{ - 'name': (item.0.host | lower | regex_replace('[/_.,# +&]', '-')) ~ '/' ~ item.1, - 'group': 'services', - 'token': gatus_push_tokens[item.0.host], - 'heartbeat': hostvars[item.0.host].service_health_heartbeat | default('11m')}] }}" - loop: "{{ host_units | subelements('units') }}" - - - name: Register the service endpoints - ansible.builtin.include_role: - name: gatus_endpoint - vars: - gatus_endpoint_default_alerts: - - type: signal - # 2: tolerate one explicit `false` from a unit caught mid-restart. - # See the note above. - failure-threshold: 2 - success-threshold: 2 - send-on-resolved: true - minimum-reminder-interval: 6h - gatus_endpoint_name: services - gatus_endpoint_external: "{{ service_endpoints }}" - -- name: Monitor systemd services on every host that has them - hosts: managed - become: yes - vars: - gatus_api: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints" - host_key: "{{ inventory_hostname | lower | regex_replace('[/_.,# +&]', '-') }}" - - tasks: - - name: Is every deployed service running? - ansible.builtin.include_role: - name: healthcheck - vars: - healthcheck_name: service-health - healthcheck_description: "systemd services on {{ inventory_hostname }}" - healthcheck_check: systemd-units - healthcheck_units: "{{ monitored_services }}" - healthcheck_units_key_prefix: "services_{{ host_key }}" - # Cadence and heartbeat window are ONE decision, and both are a property - # of the host: a host can set service_health_on_calendar and a matching - # service_health_heartbeat in its host_vars. None does today; everything - # runs 5min/11m. (monerod-box was daily during its initial sync.) - # - # An empty healthcheck_on_calendar means "use healthcheck_interval" - - # see the timer template in roles/healthcheck. - healthcheck_interval: "5min" - healthcheck_on_calendar: "{{ service_health_on_calendar | default('') }}" - healthcheck_randomized_delay: "{{ service_health_randomized_delay | default('30') }}" - healthcheck_boot_delay: "2min" - # The per-unit results go to keys under this collection; the role's own - # single-result push is unused here, so only the base is set. - healthcheck_push_base: "{{ gatus_api }}" - healthcheck_push_token: "{{ gatus_push_tokens[inventory_hostname] }}" - when: monitored_services | default([]) | length > 0 diff --git a/ansible/infra/402_public_monitoring.yml b/ansible/infra/402_public_monitoring.yml deleted file mode 100644 index 3ff5b23..0000000 --- a/ansible/infra/402_public_monitoring.yml +++ /dev/null @@ -1,213 +0,0 @@ ---- -# Domain expiry, DNS correctness, and public endpoint reachability. -# -# These are the first checks in the estate that PULL rather than push, and that -# is the right way round for them: all three are about how the outside world -# sees us, so they must be measured from outside. Gatus polls from the -# observability host and needs nothing installed anywhere else - there is no -# script, no timer and no push token, because nothing is reporting in. (The -# provider-balance check does carry an API credential - see below.) -# -# That also means these have no heartbeat. A heartbeat answers "did the thing -# that was supposed to report in do so"; when Gatus does the checking itself, -# failure is immediate and self-evident. - -- name: Register the public-facing checks with Gatus - hosts: observability - become: yes - - vars: - # Expected A records, derived from inventory rather than written down again. - # The estate's recurring bug is an address recorded in a second place and - # then left behind when the machine moved, so the check asserts against - # ansible_host - if a box is renumbered, inventory is the one edit. - dns_records: - - {sub: "{{ subdomains.gatus }}", host: spacey} - - {sub: "{{ subdomains.headscale }}", host: spacey} - - {sub: "{{ subdomains.vaultwarden }}", host: vipy} - - {sub: "{{ subdomains.forgejo }}", host: vipy} - - {sub: "{{ subdomains.lnbits }}", host: vipy} - - {sub: "{{ subdomains.personal_blog }}", host: vipy} - - {sub: "{{ subdomains.memos }}", host: vipy} - - {sub: "{{ subdomains.mempool }}", host: vipy} - - {sub: "{{ subdomains.datum_gateway }}", host: vipy} - - {sub: "{{ subdomains.monero }}", host: vipy} - - {sub: "{{ subdomains.monero_explorer }}", host: vipy} - - {sub: "{{ subdomains.music }}", host: vipy} - - {sub: "{{ subdomains.jar_lid_archive_staging }}", host: vipy} - # Not under root_domain, so named in full. arbret.com's zone is NOT in - # Porkbun (mynymdns) - this is the check that would have caught it still - # pointing at prd-arbret's recycled IP after the 2026-10-17 rebuild. - - {fqdn: arbret.com, host: vipy} - - {fqdn: www.arbret.com, host: vipy} - - # A public resolver on purpose: this must test what the internet sees, not - # what a local cache or the tailnet's MagicDNS happens to answer. - dns_resolver: "1.1.1.1" - - # Expected status per site, checked live before being written down. - # 401 is the CORRECT answer for the two behind basic auth - asserting 200 - # there would go green precisely when the auth broke. - public_sites: - - {name: gatus, sub: "{{ subdomains.gatus }}", path: "/", status: 401} - - {name: headscale, sub: "{{ subdomains.headscale }}", path: "/health", status: 200} - - {name: vaultwarden, sub: "{{ subdomains.vaultwarden }}", path: "/", status: 200} - - {name: forgejo, sub: "{{ subdomains.forgejo }}", path: "/", status: 200} - - {name: lnbits, sub: "{{ subdomains.lnbits }}", path: "/", status: 200} - - {name: blog, sub: "{{ subdomains.personal_blog }}", path: "/", status: 200} - - {name: memos, sub: "{{ subdomains.memos }}", path: "/", status: 200} - - {name: mempool, sub: "{{ subdomains.mempool }}", path: "/", status: 200} - - {name: datum, sub: "{{ subdomains.datum_gateway }}", path: "/", status: 401} - # 401 is CORRECT: monerod answers an unauthenticated request with an HTTP - # Digest challenge. Asserting 200 here would go green precisely when - # rpc-login broke and the node started serving the world for free. - - {name: monero, sub: "{{ subdomains.monero }}", path: "/get_info", status: 401} - - {name: exmrplore, sub: "{{ subdomains.monero_explorer }}", path: "/", status: 200} - # /app/, not "/": checked live, "/" answers 302 (Navidrome redirects to - # the SPA) and only /app/ returns 200 with the real UI. 200 rather than - # 401 because Navidrome has no basic auth in front of it - the Subsonic - # API the mobile clients speak cannot carry a proxy credential, so its - # own login is the gate. See roles/navidrome/README.md. - - {name: music, sub: "{{ subdomains.music }}", path: "/app/", status: 200} - # /health/ through the whole public path: DNS, vipy's Caddy and cert, the - # tailnet hop, gunicorn, and the database and bucket behind it (503 if - # either is down). "/" would only prove the login redirect works. - - {name: jars-staging, sub: "{{ subdomains.jar_lid_archive_staging }}", path: "/health/", status: 200} - # arbret (runs on vipy since 2026-10-17; deployed by the arbret repo, not - # this one). /api/health answers 200 + "status":"ok" when healthy and 503 - # "degraded" otherwise, DB down included - per the arbret owner. The body - # condition guards against a 200 that is not actually healthy. - - {name: arbret, fqdn: arbret.com, path: "/", status: 200} - - {name: arbret-health, fqdn: arbret.com, path: "/api/health", status: 200, - conditions: ['[BODY].status == ok']} - - # Ports published from the edge host by socket_proxy. - public_tcp: - - {name: bitcoin-p2p, host: vipy, port: "{{ hostvars['knots_box_local'].bitcoin_p2p_port }}"} - - {name: fulcrum-ssl, host: vipy, port: "{{ hostvars['fulcrum_box_local'].fulcrum_ssl_port }}"} - - {name: datum-stratum, host: vipy, port: "{{ hostvars['knots_box_local'].datum_gateway_stratum_port }}"} - - tasks: - # ── Domain expiry ──────────────────────────────────────────────────────── - # Each domain needs a URL SCHEME: Gatus derives the endpoint type from the - # prefix (endpoint.Type()), so a bare "example.com" is UNKNOWN and the whole - # config is rejected. No status is asserted, only the WHOIS/RDAP expiry, so - # whatever the apex serves - a real site, or the registrar's parking page - - # is irrelevant. - # - # 24h, and upstream enforces a 5m minimum for DOMAIN_EXPIRATION anyway - # because it uses a free whois service that must not be hammered. - # 336h = 14 days of runway, because renewal is a manual act at the registrar. - - name: Build the domain endpoints - ansible.builtin.set_fact: - domain_endpoints: "{{ domain_endpoints | default([]) + [{ - 'name': item, - 'group': 'domain', - 'url': 'https://' ~ item, - 'interval': '24h', - 'conditions': ['[DOMAIN_EXPIRATION] > 336h'], - 'alerts': [{'type': 'signal', 'failure-threshold': 1, - 'success-threshold': 1, 'send-on-resolved': true, - 'minimum-reminder-interval': '168h'}]}] }}" - loop: "{{ monitored_domains }}" - - # ── DNS ────────────────────────────────────────────────────────────────── - # 6h, not daily: a DNS query is cheap and a wrong record is an outage. The - # domain check stays at 24h because it does a WHOIS/RDAP lookup against a - # free service. Alert on the first failure - at a 6h interval, waiting for - # three would be nearly a day. - - name: Build the DNS endpoints - ansible.builtin.set_fact: - dns_endpoints: "{{ dns_endpoints | default([]) + [{ - 'name': (item.fqdn if item.fqdn is defined else item.sub ~ '.' ~ root_domain), - 'group': 'dns', - 'url': dns_resolver, - 'interval': item.interval | default('6h'), - 'dns': {'query-type': 'A', 'query-name': (item.fqdn if item.fqdn is defined else item.sub ~ '.' ~ root_domain)}, - 'conditions': ['[DNS_RCODE] == NOERROR', - '[BODY] == ' ~ hostvars[item.host].ansible_host], - 'alerts': [{'type': 'signal', 'failure-threshold': 1, - 'success-threshold': 1, 'send-on-resolved': true, - 'minimum-reminder-interval': '24h'}]}] }}" - loop: "{{ dns_records }}" - - # ── Public HTTP ────────────────────────────────────────────────────────── - # failure-threshold 3 at a 5m interval = 15 minutes. A pulled endpoint has - # no heartbeat window, so unlike the push checks the tolerance has to live - # in the threshold - and one failed poll of a public site is usually a blip. - - name: Build the public HTTP endpoints - ansible.builtin.set_fact: - http_endpoints: "{{ http_endpoints | default([]) + [{ - 'name': item.name, - 'group': 'public', - 'url': 'https://' ~ (item.fqdn if item.fqdn is defined else item.sub ~ '.' ~ root_domain) ~ item.path, - 'interval': item.interval | default('5m'), - 'conditions': ['[STATUS] == ' ~ item.status, - '[CERTIFICATE_EXPIRATION] > 168h'] - + item.conditions | default([]), - 'alerts': [{'type': 'signal', 'failure-threshold': item.threshold | default(3), - 'success-threshold': 2, 'send-on-resolved': true, - 'minimum-reminder-interval': '6h'}]}] }}" - loop: "{{ public_sites }}" - - # ── Public TCP ─────────────────────────────────────────────────────────── - - name: Build the public TCP endpoints - ansible.builtin.set_fact: - tcp_endpoints: "{{ tcp_endpoints | default([]) + [{ - 'name': item.name, - 'group': 'public', - 'url': 'tcp://' ~ hostvars[item.host].ansible_host ~ ':' ~ item.port, - 'interval': '5m', - 'conditions': ['[CONNECTED] == true'], - 'alerts': [{'type': 'signal', 'failure-threshold': 3, - 'success-threshold': 2, 'send-on-resolved': true, - 'minimum-reminder-interval': '6h'}]}] }}" - loop: "{{ public_tcp }}" - - # ── Provider balance ───────────────────────────────────────────────────── - # 99stack is prepaid: when the balance runs out the servers stop. The one - # pulled check here that carries a credential - unlike everything above, - # the API will not answer anonymously (403 "Authorization token is - # missing"). - # - # The condition is RUNWAY, not balance. billing.secondsdue is the - # provider's own "seconds until you run out at current usage", so it - # follows the spend as servers come and go; a fixed balance threshold - # would be one more number written down once and left behind. 336h of - # runway, same as domain expiry, because topping up is a manual act too. - # - # The token is account-wide - 99stack has no read-only scope - so it can - # create and destroy servers. It lands in Gatus's endpoints file (0640 - # root:gatus), the same place as the push tokens. - - name: Require the 99stack API token - ansible.builtin.assert: - that: - - ninetyninestack_api_token is defined - - ninetyninestack_api_token | length > 0 - fail_msg: >- - ninetyninestack_api_token must be in group_vars/all/vault.yml - an - API token from the 99stack control panel. - quiet: true - - - name: Build the provider endpoints - ansible.builtin.set_fact: - provider_endpoints: - - name: 99stack-balance - group: provider - url: https://api.99stack.com/v1.5/account/info - interval: 6h - headers: - Authorization: "Bearer {{ ninetyninestack_api_token }}" - conditions: - - "[STATUS] == 200" - - "[BODY].billing.secondsdue > {{ 14 * 86400 }}" - alerts: - - {type: signal, failure-threshold: 1, success-threshold: 1, - send-on-resolved: true, minimum-reminder-interval: 24h} - - - name: Register the public-facing endpoints - ansible.builtin.include_role: - name: gatus_endpoint - vars: - gatus_endpoint_name: public - gatus_endpoint_pulled: "{{ domain_endpoints + dns_endpoints + http_endpoints + tcp_endpoints + provider_endpoints }}" diff --git a/ansible/infra/403_service_probe_registration.yml b/ansible/infra/403_service_probe_registration.yml deleted file mode 100644 index f8cbb91..0000000 --- a/ansible/infra/403_service_probe_registration.yml +++ /dev/null @@ -1,98 +0,0 @@ ---- -# The per-service health probes. -# -# These are NOT the same thing as the systemd checks in infra/401. Those answer -# "is the unit running"; these answer "does the service actually respond" - an -# RPC call to bitcoind, a TCP connect to Fulcrum's Electrum port, an HTTP fetch -# from the Mempool backend. A process can be perfectly `active` and useless, -# which is precisely the gap these close. -# -# The checks themselves live in each service's own role, deployed by that -# service's playbook. This play only registers where they report, because the -# endpoints must exist in Gatus before the first push arrives. -# -# They used to push to Uptime Kuma. The scripts now POST with a bearer token -# instead of GETting ?status=up, and each host uses its own token. - -# ───────────────────────────────────────────────────────────────────────────── -# Alerting thresholds, and why they differ by check type. -# -# `failure-threshold` counts CONSECUTIVE failures, but "consecutive" means a -# different amount of wall-clock time per check: -# -# push/heartbeat endpoints a failure is produced once per heartbeat window -# pulled endpoints a failure is produced once per interval -# -# So the default of 3 would mean 33 minutes on an 11m heartbeat and over a day -# on a 7h one - and the heartbeat window ALREADY encodes the tolerance. An 11m -# window on a 5-minute push is precisely "one missed push forgiven"; stacking a -# threshold of 3 on top triples a tolerance that was already chosen. -# -# Hence: push endpoints alert on the FIRST heartbeat failure. Pulled endpoints -# have no built-in tolerance, so the threshold is where it belongs for them. -# ───────────────────────────────────────────────────────────────────────────── -- name: Register the per-service probes with Gatus - hosts: observability - become: yes - - vars: - probes: - # failure_threshold 4: this probe runs every MINUTE, and bitcoind turns RPC - # away for 1-3 minutes when a block lands - validation on the saturated - # HDD plus Fulcrum/mempool/DATUM all fetching the block at once (93 - # "work queue depth exceeded" in one day, 2026-10-18). At 1, every slow - # block paged. 4 consecutive failed minutes is past any block burst seen, - # and a real outage still alerts within ~5 minutes. - - {name: bitcoin-knots, host: knots_box_local, failure_threshold: 4} - - {name: datum-gateway, host: knots_box_local} - - {name: fulcrum, host: fulcrum_box_local} - - {name: phoenixd, host: vipy} - - {name: forgejo-runner, host: forgejo_runner_local} - - {name: mempool-mariadb, host: mempool_box_local} - - {name: mempool-backend, host: mempool_box_local} - - {name: mempool-frontend, host: mempool_box_local} - # Every 5 minutes like the rest since 2026-10-17 (they were daily while - # the node synced on a rotating disk). Cadence lives in - # roles/{monero,xmrblocks}/defaults/main.yml; it and this window are one - # decision and must move together. - - {name: monerod, host: monerod_box_local} - - {name: xmrblocks, host: monerod_box_local} - - {name: navidrome, host: music_box_local} - - {name: jar-lid-archive-staging, host: jar_lid_staging_local} - # arbret (on vipy, deployed by the arbret repo). `host` here is the KEY in - # gatus_push_tokens, and arbret has a dedicated `arbret` token rather than - # vipy's - so it cannot write vipy's own results, and vice versa. - # Windows are two runs plus slack, as for the 5m probes above: - # analytics hourly at :00 -> 130m - # beancount daily at 03:00 -> 26h - # Cadence lives in the arbret repo; tell us if it changes. - - {name: arbret-analytics, host: arbret, heartbeat: '130m'} - - {name: arbret-beancount, host: arbret, heartbeat: '26h'} - - # The alert every probe gets. failure-threshold is 1 - the heartbeat window - # is the tolerance, see the note above - unless a probe sets - # failure_threshold, for checks that legitimately fail for a minute or two. - probe_alert: - type: signal - failure-threshold: 1 - success-threshold: 2 - send-on-resolved: true - minimum-reminder-interval: 6h - - tasks: - - name: Build the probe endpoint list - ansible.builtin.set_fact: - probe_endpoints: "{{ probe_endpoints | default([]) + [{ - 'name': item.name, - 'group': 'probe', - 'token': gatus_push_tokens[item.host], - 'heartbeat': item.heartbeat | default('11m'), - 'alerts': [probe_alert | combine({'failure-threshold': item.failure_threshold | default(1)})]}] }}" - loop: "{{ probes }}" - - - name: Register the probe endpoints - ansible.builtin.include_role: - name: gatus_endpoint - vars: - gatus_endpoint_name: probes - gatus_endpoint_external: "{{ probe_endpoints }}" diff --git a/ansible/infra/409_remove_legacy_monitoring.yml b/ansible/infra/409_remove_legacy_monitoring.yml deleted file mode 100644 index 8f57e46..0000000 --- a/ansible/infra/409_remove_legacy_monitoring.yml +++ /dev/null @@ -1,109 +0,0 @@ ---- -# Remove the Uptime-Kuma-era monitoring that 400/401/402 replaced. -# -# Deleting the playbooks that installed these is NOT enough: the units are on -# the hosts, enabled, and keep firing regardless of what the repo says. Two of -# them still push to https://uptime.contrapeso.xyz every 15 minutes. A playbook -# that is deleted without a cleanup leaves its output running forever, with -# nothing in the repo left to explain it. -# -# What replaced what, all verified against the deployed scripts before removal: -# -# disk-usage-monitor -> disk-usage-healthcheck (infra/400) -# The old one checked ONLY "/" at 80%. The replacement walks every real -# filesystem, excluding tmpfs/devtmpfs/squashfs/overlay, at 85%. Strictly -# more coverage, so nothing is lost. -# -# system-healthcheck -> liveness-healthcheck (infra/400) -# The old script computed uptime and pushed. That is exactly a liveness -# heartbeat and nothing more. -# -# nodito-cpu-temp-monitor -> cpu-temp-healthcheck (infra/400) -# zfs-health-monitor -> zfs-health-healthcheck (infra/400) -# The ZFS check logic was ported verbatim - same five conditions - so only -# the reporting transport changed. -# -# NOT removed, because they are not monitoring: -# zfs-monthly-scrub.{timer,service} the actual scrub (infra/nodito/32) -# pull-backups, check-backups the backup machinery (playbooks/backups) -# -# This play is idempotent and kept permanently rather than run once and deleted: -# on a host that never had these it does nothing, and it guarantees a rebuilt or -# restored machine cannot quietly bring them back. - -- name: Remove the legacy Uptime Kuma monitoring units - hosts: managed - become: yes - - vars: - legacy_units: - - disk-usage-monitor - - system-healthcheck - - nodito-cpu-temp-monitor - - zfs-health-monitor - legacy_dirs: - - /opt/disk-monitoring - - /opt/system-healthcheck - - /opt/nodito-monitoring - - /opt/zfs-monitoring - - tasks: - - name: Find which legacy units exist here - ansible.builtin.stat: - path: "/etc/systemd/system/{{ item.0 }}.{{ item.1 }}" - register: legacy_unit_files - loop: "{{ legacy_units | product(['timer', 'service']) | list }}" - - # Stop and disable BEFORE deleting the unit file: systemd cannot disable a - # unit whose file has already gone, which would leave a dangling symlink in - # multi-user.target.wants and a warning on every daemon-reload. - - name: Stop and disable the legacy units - ansible.builtin.systemd: - name: "{{ item.item.0 }}.{{ item.item.1 }}" - state: stopped - enabled: no - loop: "{{ legacy_unit_files.results }}" - loop_control: - label: "{{ item.item.0 }}.{{ item.item.1 }}" - when: item.stat.exists - failed_when: false - - - name: Remove the legacy unit files - ansible.builtin.file: - path: "/etc/systemd/system/{{ item.item.0 }}.{{ item.item.1 }}" - state: absent - loop: "{{ legacy_unit_files.results }}" - loop_control: - label: "{{ item.item.0 }}.{{ item.item.1 }}" - when: item.stat.exists - - - name: Reload systemd - ansible.builtin.systemd: - daemon_reload: yes - - - name: Remove the legacy monitoring scripts and their logs - ansible.builtin.file: - path: "{{ item }}" - state: absent - loop: "{{ legacy_dirs }}" - - # An orphan predating all of this: mode 0644, not executable, referenced by - # no unit and no cron entry, pushing to a Kuma monitor. Superseded by - # ups-status-healthcheck. - - name: Remove the orphaned hand-written UPS heartbeat - ansible.builtin.file: - path: /usr/local/bin/ups-heartbeat.sh - state: absent - - - name: Confirm nothing still pushes to Uptime Kuma - ansible.builtin.shell: >- - grep -rl "uptime.contrapeso.xyz" /etc/systemd/system /usr/local/bin /opt 2>/dev/null || true - register: kuma_refs - changed_when: false - - - name: Report any remaining references - ansible.builtin.debug: - msg: >- - {{ 'clean - nothing references Uptime Kuma' - if kuma_refs.stdout | trim | length == 0 - else 'STILL REFERENCING KUMA: ' ~ kuma_refs.stdout_lines | join(', ') }} diff --git a/ansible/infra/410_disk_usage_alerts.yml b/ansible/infra/410_disk_usage_alerts.yml new file mode 100644 index 0000000..de02f53 --- /dev/null +++ b/ansible/infra/410_disk_usage_alerts.yml @@ -0,0 +1,330 @@ +- name: Deploy Disk Usage Monitoring + hosts: all + become: yes + vars_files: + - ../infra_vars.yml + - ../services_config.yml + - ../infra_secrets.yml + + vars: + disk_usage_threshold_percent: 80 + disk_check_interval_minutes: 15 + monitored_mount_point: "/" + monitoring_script_dir: /opt/disk-monitoring + monitoring_script_path: "{{ monitoring_script_dir }}/disk_usage_monitor.sh" + log_file: "{{ monitoring_script_dir }}/disk_usage_monitor.log" + systemd_service_name: disk-usage-monitor + # Uptime Kuma configuration (auto-configured from services_config.yml and infra_secrets.yml) + uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}" + ntfy_topic: "{{ service_settings.ntfy.topic }}" + + tasks: + - name: Validate Uptime Kuma configuration + assert: + that: + - uptime_kuma_api_url is defined + - uptime_kuma_api_url != "" + - uptime_kuma_username is defined + - uptime_kuma_username != "" + - uptime_kuma_password is defined + - uptime_kuma_password != "" + fail_msg: "uptime_kuma_api_url, uptime_kuma_username and uptime_kuma_password must be set" + + - name: Get hostname for monitor identification + command: hostname + register: host_name + changed_when: false + + - name: Set monitor name and group based on hostname and mount point + set_fact: + monitor_name: "disk-usage-{{ host_name.stdout }}-{{ monitored_mount_point | replace('/', 'root') }}" + monitor_friendly_name: "Disk Usage: {{ host_name.stdout }} ({{ monitored_mount_point }})" + uptime_kuma_monitor_group: "{{ host_name.stdout }} - infra" + + - name: Create Uptime Kuma monitor setup script + copy: + dest: /tmp/setup_uptime_kuma_monitor.py + content: | + #!/usr/bin/env python3 + import sys + import json + from uptime_kuma_api import UptimeKumaApi + + def main(): + api_url = sys.argv[1] + username = sys.argv[2] + password = sys.argv[3] + group_name = sys.argv[4] + monitor_name = sys.argv[5] + monitor_description = sys.argv[6] + interval = int(sys.argv[7]) + ntfy_topic = sys.argv[8] if len(sys.argv) > 8 else "alerts" + + api = UptimeKumaApi(api_url, timeout=60, wait_events=2.0) + api.login(username, password) + + # Get all monitors + monitors = api.get_monitors() + + # Get all notifications and find ntfy notification + notifications = api.get_notifications() + ntfy_notification = next((n for n in notifications if n.get('name') == f'ntfy ({ntfy_topic})'), None) + notification_id_list = {} + if ntfy_notification: + notification_id_list[ntfy_notification['id']] = True + + # Find or create group + group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None) + if not group: + group_result = api.add_monitor(type='group', name=group_name) + # Refresh to get the full group object with id + monitors = api.get_monitors() + group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None) + + # Find or create/update push monitor + existing_monitor = next((m for m in monitors if m.get('name') == monitor_name), None) + + monitor_data = { + 'type': 'push', + 'name': monitor_name, + 'parent': group['id'], + 'interval': interval, + 'upsideDown': True, + 'description': monitor_description, + 'notificationIDList': notification_id_list + } + + if existing_monitor: + monitor = api.edit_monitor(existing_monitor['id'], **monitor_data) + # Refresh to get the full monitor object with pushToken + monitors = api.get_monitors() + monitor = next((m for m in monitors if m.get('name') == monitor_name), None) + else: + monitor_result = api.add_monitor(**monitor_data) + # Refresh to get the full monitor object with pushToken + monitors = api.get_monitors() + monitor = next((m for m in monitors if m.get('name') == monitor_name), None) + + # Output result as JSON + result = { + 'monitor_id': monitor['id'], + 'push_token': monitor['pushToken'], + 'group_name': group_name, + 'group_id': group['id'], + 'monitor_name': monitor_name + } + print(json.dumps(result)) + + api.disconnect() + + if __name__ == '__main__': + main() + mode: '0755' + delegate_to: localhost + become: no + + - name: Run Uptime Kuma monitor setup script + command: > + {{ ansible_playbook_python }} + /tmp/setup_uptime_kuma_monitor.py + "{{ uptime_kuma_api_url }}" + "{{ uptime_kuma_username }}" + "{{ uptime_kuma_password }}" + "{{ uptime_kuma_monitor_group }}" + "{{ monitor_name }}" + "{{ monitor_friendly_name }} - Alerts when usage exceeds {{ disk_usage_threshold_percent }}%" + "{{ (disk_check_interval_minutes * 60) + 60 }}" + "{{ ntfy_topic }}" + register: monitor_setup_result + delegate_to: localhost + become: no + changed_when: false + + - name: Parse monitor setup result + set_fact: + monitor_info_parsed: "{{ monitor_setup_result.stdout | from_json }}" + + - name: Set push URL and monitor ID as facts + set_fact: + uptime_kuma_disk_usage_push_url: "{{ uptime_kuma_api_url }}/api/push/{{ monitor_info_parsed.push_token }}" + uptime_kuma_monitor_id: "{{ monitor_info_parsed.monitor_id }}" + + - name: Install required packages for disk monitoring + package: + name: + - curl + state: present + + - name: Create monitoring script directory + file: + path: "{{ monitoring_script_dir }}" + state: directory + owner: root + group: root + mode: '0755' + + - name: Create disk usage monitoring script + copy: + dest: "{{ monitoring_script_path }}" + content: | + #!/bin/bash + + # Disk Usage Monitoring Script + # Monitors disk usage and sends alerts to Uptime Kuma + # Mode: "No news is good news" - only sends alerts when disk usage is HIGH + + LOG_FILE="{{ log_file }}" + USAGE_THRESHOLD="{{ disk_usage_threshold_percent }}" + UPTIME_KUMA_URL="{{ uptime_kuma_disk_usage_push_url }}" + MOUNT_POINT="{{ monitored_mount_point }}" + + # Function to log messages + log_message() { + echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE" + } + + # Function to get disk usage percentage + get_disk_usage() { + local mount_point="$1" + local usage="" + + # Get disk usage percentage (without % sign) + usage=$(df -h "$mount_point" 2>/dev/null | awk 'NR==2 {gsub(/%/, "", $5); print $5}') + + if [ -z "$usage" ]; then + log_message "ERROR: Could not read disk usage for $mount_point" + return 1 + fi + + echo "$usage" + } + + # Function to get disk usage details + get_disk_details() { + local mount_point="$1" + df -h "$mount_point" 2>/dev/null | awk 'NR==2 {print "Used: "$3" / Total: "$2" ("$5" full)"}' + } + + # Function to send alert to Uptime Kuma when disk usage exceeds threshold + # With upside-down mode enabled, sending status=up will trigger an alert + send_uptime_kuma_alert() { + local usage="$1" + local details="$2" + local message="DISK FULL WARNING: ${MOUNT_POINT} is ${usage}% full (Threshold: ${USAGE_THRESHOLD}%) - ${details}" + + log_message "ALERT: $message" + + # Send push notification to Uptime Kuma with status=up + # In upside-down mode, status=up is treated as down/alert + response=$(curl -s -w "\n%{http_code}" -G \ + --data-urlencode "status=up" \ + --data-urlencode "msg=$message" \ + "$UPTIME_KUMA_URL" 2>&1) + http_code=$(echo "$response" | tail -n1) + + if [ "$http_code" = "200" ] || [ "$http_code" = "201" ]; then + log_message "Alert sent successfully to Uptime Kuma (HTTP $http_code)" + else + log_message "ERROR: Failed to send alert to Uptime Kuma (HTTP $http_code)" + fi + } + + # Main monitoring logic + main() { + log_message "Starting disk usage check for $MOUNT_POINT" + + # Get current disk usage + current_usage=$(get_disk_usage "$MOUNT_POINT") + + if [ $? -ne 0 ] || [ -z "$current_usage" ]; then + log_message "ERROR: Could not read disk usage" + exit 1 + fi + + # Get disk details + disk_details=$(get_disk_details "$MOUNT_POINT") + + log_message "Current disk usage: ${current_usage}% - $disk_details" + + # Check if usage exceeds threshold + if [ "$current_usage" -gt "$USAGE_THRESHOLD" ]; then + log_message "WARNING: Disk usage ${current_usage}% exceeds threshold ${USAGE_THRESHOLD}%" + send_uptime_kuma_alert "$current_usage" "$disk_details" + else + log_message "Disk usage is within normal range - no alert needed (no news is good news)" + fi + } + + # Run main function + main + owner: root + group: root + mode: '0755' + + - name: Create systemd service for disk usage monitoring + copy: + dest: "/etc/systemd/system/{{ systemd_service_name }}.service" + content: | + [Unit] + Description=Disk Usage Monitor + After=network.target + + [Service] + Type=oneshot + ExecStart={{ monitoring_script_path }} + User=root + StandardOutput=journal + StandardError=journal + + [Install] + WantedBy=multi-user.target + owner: root + group: root + mode: '0644' + + - name: Create systemd timer for disk usage monitoring + copy: + dest: "/etc/systemd/system/{{ systemd_service_name }}.timer" + content: | + [Unit] + Description=Run Disk Usage Monitor every {{ disk_check_interval_minutes }} minute(s) + Requires={{ systemd_service_name }}.service + + [Timer] + OnBootSec={{ disk_check_interval_minutes }}min + OnUnitActiveSec={{ disk_check_interval_minutes }}min + Persistent=true + + [Install] + WantedBy=timers.target + owner: root + group: root + mode: '0644' + + - name: Reload systemd daemon + systemd: + daemon_reload: yes + + - name: Enable and start disk usage monitoring timer + systemd: + name: "{{ systemd_service_name }}.timer" + enabled: yes + state: started + + - name: Test disk usage monitoring script + command: "{{ monitoring_script_path }}" + register: script_test + changed_when: false + + - name: Verify script execution + assert: + that: + - script_test.rc == 0 + fail_msg: "Disk usage monitoring script failed to execute properly" + + - name: Clean up temporary Uptime Kuma setup script + file: + path: /tmp/setup_uptime_kuma_monitor.py + state: absent + delegate_to: localhost + become: no diff --git a/ansible/infra/420_system_healthcheck.yml b/ansible/infra/420_system_healthcheck.yml new file mode 100644 index 0000000..fa507bd --- /dev/null +++ b/ansible/infra/420_system_healthcheck.yml @@ -0,0 +1,312 @@ +- name: Deploy System Healthcheck Monitoring + hosts: all + become: yes + vars_files: + - ../infra_vars.yml + - ../services_config.yml + - ../infra_secrets.yml + + vars: + healthcheck_interval_seconds: 60 # Send healthcheck every 60 seconds (1 minute) + healthcheck_timeout_seconds: 90 # Uptime Kuma should alert if no ping received within 90s + healthcheck_retries: 1 # Number of retries before alerting + monitoring_script_dir: /opt/system-healthcheck + monitoring_script_path: "{{ monitoring_script_dir }}/system_healthcheck.sh" + log_file: "{{ monitoring_script_dir }}/system_healthcheck.log" + systemd_service_name: system-healthcheck + # Uptime Kuma configuration (auto-configured from services_config.yml and infra_secrets.yml) + uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}" + ntfy_topic: "{{ service_settings.ntfy.topic }}" + + tasks: + - name: Validate Uptime Kuma configuration + assert: + that: + - uptime_kuma_api_url is defined + - uptime_kuma_api_url != "" + - uptime_kuma_username is defined + - uptime_kuma_username != "" + - uptime_kuma_password is defined + - uptime_kuma_password != "" + fail_msg: "uptime_kuma_api_url, uptime_kuma_username and uptime_kuma_password must be set" + + - name: Get hostname for monitor identification + command: hostname + register: host_name + changed_when: false + + - name: Set monitor name and group based on hostname + set_fact: + monitor_name: "system-healthcheck-{{ host_name.stdout }}" + monitor_friendly_name: "System Healthcheck: {{ host_name.stdout }}" + uptime_kuma_monitor_group: "{{ host_name.stdout }} - infra" + + - name: Create Uptime Kuma monitor setup script + copy: + dest: /tmp/setup_uptime_kuma_healthcheck_monitor.py + content: | + #!/usr/bin/env python3 + import sys + import json + from uptime_kuma_api import UptimeKumaApi + + def main(): + api_url = sys.argv[1] + username = sys.argv[2] + password = sys.argv[3] + group_name = sys.argv[4] + monitor_name = sys.argv[5] + monitor_description = sys.argv[6] + interval = int(sys.argv[7]) + retries = int(sys.argv[8]) + ntfy_topic = sys.argv[9] if len(sys.argv) > 9 else "alerts" + + api = UptimeKumaApi(api_url, timeout=60, wait_events=2.0) + api.login(username, password) + + # Get all monitors + monitors = api.get_monitors() + + # Get all notifications and find ntfy notification + notifications = api.get_notifications() + ntfy_notification = next((n for n in notifications if n.get('name') == f'ntfy ({ntfy_topic})'), None) + notification_id_list = {} + if ntfy_notification: + notification_id_list[ntfy_notification['id']] = True + + # Find or create group + group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None) + if not group: + group_result = api.add_monitor(type='group', name=group_name) + # Refresh to get the full group object with id + monitors = api.get_monitors() + group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None) + + # Find or create/update push monitor + existing_monitor = next((m for m in monitors if m.get('name') == monitor_name), None) + + monitor_data = { + 'type': 'push', + 'name': monitor_name, + 'parent': group['id'], + 'interval': interval, + 'upsideDown': False, # Normal mode: receiving pings = healthy + 'maxretries': retries, + 'description': monitor_description, + 'notificationIDList': notification_id_list + } + + if existing_monitor: + monitor = api.edit_monitor(existing_monitor['id'], **monitor_data) + # Refresh to get the full monitor object with pushToken + monitors = api.get_monitors() + monitor = next((m for m in monitors if m.get('name') == monitor_name), None) + else: + monitor_result = api.add_monitor(**monitor_data) + # Refresh to get the full monitor object with pushToken + monitors = api.get_monitors() + monitor = next((m for m in monitors if m.get('name') == monitor_name), None) + + # Output result as JSON + result = { + 'monitor_id': monitor['id'], + 'push_token': monitor['pushToken'], + 'group_name': group_name, + 'group_id': group['id'], + 'monitor_name': monitor_name + } + print(json.dumps(result)) + + api.disconnect() + + if __name__ == '__main__': + main() + mode: '0755' + delegate_to: localhost + become: no + + - name: Run Uptime Kuma monitor setup script + command: > + {{ ansible_playbook_python }} + /tmp/setup_uptime_kuma_healthcheck_monitor.py + "{{ uptime_kuma_api_url }}" + "{{ uptime_kuma_username }}" + "{{ uptime_kuma_password }}" + "{{ uptime_kuma_monitor_group }}" + "{{ monitor_name }}" + "{{ monitor_friendly_name }} - Regular healthcheck ping every {{ healthcheck_interval_seconds }}s" + "{{ healthcheck_timeout_seconds }}" + "{{ healthcheck_retries }}" + "{{ ntfy_topic }}" + register: monitor_setup_result + delegate_to: localhost + become: no + changed_when: false + + - name: Parse monitor setup result + set_fact: + monitor_info_parsed: "{{ monitor_setup_result.stdout | from_json }}" + + - name: Set push URL and monitor ID as facts + set_fact: + uptime_kuma_healthcheck_push_url: "{{ uptime_kuma_api_url }}/api/push/{{ monitor_info_parsed.push_token }}" + uptime_kuma_monitor_id: "{{ monitor_info_parsed.monitor_id }}" + + - name: Install required packages for healthcheck monitoring + package: + name: + - curl + state: present + + - name: Create monitoring script directory + file: + path: "{{ monitoring_script_dir }}" + state: directory + owner: root + group: root + mode: '0755' + + - name: Create system healthcheck script + copy: + dest: "{{ monitoring_script_path }}" + content: | + #!/bin/bash + + # System Healthcheck Script + # Sends regular heartbeat pings to Uptime Kuma + # This ensures the system is running and able to communicate + + LOG_FILE="{{ log_file }}" + UPTIME_KUMA_URL="{{ uptime_kuma_healthcheck_push_url }}" + HOSTNAME=$(hostname) + + # Function to log messages + log_message() { + echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE" + } + + # Function to send healthcheck ping to Uptime Kuma + send_healthcheck() { + local uptime_seconds=$(awk '{print int($1)}' /proc/uptime) + local uptime_days=$((uptime_seconds / 86400)) + local uptime_hours=$(((uptime_seconds % 86400) / 3600)) + local uptime_minutes=$(((uptime_seconds % 3600) / 60)) + + local message="System healthy - Uptime: ${uptime_days}d ${uptime_hours}h ${uptime_minutes}m" + + log_message "Sending healthcheck ping: $message" + + # Send push notification to Uptime Kuma with status=up + encoded_message=$(printf '%s\n' "$message" | sed 's/ /%20/g; s/(/%28/g; s/)/%29/g; s/:/%3A/g; s/\//%2F/g') + response=$(curl -s -w "\n%{http_code}" "$UPTIME_KUMA_URL?status=up&msg=$encoded_message" 2>&1) + http_code=$(echo "$response" | tail -n1) + + if [ "$http_code" = "200" ] || [ "$http_code" = "201" ]; then + log_message "Healthcheck ping sent successfully (HTTP $http_code)" + else + log_message "ERROR: Failed to send healthcheck ping (HTTP $http_code)" + return 1 + fi + } + + # Main healthcheck logic + main() { + log_message "Starting system healthcheck for $HOSTNAME" + + # Send healthcheck ping + if send_healthcheck; then + log_message "Healthcheck completed successfully" + else + log_message "ERROR: Healthcheck failed" + exit 1 + fi + } + + # Run main function + main + owner: root + group: root + mode: '0755' + + - name: Create systemd service for system healthcheck + copy: + dest: "/etc/systemd/system/{{ systemd_service_name }}.service" + content: | + [Unit] + Description=System Healthcheck Monitor + After=network.target + + [Service] + Type=oneshot + ExecStart={{ monitoring_script_path }} + User=root + StandardOutput=journal + StandardError=journal + + [Install] + WantedBy=multi-user.target + owner: root + group: root + mode: '0644' + + - name: Create systemd timer for system healthcheck + copy: + dest: "/etc/systemd/system/{{ systemd_service_name }}.timer" + content: | + [Unit] + Description=Run System Healthcheck every minute + Requires={{ systemd_service_name }}.service + + [Timer] + OnBootSec=30sec + OnUnitActiveSec={{ healthcheck_interval_seconds }}sec + Persistent=true + + [Install] + WantedBy=timers.target + owner: root + group: root + mode: '0644' + + - name: Reload systemd daemon + systemd: + daemon_reload: yes + + - name: Enable and start system healthcheck timer + systemd: + name: "{{ systemd_service_name }}.timer" + enabled: yes + state: started + + - name: Test system healthcheck script + command: "{{ monitoring_script_path }}" + register: script_test + changed_when: false + + - name: Verify script execution + assert: + that: + - script_test.rc == 0 + fail_msg: "System healthcheck script failed to execute properly" + + - name: Display monitor information + debug: + msg: | + ✓ System healthcheck monitoring deployed successfully! + + Monitor Name: {{ monitor_friendly_name }} + Monitor Group: {{ uptime_kuma_monitor_group }} + Healthcheck Interval: Every {{ healthcheck_interval_seconds }} seconds (1 minute) + Timeout: {{ healthcheck_timeout_seconds }} seconds (90s) + Retries: {{ healthcheck_retries }} + + The system will send a heartbeat ping every minute. + Uptime Kuma will alert if no ping is received within 90 seconds (with 1 retry). + + - name: Clean up temporary Uptime Kuma setup script + file: + path: /tmp/setup_uptime_kuma_healthcheck_monitor.py + state: absent + delegate_to: localhost + become: no + diff --git a/ansible/infra/430_cpu_temp_alerts.yml b/ansible/infra/430_cpu_temp_alerts.yml new file mode 100644 index 0000000..3b87102 --- /dev/null +++ b/ansible/infra/430_cpu_temp_alerts.yml @@ -0,0 +1,316 @@ +- name: Deploy CPU Temperature Monitoring + hosts: nodito_host + become: yes + vars_files: + - ../infra_vars.yml + - ../services_config.yml + - ../infra_secrets.yml + + vars: + temp_threshold_celsius: 80 + temp_check_interval_minutes: 1 + monitoring_script_dir: /opt/nodito-monitoring + monitoring_script_path: "{{ monitoring_script_dir }}/cpu_temp_monitor.sh" + log_file: "{{ monitoring_script_dir }}/cpu_temp_monitor.log" + systemd_service_name: nodito-cpu-temp-monitor + uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}" + ntfy_topic: "{{ service_settings.ntfy.topic }}" + + tasks: + - name: Validate Uptime Kuma configuration + assert: + that: + - uptime_kuma_api_url is defined + - uptime_kuma_api_url != "" + - uptime_kuma_username is defined + - uptime_kuma_username != "" + - uptime_kuma_password is defined + - uptime_kuma_password != "" + fail_msg: "uptime_kuma_api_url, uptime_kuma_username and uptime_kuma_password must be set" + + - name: Get hostname for monitor identification + command: hostname + register: host_name + changed_when: false + + - name: Set monitor name and group based on hostname + set_fact: + monitor_name: "cpu-temp-{{ host_name.stdout }}" + monitor_friendly_name: "CPU Temperature: {{ host_name.stdout }}" + uptime_kuma_monitor_group: "{{ host_name.stdout }} - infra" + + - name: Create Uptime Kuma CPU temperature monitor setup script + copy: + dest: /tmp/setup_uptime_kuma_cpu_temp_monitor.py + content: | + #!/usr/bin/env python3 + import sys + import json + from uptime_kuma_api import UptimeKumaApi + + def main(): + api_url = sys.argv[1] + username = sys.argv[2] + password = sys.argv[3] + group_name = sys.argv[4] + monitor_name = sys.argv[5] + monitor_description = sys.argv[6] + interval = int(sys.argv[7]) + ntfy_topic = sys.argv[8] if len(sys.argv) > 8 else "alerts" + + api = UptimeKumaApi(api_url, timeout=60, wait_events=2.0) + api.login(username, password) + + monitors = api.get_monitors() + notifications = api.get_notifications() + + ntfy_notification = next((n for n in notifications if n.get('name') == f'ntfy ({ntfy_topic})'), None) + notification_id_list = {} + if ntfy_notification: + notification_id_list[ntfy_notification['id']] = True + + group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None) + if not group: + api.add_monitor(type='group', name=group_name) + monitors = api.get_monitors() + group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None) + + existing_monitor = next((m for m in monitors if m.get('name') == monitor_name), None) + + monitor_data = { + 'type': 'push', + 'name': monitor_name, + 'parent': group['id'], + 'interval': interval, + 'upsideDown': True, + 'description': monitor_description, + 'notificationIDList': notification_id_list + } + + if existing_monitor: + api.edit_monitor(existing_monitor['id'], **monitor_data) + else: + api.add_monitor(**monitor_data) + + monitors = api.get_monitors() + monitor = next((m for m in monitors if m.get('name') == monitor_name), None) + + result = { + 'monitor_id': monitor['id'], + 'push_token': monitor['pushToken'], + 'group_name': group_name, + 'group_id': group['id'], + 'monitor_name': monitor_name + } + print(json.dumps(result)) + + api.disconnect() + + if __name__ == '__main__': + main() + mode: '0755' + delegate_to: localhost + become: no + + - name: Run Uptime Kuma monitor setup script + command: > + {{ ansible_playbook_python }} + /tmp/setup_uptime_kuma_cpu_temp_monitor.py + "{{ uptime_kuma_api_url }}" + "{{ uptime_kuma_username }}" + "{{ uptime_kuma_password }}" + "{{ uptime_kuma_monitor_group }}" + "{{ monitor_name }}" + "{{ monitor_friendly_name }} - Alerts when temperature exceeds {{ temp_threshold_celsius }}°C" + "{{ (temp_check_interval_minutes * 60) + 60 }}" + "{{ ntfy_topic }}" + register: monitor_setup_result + delegate_to: localhost + become: no + changed_when: false + + - name: Parse monitor setup result + set_fact: + monitor_info_parsed: "{{ monitor_setup_result.stdout | from_json }}" + + - name: Set push URL and monitor ID as facts + set_fact: + uptime_kuma_cpu_temp_push_url: "{{ uptime_kuma_api_url }}/api/push/{{ monitor_info_parsed.push_token }}" + uptime_kuma_monitor_id: "{{ monitor_info_parsed.monitor_id }}" + + - name: Install required packages for temperature monitoring + package: + name: + - lm-sensors + - curl + - jq + - bc + state: present + + - name: Create monitoring script directory + file: + path: "{{ monitoring_script_dir }}" + state: directory + owner: root + group: root + mode: '0755' + + - name: Create CPU temperature monitoring script + copy: + dest: "{{ monitoring_script_path }}" + content: | + #!/bin/bash + + # CPU Temperature Monitoring Script + # Monitors CPU temperature and sends alerts to Uptime Kuma + + LOG_FILE="{{ log_file }}" + TEMP_THRESHOLD="{{ temp_threshold_celsius }}" + UPTIME_KUMA_URL="{{ uptime_kuma_cpu_temp_push_url }}" + + log_message() { + echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE" + } + + get_cpu_temp() { + local temp="" + + if command -v sensors >/dev/null 2>&1; then + temp=$(sensors 2>/dev/null | grep -E "Core 0|Package id 0|Tdie|Tctl" | head -1 | grep -oE '[0-9]+\.[0-9]+°C' | grep -oE '[0-9]+\.[0-9]+') + fi + + if [ -z "$temp" ] && [ -f /sys/class/thermal/thermal_zone0/temp ]; then + temp=$(cat /sys/class/thermal/thermal_zone0/temp) + temp=$(echo "scale=1; $temp/1000" | bc -l 2>/dev/null || echo "$temp") + fi + + if [ -z "$temp" ] && command -v acpi >/dev/null 2>&1; then + temp=$(acpi -t 2>/dev/null | grep -oE '[0-9]+\.[0-9]+' | head -1) + fi + + echo "$temp" + } + + send_uptime_kuma_alert() { + local temp="$1" + local message="CPU Temperature Alert: ${temp}°C (Threshold: ${TEMP_THRESHOLD}°C)" + + log_message "ALERT: $message" + + encoded_message=$(printf '%s\n' "$message" | sed 's/ /%20/g; s/°/%C2%B0/g; s/(/%28/g; s/)/%29/g; s/:/%3A/g') + response=$(curl -s -w "\n%{http_code}" "$UPTIME_KUMA_URL?status=up&msg=$encoded_message" 2>&1) + http_code=$(echo "$response" | tail -n1) + + if [ "$http_code" = "200" ] || [ "$http_code" = "201" ]; then + log_message "Alert sent successfully to Uptime Kuma (HTTP $http_code)" + else + log_message "ERROR: Failed to send alert to Uptime Kuma (HTTP $http_code)" + fi + } + + main() { + log_message "Starting CPU temperature check" + + current_temp=$(get_cpu_temp) + + if [ -z "$current_temp" ]; then + log_message "ERROR: Could not read CPU temperature" + exit 1 + fi + + log_message "Current CPU temperature: ${current_temp}°C" + + if (( $(echo "$current_temp > $TEMP_THRESHOLD" | bc -l) )); then + log_message "WARNING: CPU temperature ${current_temp}°C exceeds threshold ${TEMP_THRESHOLD}°C" + send_uptime_kuma_alert "$current_temp" + else + log_message "CPU temperature is within normal range" + fi + } + + main + owner: root + group: root + mode: '0755' + + - name: Create systemd service for CPU temperature monitoring + copy: + dest: "/etc/systemd/system/{{ systemd_service_name }}.service" + content: | + [Unit] + Description=CPU Temperature Monitor + After=network.target + + [Service] + Type=oneshot + ExecStart={{ monitoring_script_path }} + User=root + StandardOutput=journal + StandardError=journal + + [Install] + WantedBy=multi-user.target + owner: root + group: root + mode: '0644' + + - name: Create systemd timer for CPU temperature monitoring + copy: + dest: "/etc/systemd/system/{{ systemd_service_name }}.timer" + content: | + [Unit] + Description=Run CPU Temperature Monitor every {{ temp_check_interval_minutes }} minute(s) + Requires={{ systemd_service_name }}.service + + [Timer] + OnBootSec={{ temp_check_interval_minutes }}min + OnUnitActiveSec={{ temp_check_interval_minutes }}min + Persistent=true + + [Install] + WantedBy=timers.target + owner: root + group: root + mode: '0644' + + - name: Reload systemd daemon + systemd: + daemon_reload: yes + + - name: Enable and start CPU temperature monitoring timer + systemd: + name: "{{ systemd_service_name }}.timer" + enabled: yes + state: started + + - name: Test CPU temperature monitoring script + command: "{{ monitoring_script_path }}" + register: script_test + changed_when: false + + - name: Verify script execution + assert: + that: + - script_test.rc == 0 + fail_msg: "CPU temperature monitoring script failed to execute properly" + + - name: Display monitoring configuration + debug: + msg: + - "CPU Temperature Monitoring configured successfully" + - "Temperature threshold: {{ temp_threshold_celsius }}°C" + - "Check interval: {{ temp_check_interval_minutes }} minute(s)" + - "Monitor Name: {{ monitor_friendly_name }}" + - "Monitor Group: {{ uptime_kuma_monitor_group }}" + - "Uptime Kuma Push URL: {{ uptime_kuma_cpu_temp_push_url }}" + - "Monitoring script: {{ monitoring_script_path }}" + - "Systemd Service: {{ systemd_service_name }}.service" + - "Systemd Timer: {{ systemd_service_name }}.timer" + + - name: Clean up temporary Uptime Kuma setup script + file: + path: /tmp/setup_uptime_kuma_cpu_temp_monitor.py + state: absent + delegate_to: localhost + become: no + diff --git a/ansible/infra/900_install_rsync.yml b/ansible/infra/900_install_rsync.yml index b690009..c0b7318 100644 --- a/ansible/infra/900_install_rsync.yml +++ b/ansible/infra/900_install_rsync.yml @@ -1,5 +1,7 @@ - name: Install rsync - hosts: managed + hosts: all + vars_files: + - ../infra_vars.yml become: true tasks: diff --git a/ansible/infra/910_docker_playbook.yml b/ansible/infra/910_docker_playbook.yml index 62b7147..8e8e430 100644 --- a/ansible/infra/910_docker_playbook.yml +++ b/ansible/infra/910_docker_playbook.yml @@ -1,5 +1,5 @@ - name: Install Docker and Docker Compose on Debian 12 - hosts: managed + hosts: all become: yes tasks: @@ -25,7 +25,6 @@ name: - ca-certificates - curl - - gnupg state: present - name: Create directory for Docker GPG key diff --git a/ansible/infra/920_join_headscale_mesh.yml b/ansible/infra/920_join_headscale_mesh.yml index 4decb5d..8d06d44 100644 --- a/ansible/infra/920_join_headscale_mesh.yml +++ b/ansible/infra/920_join_headscale_mesh.yml @@ -1,10 +1,14 @@ - name: Join machine to headscale mesh network - hosts: managed + hosts: all become: yes + vars_files: + - ../infra_vars.yml + - ../services_config.yml vars: headscale_host_name: "spacey" headscale_subdomain: "{{ subdomains.headscale }}" headscale_domain: "https://{{ headscale_subdomain }}.{{ root_domain }}" + headscale_namespace: "{{ service_settings.headscale.namespace }}" tasks: - name: Set facts for headscale server connection diff --git a/ansible/infra/nodito/30_proxmox_bootstrap_playbook.yml b/ansible/infra/nodito/30_proxmox_bootstrap_playbook.yml index 4edab06..02c6679 100644 --- a/ansible/infra/nodito/30_proxmox_bootstrap_playbook.yml +++ b/ansible/infra/nodito/30_proxmox_bootstrap_playbook.yml @@ -1,6 +1,8 @@ - name: Bootstrap Nodito SSH Key Access - hosts: hypervisor + hosts: nodito_host become: true + vars_files: + - ../infra_vars.yml tasks: - name: Install sudo package diff --git a/ansible/infra/nodito/31_proxmox_community_repos_playbook.yml b/ansible/infra/nodito/31_proxmox_community_repos_playbook.yml index 0fab184..b0be2ef 100644 --- a/ansible/infra/nodito/31_proxmox_community_repos_playbook.yml +++ b/ansible/infra/nodito/31_proxmox_community_repos_playbook.yml @@ -1,6 +1,8 @@ - name: Switch Proxmox VE from Enterprise to Community Repositories - hosts: hypervisor + hosts: nodito_host become: true + vars_files: + - ../infra_vars.yml tasks: - name: Check for deb822 sources format diff --git a/ansible/infra/nodito/32_zfs_pool_setup_playbook.yml b/ansible/infra/nodito/32_zfs_pool_setup_playbook.yml index 2efbf8c..cb72328 100644 --- a/ansible/infra/nodito/32_zfs_pool_setup_playbook.yml +++ b/ansible/infra/nodito/32_zfs_pool_setup_playbook.yml @@ -1,6 +1,9 @@ - name: Setup ZFS RAID 1 Pool for Proxmox Storage - hosts: hypervisor + hosts: nodito_host become: true + vars_files: + - ../infra_vars.yml + - nodito_vars.yml tasks: - name: Verify Proxmox VE is running @@ -136,18 +139,17 @@ Config file exists: {{ storage_cfg_file.stat.exists }} Storage check result: {{ storage_exists_check.rc }} Pool exists: {{ zfs_pool_exists.rc == 0 }} + Will remove storage: {{ zfs_pool_exists.rc == 0 and storage_exists_check.rc == 0 }} Will add storage: {{ zfs_pool_exists.rc == 0 and storage_exists_check.rc != 0 }} - # Registration is add-only on purpose. There used to be a "Remove existing - # storage if it exists" task here that ran `pvesm remove` whenever the - # storage WAS present, paired with an add that only ran when it was ABSENT. - # The two conditions are mutually exclusive, so a real run against a - # correctly-configured hypervisor removed the storage entry backing every VM - # and never put it back. It would also have dropped `mountpoint /var/lib/vz`, - # which the live entry has and which `pvesm add` below does not set. - # - # If the storage entry ever needs its options changed, edit - # /etc/pve/storage.cfg or use `pvesm set` - do not re-register it from here. + - name: Remove existing storage if it exists + command: pvesm remove {{ zfs_pool_name }} + register: pvesm_remove_result + failed_when: false + when: + - zfs_pool_exists.rc == 0 + - storage_exists_check.rc == 0 + - name: Add ZFS pool storage to Proxmox using pvesm command: > pvesm add zfspool {{ zfs_pool_name }} @@ -169,56 +171,498 @@ msg: "ZFS pool {{ zfs_pool_name }} is not in a healthy state" when: "'ONLINE' not in final_zfs_status.stdout" -# ───────────────────────────────────────────────────────────────────────────── -# The monthly scrub. -# -# The ZFS HEALTH CHECK that used to share this play is gone: it is now the -# zfs-health check in infra/400_host_monitoring.yml, which carries the same five -# conditions - pool state, device states, resilver in progress, read/write/ -# checksum errors, and errors from the last scan - but reports to Gatus like -# every other host check instead of owning its own push plumbing. -# -# The scrub itself stays here, because it is not monitoring: it is the -# maintenance that gives the health check something true to report. A pool that -# is never scrubbed has no idea whether it is healthy. -# ───────────────────────────────────────────────────────────────────────────── -- name: Schedule the monthly ZFS scrub - hosts: hypervisor +- name: Setup ZFS Pool Health Monitoring and Monthly Scrubs + hosts: nodito become: true vars_files: - ../../infra_vars.yml + - ../../services_config.yml + - ../../infra_secrets.yml + - nodito_vars.yml vars: + zfs_check_interval_seconds: 86400 # 24 hours + zfs_check_timeout_seconds: 90000 # ~25 hours (interval + buffer) + zfs_check_retries: 1 + zfs_monitoring_script_dir: /opt/zfs-monitoring + zfs_monitoring_script_path: "{{ zfs_monitoring_script_dir }}/zfs_health_monitor.sh" + zfs_log_file: "{{ zfs_monitoring_script_dir }}/zfs_health_monitor.log" + zfs_systemd_health_service_name: zfs-health-monitor zfs_systemd_scrub_service_name: zfs-monthly-scrub + uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}" + ntfy_topic: "{{ service_settings.ntfy.topic }}" tasks: + - name: Validate Uptime Kuma configuration + assert: + that: + - uptime_kuma_api_url is defined + - uptime_kuma_api_url != "" + - uptime_kuma_username is defined + - uptime_kuma_username != "" + - uptime_kuma_password is defined + - uptime_kuma_password != "" + fail_msg: "uptime_kuma_api_url, uptime_kuma_username and uptime_kuma_password must be set" + + - name: Get hostname for monitor identification + command: hostname + register: host_name + changed_when: false + + - name: Set monitor name and group based on hostname + set_fact: + monitor_name: "zfs-health-{{ host_name.stdout }}" + monitor_friendly_name: "ZFS Pool Health: {{ host_name.stdout }}" + uptime_kuma_monitor_group: "{{ host_name.stdout }} - infra" + + - name: Create Uptime Kuma ZFS health monitor setup script + copy: + dest: /tmp/setup_uptime_kuma_zfs_monitor.py + content: | + #!/usr/bin/env python3 + import sys + import json + from uptime_kuma_api import UptimeKumaApi + + def main(): + api_url = sys.argv[1] + username = sys.argv[2] + password = sys.argv[3] + group_name = sys.argv[4] + monitor_name = sys.argv[5] + monitor_description = sys.argv[6] + interval = int(sys.argv[7]) + retries = int(sys.argv[8]) + ntfy_topic = sys.argv[9] if len(sys.argv) > 9 else "alerts" + + api = UptimeKumaApi(api_url, timeout=120, wait_events=2.0) + api.login(username, password) + + # Get all monitors + monitors = api.get_monitors() + + # Get all notifications and find ntfy notification + notifications = api.get_notifications() + ntfy_notification = next((n for n in notifications if n.get('name') == f'ntfy ({ntfy_topic})'), None) + notification_id_list = {} + if ntfy_notification: + notification_id_list[ntfy_notification['id']] = True + + # Find or create group + group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None) + if not group: + group_result = api.add_monitor(type='group', name=group_name) + # Refresh to get the full group object with id + monitors = api.get_monitors() + group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None) + + # Find or create/update push monitor + existing_monitor = next((m for m in monitors if m.get('name') == monitor_name), None) + + monitor_data = { + 'type': 'push', + 'name': monitor_name, + 'parent': group['id'], + 'interval': interval, + 'upsideDown': False, # Normal heartbeat mode: receiving pings = healthy + 'maxretries': retries, + 'description': monitor_description, + 'notificationIDList': notification_id_list + } + + if existing_monitor: + monitor = api.edit_monitor(existing_monitor['id'], **monitor_data) + # Refresh to get the full monitor object with pushToken + monitors = api.get_monitors() + monitor = next((m for m in monitors if m.get('name') == monitor_name), None) + else: + monitor_result = api.add_monitor(**monitor_data) + # Refresh to get the full monitor object with pushToken + monitors = api.get_monitors() + monitor = next((m for m in monitors if m.get('name') == monitor_name), None) + + # Output result as JSON + result = { + 'monitor_id': monitor['id'], + 'push_token': monitor['pushToken'], + 'group_name': group_name, + 'group_id': group['id'], + 'monitor_name': monitor_name + } + print(json.dumps(result)) + + api.disconnect() + + if __name__ == '__main__': + main() + mode: '0755' + delegate_to: localhost + become: no + + - name: Run Uptime Kuma ZFS monitor setup script + command: > + {{ ansible_playbook_python }} + /tmp/setup_uptime_kuma_zfs_monitor.py + "{{ uptime_kuma_api_url }}" + "{{ uptime_kuma_username }}" + "{{ uptime_kuma_password }}" + "{{ uptime_kuma_monitor_group }}" + "{{ monitor_name }}" + "{{ monitor_friendly_name }} - Daily health check for pool {{ zfs_pool_name }}" + "{{ zfs_check_timeout_seconds }}" + "{{ zfs_check_retries }}" + "{{ ntfy_topic }}" + register: monitor_setup_result + delegate_to: localhost + become: no + changed_when: false + + - name: Parse monitor setup result + set_fact: + monitor_info_parsed: "{{ monitor_setup_result.stdout | from_json }}" + + - name: Set push URL and monitor ID as facts + set_fact: + uptime_kuma_zfs_push_url: "{{ uptime_kuma_api_url }}/api/push/{{ monitor_info_parsed.push_token }}" + uptime_kuma_monitor_id: "{{ monitor_info_parsed.monitor_id }}" + + - name: Install required packages for ZFS monitoring + package: + name: + - curl + - jq + state: present + + - name: Create monitoring script directory + file: + path: "{{ zfs_monitoring_script_dir }}" + state: directory + owner: root + group: root + mode: '0755' + + - name: Create ZFS health monitoring script + copy: + dest: "{{ zfs_monitoring_script_path }}" + content: | + #!/bin/bash + + # ZFS Pool Health Monitoring Script + # Checks ZFS pool health using JSON output and sends heartbeat to Uptime Kuma if healthy + # If any issues detected, does NOT send heartbeat (triggers timeout alert) + + LOG_FILE="{{ zfs_log_file }}" + UPTIME_KUMA_URL="{{ uptime_kuma_zfs_push_url }}" + POOL_NAME="{{ zfs_pool_name }}" + HOSTNAME=$(hostname) + + # Function to log messages + log_message() { + echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE" + } + + # Function to check pool health using JSON output + check_pool_health() { + local pool="$1" + local issues_found=0 + + # Get pool status as JSON + local pool_json + pool_json=$(zpool status -j "$pool" 2>&1) + + if [ $? -ne 0 ]; then + log_message "ERROR: Failed to get pool status for $pool" + log_message " -> $pool_json" + return 1 + fi + + # Check 1: Pool state must be ONLINE + local pool_state + pool_state=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].state') + + if [ "$pool_state" != "ONLINE" ]; then + log_message "ISSUE: Pool state is $pool_state (expected ONLINE)" + issues_found=1 + else + log_message "OK: Pool state is ONLINE" + fi + + # Check 2: Check all vdevs and devices for non-ONLINE states + local bad_states + bad_states=$(echo "$pool_json" | jq -r --arg pool "$pool" ' + .pools[$pool].vdevs[] | + .. | objects | + select(.state? and .state != "ONLINE") | + "\(.name // "unknown"): \(.state)" + ' 2>/dev/null) + + if [ -n "$bad_states" ]; then + log_message "ISSUE: Found devices not in ONLINE state:" + echo "$bad_states" | while read -r line; do + log_message " -> $line" + done + issues_found=1 + else + log_message "OK: All devices are ONLINE" + fi + + # Check 3: Check for resilvering in progress + local scan_function scan_state + scan_function=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.function // "NONE"') + scan_state=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.state // "NONE"') + + if [ "$scan_function" = "RESILVER" ] && [ "$scan_state" = "SCANNING" ]; then + local resilver_progress + resilver_progress=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.issued // "unknown"') + log_message "ISSUE: Pool is currently resilvering (disk reconstruction in progress) - ${resilver_progress} processed" + issues_found=1 + fi + + # Check 4: Check for read/write/checksum errors on all devices + # Note: ZFS JSON output has error counts as strings, so convert to numbers for comparison + local devices_with_errors + devices_with_errors=$(echo "$pool_json" | jq -r --arg pool "$pool" ' + .pools[$pool].vdevs[] | + .. | objects | + select(.name? and ((.read_errors // "0" | tonumber) > 0 or (.write_errors // "0" | tonumber) > 0 or (.checksum_errors // "0" | tonumber) > 0)) | + "\(.name): read=\(.read_errors // 0) write=\(.write_errors // 0) cksum=\(.checksum_errors // 0)" + ' 2>/dev/null) + + if [ -n "$devices_with_errors" ]; then + log_message "ISSUE: Found devices with I/O errors:" + echo "$devices_with_errors" | while read -r line; do + log_message " -> $line" + done + issues_found=1 + else + log_message "OK: No read/write/checksum errors detected" + fi + + # Check 5: Check for scan errors (from last scrub/resilver) + local scan_errors + scan_errors=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.errors // "0"') + + if [ "$scan_errors" != "0" ] && [ "$scan_errors" != "null" ] && [ -n "$scan_errors" ]; then + log_message "ISSUE: Last scan reported $scan_errors errors" + issues_found=1 + else + log_message "OK: No scan errors" + fi + + return $issues_found + } + + # Function to get last scrub info for status message + get_scrub_info() { + local pool="$1" + local pool_json + pool_json=$(zpool status -j "$pool" 2>/dev/null) + + local scan_func scan_state scan_start + scan_func=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.function // "NONE"') + scan_state=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.state // "NONE"') + scan_start=$(echo "$pool_json" | jq -r --arg pool "$pool" '.pools[$pool].scan_stats.start_time // ""') + + if [ "$scan_func" = "SCRUB" ] && [ "$scan_state" = "SCANNING" ]; then + echo "scrub in progress (started $scan_start)" + elif [ "$scan_func" = "SCRUB" ] && [ -n "$scan_start" ]; then + echo "last scrub: $scan_start" + else + echo "no scrub history" + fi + } + + # Function to send heartbeat to Uptime Kuma + send_heartbeat() { + local message="$1" + + log_message "Sending heartbeat to Uptime Kuma: $message" + + # URL encode the message + local encoded_message + encoded_message=$(printf '%s\n' "$message" | sed 's/ /%20/g; s/(/%28/g; s/)/%29/g; s/:/%3A/g; s/\//%2F/g') + + local response http_code + response=$(curl -s -w "\n%{http_code}" "$UPTIME_KUMA_URL?status=up&msg=$encoded_message" 2>&1) + http_code=$(echo "$response" | tail -n1) + + if [ "$http_code" = "200" ] || [ "$http_code" = "201" ]; then + log_message "Heartbeat sent successfully (HTTP $http_code)" + return 0 + else + log_message "ERROR: Failed to send heartbeat (HTTP $http_code)" + return 1 + fi + } + + # Main health check logic + main() { + log_message "==========================================" + log_message "Starting ZFS health check for pool: $POOL_NAME on $HOSTNAME" + + # Run all health checks + if check_pool_health "$POOL_NAME"; then + # All checks passed - send heartbeat + local scrub_info + scrub_info=$(get_scrub_info "$POOL_NAME") + + local message="Pool $POOL_NAME healthy ($scrub_info)" + send_heartbeat "$message" + + log_message "Health check completed: ALL OK" + exit 0 + else + # Issues found - do NOT send heartbeat (will trigger timeout alert) + log_message "Health check completed: ISSUES DETECTED - NOT sending heartbeat" + log_message "Uptime Kuma will alert after timeout due to missing heartbeat" + exit 1 + fi + } + + # Run main function + main + owner: root + group: root + mode: '0755' + + - name: Create systemd service for ZFS health monitoring + copy: + dest: "/etc/systemd/system/{{ zfs_systemd_health_service_name }}.service" + content: | + [Unit] + Description=ZFS Pool Health Monitor + After=zfs.target network.target + + [Service] + Type=oneshot + ExecStart={{ zfs_monitoring_script_path }} + User=root + StandardOutput=journal + StandardError=journal + + [Install] + WantedBy=multi-user.target + owner: root + group: root + mode: '0644' + + - name: Create systemd timer for daily ZFS health monitoring + copy: + dest: "/etc/systemd/system/{{ zfs_systemd_health_service_name }}.timer" + content: | + [Unit] + Description=Run ZFS Pool Health Monitor daily + Requires={{ zfs_systemd_health_service_name }}.service + + [Timer] + OnBootSec=5min + OnUnitActiveSec={{ zfs_check_interval_seconds }}sec + Persistent=true + + [Install] + WantedBy=timers.target + owner: root + group: root + mode: '0644' + - name: Create systemd service for ZFS monthly scrub - template: - src: templates/zfs-monthly-scrub.service.j2 + copy: dest: "/etc/systemd/system/{{ zfs_systemd_scrub_service_name }}.service" + content: | + [Unit] + Description=ZFS Monthly Scrub for {{ zfs_pool_name }} + After=zfs.target + + [Service] + Type=oneshot + ExecStart=/sbin/zpool scrub {{ zfs_pool_name }} + User=root + StandardOutput=journal + StandardError=journal + + [Install] + WantedBy=multi-user.target owner: root group: root mode: '0644' - name: Create systemd timer for monthly ZFS scrub - template: - src: templates/zfs-monthly-scrub.timer.j2 + copy: dest: "/etc/systemd/system/{{ zfs_systemd_scrub_service_name }}.timer" + content: | + [Unit] + Description=Run ZFS Scrub on last day of every month at 4:00 AM + Requires={{ zfs_systemd_scrub_service_name }}.service + + [Timer] + OnCalendar=*-*~01 04:00:00 + Persistent=true + + [Install] + WantedBy=timers.target owner: root group: root mode: '0644' - - name: Enable and start the monthly scrub timer + - name: Reload systemd daemon + systemd: + daemon_reload: yes + + - name: Enable and start ZFS health monitoring timer + systemd: + name: "{{ zfs_systemd_health_service_name }}.timer" + enabled: yes + state: started + + - name: Enable and start ZFS monthly scrub timer systemd: name: "{{ zfs_systemd_scrub_service_name }}.timer" enabled: yes state: started - daemon_reload: yes - - name: Report the scrub schedule + - name: Test ZFS health monitoring script + command: "{{ zfs_monitoring_script_path }}" + register: script_test + changed_when: false + + - name: Verify script execution + assert: + that: + - script_test.rc == 0 + fail_msg: "ZFS health monitoring script failed - check pool health" + + - name: Display monitoring configuration debug: - msg: >- - Monthly scrub of {{ zfs_pool_name }}: - last day of each month at 04:00. - Health is reported separately by the zfs-health check - (infra/400_host_monitoring.yml). + msg: | + ✓ ZFS Pool Health Monitoring deployed successfully! + + Monitor Name: {{ monitor_friendly_name }} + Monitor Group: {{ uptime_kuma_monitor_group }} + Pool Name: {{ zfs_pool_name }} + + Health Check: + - Frequency: Every {{ zfs_check_interval_seconds }} seconds (24 hours) + - Timeout: {{ zfs_check_timeout_seconds }} seconds (~25 hours) + - Script: {{ zfs_monitoring_script_path }} + - Log: {{ zfs_log_file }} + - Service: {{ zfs_systemd_health_service_name }}.service + - Timer: {{ zfs_systemd_health_service_name }}.timer + + Monthly Scrub: + - Schedule: Last day of month at 4:00 AM + - Service: {{ zfs_systemd_scrub_service_name }}.service + - Timer: {{ zfs_systemd_scrub_service_name }}.timer + + Conditions monitored: + - Pool state (must be ONLINE) + - Device states (no DEGRADED/FAULTED/OFFLINE/UNAVAIL) + - Resilver status (alerts if resilvering) + - Read/Write/Checksum errors + - Scrub errors + + - name: Clean up temporary Uptime Kuma setup script + file: + path: /tmp/setup_uptime_kuma_zfs_monitor.py + state: absent + delegate_to: localhost + become: no diff --git a/ansible/infra/nodito/33_proxmox_debian_cloud_template.yml b/ansible/infra/nodito/33_proxmox_debian_cloud_template.yml index faf93a5..e8f8332 100644 --- a/ansible/infra/nodito/33_proxmox_debian_cloud_template.yml +++ b/ansible/infra/nodito/33_proxmox_debian_cloud_template.yml @@ -1,6 +1,9 @@ - name: Create Proxmox template from Debian cloud image (no VM clone) - hosts: hypervisor + hosts: nodito_host become: true + vars_files: + - ../../infra_vars.yml + - nodito_vars.yml vars: # Defaults (override via vars_files or --extra-vars as needed) diff --git a/ansible/infra/nodito/34_nut_ups_setup_playbook.yml b/ansible/infra/nodito/34_nut_ups_setup_playbook.yml index 12fe720..02468d5 100644 --- a/ansible/infra/nodito/34_nut_ups_setup_playbook.yml +++ b/ansible/infra/nodito/34_nut_ups_setup_playbook.yml @@ -1,33 +1,12 @@ - name: Setup NUT (Network UPS Tools) for CyberPower UPS - hosts: hypervisor + hosts: nodito_host become: true + vars_files: + - ../../infra_vars.yml + - nodito_vars.yml + - nodito_secrets.yml tasks: - # ------------------------------------------------------------------ - # Safety catch - # - # /etc/nut/upsd.users and /etc/nut/upsmon.conf on nodito were written by - # hand in January 2026 and carry a working password. host_vars/nodito/vault.yml - # (formerly infra/nodito/nodito_secrets.yml) still holds the literal string - # CHANGE_ME_TO_SECURE_PASSWORD, so running this play would overwrite that - # working pair with a placeholder and restart NUT - leaving the hypervisor's - # UPS unmonitored and unable to trigger a clean shutdown on mains loss. - # - # Until the real password is put in the vault, stop here. - # ansible-vault edit host_vars/nodito/vault.yml - # ------------------------------------------------------------------ - - name: Refuse to run with a placeholder UPS password - assert: - that: - - ups_password is defined - - ups_password | length > 0 - - ups_password != "CHANGE_ME_TO_SECURE_PASSWORD" - fail_msg: >- - ups_password is unset or still the placeholder. Applying this play would - overwrite the working /etc/nut/upsd.users and /etc/nut/upsmon.conf on - nodito and restart NUT. Put the real password in the vault first: - ansible-vault edit host_vars/nodito/vault.yml - # ------------------------------------------------------------------ # Installation # ------------------------------------------------------------------ @@ -96,45 +75,90 @@ # Configuration files # ------------------------------------------------------------------ - name: Configure NUT mode (standalone) - template: + copy: dest: /etc/nut/nut.conf - src: templates/nut.conf.j2 + content: | + # Managed by Ansible + MODE=standalone owner: root group: nut mode: "0640" notify: Restart NUT services - name: Configure UPS device - template: + copy: dest: /etc/nut/ups.conf - src: templates/ups.conf.j2 + content: | + # Managed by Ansible + [{{ ups_name }}] + driver = {{ ups_driver }} + port = {{ ups_port }} + desc = "{{ ups_desc }}" + offdelay = {{ ups_offdelay }} + ondelay = {{ ups_ondelay }} owner: root group: nut mode: "0640" notify: Restart NUT services - name: Configure upsd to listen on localhost - template: + copy: dest: /etc/nut/upsd.conf - src: templates/upsd.conf.j2 + content: | + # Managed by Ansible + LISTEN 127.0.0.1 3493 owner: root group: nut mode: "0640" notify: Restart NUT services - name: Configure upsd users - template: + copy: dest: /etc/nut/upsd.users - src: templates/upsd.users.j2 + content: | + # Managed by Ansible + [{{ ups_user }}] + password = {{ ups_password }} + upsmon master owner: root group: nut mode: "0640" notify: Restart NUT services - name: Configure upsmon - template: + copy: dest: /etc/nut/upsmon.conf - src: templates/upsmon.conf.j2 + content: | + # Managed by Ansible + MONITOR {{ ups_name }}@localhost 1 {{ ups_user }} {{ ups_password }} master + + MINSUPPLIES 1 + SHUTDOWNCMD "/sbin/shutdown -h +0" + POLLFREQ 5 + POLLFREQALERT 5 + HOSTSYNC 15 + DEADTIME 15 + POWERDOWNFLAG /etc/killpower + + # Notifications + NOTIFYMSG ONLINE "UPS %s on line power" + NOTIFYMSG ONBATT "UPS %s on battery" + NOTIFYMSG LOWBATT "UPS %s battery is low" + NOTIFYMSG FSD "UPS %s: forced shutdown in progress" + NOTIFYMSG COMMOK "Communications with UPS %s established" + NOTIFYMSG COMMBAD "Communications with UPS %s lost" + NOTIFYMSG SHUTDOWN "Auto logout and shutdown proceeding" + NOTIFYMSG REPLBATT "UPS %s battery needs replacing" + + # Log all events to syslog + NOTIFYFLAG ONLINE SYSLOG + NOTIFYFLAG ONBATT SYSLOG + NOTIFYFLAG LOWBATT SYSLOG + NOTIFYFLAG FSD SYSLOG + NOTIFYFLAG COMMOK SYSLOG + NOTIFYFLAG COMMBAD SYSLOG + NOTIFYFLAG SHUTDOWN SYSLOG + NOTIFYFLAG REPLBATT SYSLOG owner: root group: nut mode: "0640" @@ -225,11 +249,321 @@ - nut-server - nut-monitor -# The UPS heartbeat play that used to live here is gone. What it deployed - -# /opt/ups-monitoring plus a ups-heartbeat timer - is now the ups-status check -# in infra/400_host_monitoring.yml, which reports to Gatus like every other -# host check instead of carrying its own push plumbing. -# -# This playbook is now purely NUT setup: the driver, upsd, upsmon and the -# shutdown behaviour. Monitoring whether the UPS is on mains is a separate -# concern and belongs with the other host checks. + +- name: Setup UPS Heartbeat Monitoring with Uptime Kuma + hosts: nodito + become: true + vars_files: + - ../../infra_vars.yml + - ../../services_config.yml + - ../../infra_secrets.yml + - nodito_vars.yml + - nodito_secrets.yml + + vars: + ups_heartbeat_interval_seconds: 60 + ups_heartbeat_timeout_seconds: 120 + ups_heartbeat_retries: 1 + ups_monitoring_script_dir: /opt/ups-monitoring + ups_monitoring_script_path: "{{ ups_monitoring_script_dir }}/ups_heartbeat.sh" + ups_log_file: "{{ ups_monitoring_script_dir }}/ups_heartbeat.log" + ups_systemd_service_name: ups-heartbeat + uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}" + ntfy_topic: "{{ service_settings.ntfy.topic }}" + + tasks: + - name: Validate Uptime Kuma configuration + assert: + that: + - uptime_kuma_api_url is defined + - uptime_kuma_api_url != "" + - uptime_kuma_username is defined + - uptime_kuma_username != "" + - uptime_kuma_password is defined + - uptime_kuma_password != "" + fail_msg: "uptime_kuma_api_url, uptime_kuma_username and uptime_kuma_password must be set" + + - name: Get hostname for monitor identification + command: hostname + register: host_name + changed_when: false + + - name: Set monitor name and group based on hostname + set_fact: + monitor_name: "ups-{{ host_name.stdout }}" + monitor_friendly_name: "UPS Status: {{ host_name.stdout }}" + uptime_kuma_monitor_group: "{{ host_name.stdout }} - infra" + + - name: Create Uptime Kuma UPS monitor setup script + copy: + dest: /tmp/setup_uptime_kuma_ups_monitor.py + content: | + #!/usr/bin/env python3 + import sys + import json + from uptime_kuma_api import UptimeKumaApi + + def main(): + api_url = sys.argv[1] + username = sys.argv[2] + password = sys.argv[3] + group_name = sys.argv[4] + monitor_name = sys.argv[5] + monitor_description = sys.argv[6] + interval = int(sys.argv[7]) + retries = int(sys.argv[8]) + ntfy_topic = sys.argv[9] if len(sys.argv) > 9 else "alerts" + + api = UptimeKumaApi(api_url, timeout=120, wait_events=2.0) + api.login(username, password) + + monitors = api.get_monitors() + notifications = api.get_notifications() + + ntfy_notification = next((n for n in notifications if n.get('name') == f'ntfy ({ntfy_topic})'), None) + notification_id_list = {} + if ntfy_notification: + notification_id_list[ntfy_notification['id']] = True + + group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None) + if not group: + api.add_monitor(type='group', name=group_name) + monitors = api.get_monitors() + group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None) + + existing_monitor = next((m for m in monitors if m.get('name') == monitor_name), None) + + monitor_data = { + 'type': 'push', + 'name': monitor_name, + 'parent': group['id'], + 'interval': interval, + 'upsideDown': False, # Normal heartbeat mode: receiving pings = healthy + 'maxretries': retries, + 'description': monitor_description, + 'notificationIDList': notification_id_list + } + + if existing_monitor: + api.edit_monitor(existing_monitor['id'], **monitor_data) + monitors = api.get_monitors() + monitor = next((m for m in monitors if m.get('name') == monitor_name), None) + else: + api.add_monitor(**monitor_data) + monitors = api.get_monitors() + monitor = next((m for m in monitors if m.get('name') == monitor_name), None) + + result = { + 'monitor_id': monitor['id'], + 'push_token': monitor['pushToken'], + 'group_name': group_name, + 'group_id': group['id'], + 'monitor_name': monitor_name + } + print(json.dumps(result)) + + api.disconnect() + + if __name__ == '__main__': + main() + mode: '0755' + delegate_to: localhost + become: no + + - name: Run Uptime Kuma UPS monitor setup script + command: > + {{ ansible_playbook_python }} + /tmp/setup_uptime_kuma_ups_monitor.py + "{{ uptime_kuma_api_url }}" + "{{ uptime_kuma_username }}" + "{{ uptime_kuma_password }}" + "{{ uptime_kuma_monitor_group }}" + "{{ monitor_name }}" + "{{ monitor_friendly_name }} - Alerts when UPS goes on battery or loses communication" + "{{ ups_heartbeat_timeout_seconds }}" + "{{ ups_heartbeat_retries }}" + "{{ ntfy_topic }}" + register: monitor_setup_result + delegate_to: localhost + become: no + changed_when: false + + - name: Parse monitor setup result + set_fact: + monitor_info_parsed: "{{ monitor_setup_result.stdout | from_json }}" + + - name: Set push URL as fact + set_fact: + uptime_kuma_ups_push_url: "{{ uptime_kuma_api_url }}/api/push/{{ monitor_info_parsed.push_token }}" + + - name: Install required packages for UPS monitoring + package: + name: + - curl + state: present + + - name: Create monitoring script directory + file: + path: "{{ ups_monitoring_script_dir }}" + state: directory + owner: root + group: root + mode: '0755' + + - name: Create UPS heartbeat monitoring script + copy: + dest: "{{ ups_monitoring_script_path }}" + content: | + #!/bin/bash + + # UPS Heartbeat Monitoring Script + # Sends heartbeat to Uptime Kuma only when UPS is on mains power + # When on battery or communication lost, no heartbeat is sent (triggers timeout alert) + + LOG_FILE="{{ ups_log_file }}" + UPTIME_KUMA_URL="{{ uptime_kuma_ups_push_url }}" + UPS_NAME="{{ ups_name }}" + + log_message() { + echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE" + } + + send_heartbeat() { + local message="$1" + + local encoded_message + encoded_message=$(printf '%s\n' "$message" | sed 's/ /%20/g; s/(/%28/g; s/)/%29/g; s/:/%3A/g; s/\//%2F/g; s/%/%25/g') + + local response http_code + response=$(curl -s -w "\n%{http_code}" "$UPTIME_KUMA_URL?status=up&msg=$encoded_message" 2>&1) + http_code=$(echo "$response" | tail -n1) + + if [ "$http_code" = "200" ] || [ "$http_code" = "201" ]; then + log_message "Heartbeat sent: $message (HTTP $http_code)" + return 0 + else + log_message "ERROR: Failed to send heartbeat (HTTP $http_code)" + return 1 + fi + } + + main() { + local status charge runtime load + + status=$(upsc ${UPS_NAME}@localhost ups.status 2>/dev/null) + + if [ -z "$status" ]; then + log_message "ERROR: Cannot communicate with UPS - NOT sending heartbeat" + exit 1 + fi + + charge=$(upsc ${UPS_NAME}@localhost battery.charge 2>/dev/null) + runtime=$(upsc ${UPS_NAME}@localhost battery.runtime 2>/dev/null) + load=$(upsc ${UPS_NAME}@localhost ups.load 2>/dev/null) + + if [[ "$status" == *"OL"* ]]; then + local message="UPS on mains (charge=${charge}% runtime=${runtime}s load=${load}%)" + send_heartbeat "$message" + exit 0 + else + log_message "UPS not on mains power (status=$status) - NOT sending heartbeat" + exit 1 + fi + } + + main + owner: root + group: root + mode: '0755' + + - name: Create systemd service for UPS heartbeat + copy: + dest: "/etc/systemd/system/{{ ups_systemd_service_name }}.service" + content: | + [Unit] + Description=UPS Heartbeat Monitor + After=network.target nut-monitor.service + + [Service] + Type=oneshot + ExecStart={{ ups_monitoring_script_path }} + User=root + StandardOutput=journal + StandardError=journal + + [Install] + WantedBy=multi-user.target + owner: root + group: root + mode: '0644' + + - name: Create systemd timer for UPS heartbeat + copy: + dest: "/etc/systemd/system/{{ ups_systemd_service_name }}.timer" + content: | + [Unit] + Description=Run UPS Heartbeat Monitor every {{ ups_heartbeat_interval_seconds }} seconds + Requires={{ ups_systemd_service_name }}.service + + [Timer] + OnBootSec=1min + OnUnitActiveSec={{ ups_heartbeat_interval_seconds }}sec + Persistent=true + + [Install] + WantedBy=timers.target + owner: root + group: root + mode: '0644' + + - name: Reload systemd daemon + systemd: + daemon_reload: yes + + - name: Enable and start UPS heartbeat timer + systemd: + name: "{{ ups_systemd_service_name }}.timer" + enabled: yes + state: started + + - name: Test UPS heartbeat script + command: "{{ ups_monitoring_script_path }}" + register: script_test + changed_when: false + + - name: Verify script execution + assert: + that: + - script_test.rc == 0 + fail_msg: "UPS heartbeat script failed - check UPS status and communication" + + - name: Display monitoring configuration + debug: + msg: + - "UPS Monitoring configured successfully" + - "" + - "NUT Configuration:" + - " UPS Name: {{ ups_name }}" + - " UPS Description: {{ ups_desc }}" + - " Off Delay: {{ ups_offdelay }}s (time after shutdown before UPS cuts power)" + - " On Delay: {{ ups_ondelay }}s (time after mains returns before UPS restores power)" + - "" + - "Uptime Kuma Monitoring:" + - " Monitor Name: {{ monitor_friendly_name }}" + - " Monitor Group: {{ uptime_kuma_monitor_group }}" + - " Push URL: {{ uptime_kuma_ups_push_url }}" + - " Heartbeat Interval: {{ ups_heartbeat_interval_seconds }}s" + - " Timeout: {{ ups_heartbeat_timeout_seconds }}s" + - "" + - "Scripts and Services:" + - " Script: {{ ups_monitoring_script_path }}" + - " Log: {{ ups_log_file }}" + - " Service: {{ ups_systemd_service_name }}.service" + - " Timer: {{ ups_systemd_service_name }}.timer" + + - name: Clean up temporary Uptime Kuma setup script + file: + path: /tmp/setup_uptime_kuma_ups_monitor.py + state: absent + delegate_to: localhost + become: no diff --git a/ansible/host_vars/nodito/main.yml b/ansible/infra/nodito/nodito_vars.yml similarity index 52% rename from ansible/host_vars/nodito/main.yml rename to ansible/infra/nodito/nodito_vars.yml index c8176af..c0002f3 100644 --- a/ansible/host_vars/nodito/main.yml +++ b/ansible/infra/nodito/nodito_vars.yml @@ -14,12 +14,7 @@ systemd_service_name: nodito-cpu-temp-monitor # ZFS Pool Configuration zfs_pool_name: "proxmox-tank-1" -# Corrected 2026-09-13: this said WX11TN0Z, a disk that is no longer in the -# machine. The live mirror is WX120LHQ + WX11TN2P - a leg was evidently -# replaced and the repo never caught up. Pool creation is guarded by -# `when: zfs_pool_exists.rc != 0` so it was inert, but it would have been -# wrong on any disaster-recovery run. -zfs_disk_1: "/dev/disk/by-id/ata-ST4000NT001-3M2101_WX120LHQ" # First disk for RAID 1 mirror +zfs_disk_1: "/dev/disk/by-id/ata-ST4000NT001-3M2101_WX11TN0Z" # First disk for RAID 1 mirror zfs_disk_2: "/dev/disk/by-id/ata-ST4000NT001-3M2101_WX11TN2P" # Second disk for RAID 1 mirror zfs_pool_mountpoint: "/var/lib/vz" @@ -31,15 +26,3 @@ ups_port: auto ups_user: counterweight ups_offdelay: 120 # Seconds after shutdown before UPS cuts outlet power ups_ondelay: 30 # Seconds after mains returns before UPS restores outlet power - -# Systemd services deployed on this host, monitored every 5 minutes. -# -# The fact lives with the machine rather than in a central map, for the same -# reason the cross-host ports do: "what runs here" is a property of the host, -# and a central list is one more thing to forget to update when a service moves. -# -# Only units WE deploy belong here. Distro units (ssh, cron) have their own -# supervision and would be noise. -monitored_services: - - nut-server - - nut-monitor diff --git a/ansible/infra/nodito/templates/nut.conf.j2 b/ansible/infra/nodito/templates/nut.conf.j2 deleted file mode 100644 index 1f8a72f..0000000 --- a/ansible/infra/nodito/templates/nut.conf.j2 +++ /dev/null @@ -1,2 +0,0 @@ -# Managed by Ansible -MODE=standalone diff --git a/ansible/infra/nodito/templates/ups.conf.j2 b/ansible/infra/nodito/templates/ups.conf.j2 deleted file mode 100644 index 0fa0a60..0000000 --- a/ansible/infra/nodito/templates/ups.conf.j2 +++ /dev/null @@ -1,9 +0,0 @@ -# Managed by Ansible -maxretry = 3 - -[{{ ups_name }}] - driver = {{ ups_driver }} - port = {{ ups_port }} - desc = "{{ ups_desc }}" - offdelay = {{ ups_offdelay }} - ondelay = {{ ups_ondelay }} diff --git a/ansible/infra/nodito/templates/upsd.conf.j2 b/ansible/infra/nodito/templates/upsd.conf.j2 deleted file mode 100644 index baddf21..0000000 --- a/ansible/infra/nodito/templates/upsd.conf.j2 +++ /dev/null @@ -1,2 +0,0 @@ -# Managed by Ansible -LISTEN 127.0.0.1 3493 diff --git a/ansible/infra/nodito/templates/upsd.users.j2 b/ansible/infra/nodito/templates/upsd.users.j2 deleted file mode 100644 index 8d43784..0000000 --- a/ansible/infra/nodito/templates/upsd.users.j2 +++ /dev/null @@ -1,4 +0,0 @@ -# Managed by Ansible -[{{ ups_user }}] - password = {{ ups_password }} - upsmon master diff --git a/ansible/infra/nodito/templates/upsmon.conf.j2 b/ansible/infra/nodito/templates/upsmon.conf.j2 deleted file mode 100644 index dc301e1..0000000 --- a/ansible/infra/nodito/templates/upsmon.conf.j2 +++ /dev/null @@ -1,34 +0,0 @@ -# Managed by Ansible -MONITOR {{ ups_name }}@localhost 1 {{ ups_user }} {{ ups_password }} master - -MINSUPPLIES 1 -SHUTDOWNCMD "/sbin/shutdown -h +0" -POLLFREQ 5 -POLLFREQALERT 5 -HOSTSYNC 15 -DEADTIME 15 -POWERDOWNFLAG "/etc/killpower" -OFFDURATION 30 -RBWARNTIME 43200 -NOCOMMWARNTIME 300 -FINALDELAY 5 - -# Notifications -NOTIFYMSG ONLINE "UPS %s on line power" -NOTIFYMSG ONBATT "UPS %s on battery" -NOTIFYMSG LOWBATT "UPS %s battery is low" -NOTIFYMSG FSD "UPS %s: forced shutdown in progress" -NOTIFYMSG COMMOK "Communications with UPS %s established" -NOTIFYMSG COMMBAD "Communications with UPS %s lost" -NOTIFYMSG SHUTDOWN "Auto logout and shutdown proceeding" -NOTIFYMSG REPLBATT "UPS %s battery needs replacing" - -# Log all events to syslog -NOTIFYFLAG ONLINE SYSLOG -NOTIFYFLAG ONBATT SYSLOG -NOTIFYFLAG LOWBATT SYSLOG -NOTIFYFLAG FSD SYSLOG -NOTIFYFLAG COMMOK SYSLOG -NOTIFYFLAG COMMBAD SYSLOG -NOTIFYFLAG SHUTDOWN SYSLOG -NOTIFYFLAG REPLBATT SYSLOG diff --git a/ansible/infra/nodito/templates/zfs-monthly-scrub.service.j2 b/ansible/infra/nodito/templates/zfs-monthly-scrub.service.j2 deleted file mode 100644 index 04b3514..0000000 --- a/ansible/infra/nodito/templates/zfs-monthly-scrub.service.j2 +++ /dev/null @@ -1,13 +0,0 @@ -[Unit] -Description=ZFS Monthly Scrub for {{ zfs_pool_name }} -After=zfs.target - -[Service] -Type=oneshot -ExecStart=/sbin/zpool scrub {{ zfs_pool_name }} -User=root -StandardOutput=journal -StandardError=journal - -[Install] -WantedBy=multi-user.target diff --git a/ansible/infra/nodito/templates/zfs-monthly-scrub.timer.j2 b/ansible/infra/nodito/templates/zfs-monthly-scrub.timer.j2 deleted file mode 100644 index 36ad3d7..0000000 --- a/ansible/infra/nodito/templates/zfs-monthly-scrub.timer.j2 +++ /dev/null @@ -1,10 +0,0 @@ -[Unit] -Description=Run ZFS Scrub on last day of every month at 4:00 AM -Requires={{ zfs_systemd_scrub_service_name }}.service - -[Timer] -OnCalendar=*-*~01 04:00:00 -Persistent=true - -[Install] -WantedBy=timers.target diff --git a/ansible/infra_secrets.yml.example b/ansible/infra_secrets.yml.example new file mode 100644 index 0000000..cddc58a --- /dev/null +++ b/ansible/infra_secrets.yml.example @@ -0,0 +1,28 @@ +# Uptime Kuma login credentials +# Used by the disk monitoring playbook to create monitors automatically + +uptime_kuma_username: "admin" +uptime_kuma_password: "your_password_here" + +# ntfy credentials +# Used for notification channel setup in Uptime Kuma + +ntfy_username: "your_ntfy_username" +ntfy_password: "your_ntfy_password" + +# headscale-ui credentials +# Used for HTTP basic authentication via Caddy +# Provide either: +# - headscale_ui_password: plain text password (will be hashed automatically) +# - headscale_ui_password_hash: pre-hashed bcrypt password (more secure, use caddy hash-password to generate) + +headscale_ui_username: "admin" +headscale_ui_password: "your_secure_password_here" +# headscale_ui_password_hash: "$2a$14$..." # Optional: pre-hashed password + +bitcoin_rpc_user: "bitcoinrpc" +bitcoin_rpc_password: "CHANGE_ME_TO_SECURE_PASSWORD" + +# Mempool MariaDB credentials +# Used by: services/mempool/deploy_mempool_playbook.yml +mariadb_mempool_password: "CHANGE_ME_TO_SECURE_PASSWORD" diff --git a/ansible/infra_vars.yml b/ansible/infra_vars.yml new file mode 100644 index 0000000..952df93 --- /dev/null +++ b/ansible/infra_vars.yml @@ -0,0 +1,4 @@ +new_user: counterweight +ssh_port: 22 +allow_ssh_from: "any" +root_domain: contrapeso.xyz diff --git a/ansible/inventory.ini b/ansible/inventory.ini deleted file mode 100644 index 64d0e96..0000000 --- a/ansible/inventory.ini +++ /dev/null @@ -1,88 +0,0 @@ -[vps] -vipy ansible_host=167.172.98.204 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua -spacey ansible_host=46.101.250.26 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua - -[nodito_host] -nodito ansible_host=192.168.1.139 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua - -# Requires the tailnet to be up on the control node. -[nodito_vms] -knots_box_local ansible_host=knots-box lan_ip=192.168.1.135 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua -fulcrum_box_local ansible_host=fulcrum-box lan_ip=192.168.1.140 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua -mempool_box_local ansible_host=mempool-box lan_ip=192.168.1.142 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua -memos_box_local ansible_host=memos-box lan_ip=192.168.1.130 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua -forgejo_runner_local ansible_host=forgejo-runner-box lan_ip=192.168.1.132 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua -arbret_staging_local ansible_host=arbret-staging-box lan_ip=192.168.1.145 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua -small_backups_local ansible_host=small-backups-box lan_ip=192.168.1.131 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua -monerod_box_local ansible_host=monerod-box lan_ip=192.168.1.146 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua -# lan_ip read off nodito's ARP table for the VM's own MAC (bc:24:11:89:fc:1f), -# not guessed. Note .149 previously belonged to another machine - its stale host -# key was still in known_hosts and SSH refused the connection, which is the -# designed behaviour described in group_vars/nodito_vms.yml. -music_box_local ansible_host=music-box lan_ip=192.168.1.149 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua -# lan_ip read from the Proxmox guest agent for VM 3300 (qm guest cmd -# network-get-interfaces), and the host key checked against the one the agent -# reads off the disk before it went into known_hosts. -jar_lid_staging_local ansible_host=jar-lid-archive-staging-box lan_ip=192.168.1.151 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/counterganzua - -# Local connection to laptop: this assumes you're running ansible commands from your personal laptop -[lapy] -localhost ansible_connection=local ansible_user=counterweight - -[edge] -vipy - -# Gatus and its Signal transport. Shares spacey with headscale since the -# 2026-10-17 rebuild; the separate `monitoring` droplet is gone. -[observability] -spacey - -[vpn_control] -spacey - -[hypervisor] -nodito - -[bitcoin] -knots_box_local - -[electrum] -fulcrum_box_local - -[mempool] -mempool_box_local - -[monero] -monerod_box_local - -[memos] -memos_box_local - -[music] -music_box_local - -# Staging only for now; a production host joins this group with its own host_vars. -[jar_lid_archive] -jar_lid_staging_local - -[ci_runner] -forgejo_runner_local - -[control] -localhost - -# Every machine Ansible may configure as a server. -# Deliberately EXCLUDES [control] (your laptop). -[managed:children] -vps -nodito_host -nodito_vms - -# Hosts that run Caddy and therefore have /etc/caddy/sites-enabled. -[caddy:children] -edge -observability -vpn_control - -[backup_store] -small_backups_local diff --git a/ansible/inventory.ini.example b/ansible/inventory.ini.example new file mode 100644 index 0000000..bde96dd --- /dev/null +++ b/ansible/inventory.ini.example @@ -0,0 +1,16 @@ +[vps] +vipy ansible_host=your.services.vps.ip ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/your-key +watchtower ansible_host=your.monitoring.vps.ip ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/your-key +spacey ansible_host=your.headscale.vps.ip ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/your-key + +[nodito_host] +nodito ansible_host=your.proxmox.ip.here ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/your-key ansible_ssh_pass=your_root_password + +[nodito_vms] +# Example node, replace with your VM names and addresses +# memos_box ansible_host=192.168.1.150 ansible_user=counterweight ansible_port=22 ansible_ssh_private_key_file=~/.ssh/your-key + +# Local connection to laptop: this assumes you're running ansible commands from your personal laptop +# Make sure to adjust the username +[lapy] +localhost ansible_connection=local ansible_user=your laptop user gpg_recipient=your_email@example.com gpg_key_id=your_gpg_key_id_here \ No newline at end of file diff --git a/ansible/playbooks/backups.yml b/ansible/playbooks/backups.yml deleted file mode 100644 index 791d51d..0000000 --- a/ansible/playbooks/backups.yml +++ /dev/null @@ -1,115 +0,0 @@ -- name: Configure the offsite backup pull - hosts: backup_store - gather_facts: yes - - tasks: - - name: Ensure the box pulls every source on a timer - ansible.builtin.include_role: - name: backup_store - vars: - # check-backups.sh reports one result per source plus one for the store - # itself, so it needs the collection URL and appends each key. - backup_store_check_push_base: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints" - backup_store_check_push_token: "{{ gatus_push_tokens[inventory_hostname] }}" - backup_store_sources: - # On vipy since 2026-10-17 (prd-arbret is gone). The arbret repo writes - # the dump to the backup_source contract - see arbret_handoff.md. - - name: arbret - source: "backup-pull@prd-vipy:/opt/backups/arbret/" - retention_days: 90 - - name: headscale - source: "backup-pull@headscale.contrapeso.xyz:/opt/backups/headscale/" - retention_days: 90 - - name: memos - source: "backup-pull@memos-box:/opt/backups/memos/" - retention_days: 90 - - name: vaultwarden - source: "backup-pull@prd-vipy:/opt/backups/vaultwarden/" - retention_days: 90 - - name: lnbits - source: "backup-pull@prd-vipy:/opt/backups/lnbits/" - retention_days: 90 - - name: forgejo - source: "backup-pull@prd-vipy:/opt/backups/forgejo/" - retention_days: 14 - # Staging: disposable data, so the long tail is as short as the local one. - - name: "{{ hostvars['jar_lid_staging_local'].jar_lid_archive_instance }}" - source: "backup-pull@{{ hostvars['jar_lid_staging_local'].ansible_host }}:/opt/backups/{{ hostvars['jar_lid_staging_local'].jar_lid_archive_instance }}/" - retention_days: "{{ hostvars['jar_lid_staging_local'].jar_lid_archive_backup_retention_days }}" - -# ───────────────────────────────────────────────────────────────────────────── -# Register the backup checks with Gatus. -# -# Two groups on purpose, because they answer different questions and fail for -# different reasons: -# -# backup-dump did the SOURCE produce an artefact? Pushed by each dump right -# after it runs, so a broken dump is visible within minutes. -# backup-store did it ARRIVE, is it fresh, non-zero, plausibly sized, and is -# retention pruning? Pushed by check-backups.sh at 05:30. -# -# The store alone could catch almost everything, because the artefact filename -# carries the source's dump timestamp - a source whose timer died still pulls -# "ok" forever, but the timestamp gives it away. What the source side adds is -# LATENCY and DIAGNOSIS: the store only learns at the next 04:00 pull, and it -# cannot tell you whether the dump broke or the pull did. -# -# arbret's dump is written by the arbret repo, not by backup_source, and it -# pushes with its own `arbret` token (the `host` below is the token key). -# ───────────────────────────────────────────────────────────────────────────── -- name: Register the backup checks with Gatus - hosts: observability - become: yes - vars: - # Sources we deploy the dump for, and the host each one runs on. - dump_sources: - - {name: headscale, host: spacey} - - {name: memos, host: memos_box_local} - - {name: vaultwarden, host: vipy} - - {name: lnbits, host: vipy} - - {name: forgejo, host: vipy} - - {name: jar-lid-archive-staging, host: jar_lid_staging_local} - - {name: arbret, host: arbret} - store_sources: [arbret, headscale, memos, vaultwarden, lnbits, forgejo, jar-lid-archive-staging] - - tasks: - # 26h, not 7h: the DUMP is genuinely daily, so the window cannot be tighter - # than a day plus slack. The store-side check catches the same fault within - # 6h by reading the artefact's dump timestamp out of the filename, so this is - # the slow backstop rather than the primary signal. - - name: Build the dump endpoint list - ansible.builtin.set_fact: - dump_endpoints: "{{ dump_endpoints | default([]) + [{ - 'name': item.name, - 'group': 'backup-dump', - 'token': gatus_push_tokens[item.host], - 'heartbeat': '26h'}] }}" - loop: "{{ dump_sources }}" - - - name: Build the store endpoint list - ansible.builtin.set_fact: - store_endpoints: "{{ store_endpoints | default([]) + [{ - 'name': item, - 'group': 'backup-store', - 'token': gatus_push_tokens['small_backups_local'], - 'heartbeat': '7h'}] }}" - loop: "{{ store_sources }}" - - - name: Register the backup endpoints - ansible.builtin.include_role: - name: gatus_endpoint - vars: - # Push endpoints: the heartbeat window is the tolerance, so alert on - # the first failure rather than waiting for three 7h windows to pass. - gatus_endpoint_default_alerts: - - type: signal - failure-threshold: 1 - success-threshold: 2 - send-on-resolved: true - minimum-reminder-interval: 12h - gatus_endpoint_name: backups - gatus_endpoint_external: "{{ dump_endpoints + store_endpoints + [{ - 'name': 'pull job', - 'group': 'backup-store', - 'token': gatus_push_tokens['small_backups_local'], - 'heartbeat': '7h'}] }}" diff --git a/ansible/requirements.yml b/ansible/requirements.yml index 16ef1fd..dd7eef4 100644 --- a/ansible/requirements.yml +++ b/ansible/requirements.yml @@ -1,10 +1,11 @@ +--- # Ansible Galaxy Collections Requirements # Install with: ansible-galaxy collection install -r requirements.yml -# No collections are currently required. -# -# lucasheld.uptime_kuma was pinned here but never used — every monitor was created -# by hand-rolled Python instead. Removed 2026-09-11 along with Uptime Kuma itself. -# See archive/uptime_kuma/. +collections: + # Uptime Kuma Ansible Collection + # Used by: infra/41_disk_usage_alerts.yml + # Provides modules to manage Uptime Kuma monitors programmatically + - name: lucasheld.uptime_kuma + version: ">=1.0.0" -collections: [] diff --git a/ansible/roles/backup_source/README.md b/ansible/roles/backup_source/README.md deleted file mode 100644 index a7794ab..0000000 --- a/ansible/roles/backup_source/README.md +++ /dev/null @@ -1,126 +0,0 @@ -# `backup_source` - -Makes a host back **itself** up: dump to stdout, encrypt with `age`, write to a -local directory, prune, on a systemd timer. `small-backups-box` pulls the -directory later (see `backup_store`). - -Modelled on `prd-arbret`, which has been doing exactly this correctly since -before the rest of the estate was migrated. - -## Usage - -```yaml -- ansible.builtin.include_role: - name: backup_source - vars: - backup_source_name: headscale - backup_source_description: "Headscale" - backup_source_dump_command: "tar -czf - -C / var/lib/headscale etc/headscale" - backup_source_stop_service: headscale - backup_source_retention_days: 7 -``` - -Produces `/opt/backups/headscale/headscale_.tar.gz.age`, -`headscale-backup.{service,timer}`, and `/usr/local/bin/headscale-backup.sh`. - -## Why the source encrypts, not the destination - -`age -r ` is asymmetric and the host holds only the **public** key, so -a compromised host cannot read its own backups — or anyone else's. The scripts -this replaces encrypted with GPG *on the laptop, after the data had already -crossed the network*, which protects the artefact at rest but not in transit. - -The matching identity lives only on lapy and is escrowed. **Lose it and every -artefact everywhere becomes noise**, including arbret's. - -## `backup_source_dump_command` writes to STDOUT - -The role pipes it into `age`, so plaintext never touches the disk. Use `-C /` -with relative paths in `tar` rather than absolute ones: it avoids tar's "removing -leading /" and makes the restore target explicit. - -## Services that are not systemd - -`backup_source_stop_service` runs `systemctl stop/start`. For anything else, -give the pair explicitly — vaultwarden is a docker compose stack, so -`systemctl stop vaultwarden` silently does nothing: - -```yaml -backup_source_stop_command: "docker compose -f /opt/vaultwarden/docker-compose.yml stop" -backup_source_start_command: "docker compose -f /opt/vaultwarden/docker-compose.yml start" -``` - -The same EXIT trap wraps both forms. The assert refuses a stop command without a -matching start command, because that combination fails in the one way you would -not notice: the service stops and never comes back. - -## More than one thing to back up - -`tar` takes several paths, so multiple files or directories are normally **one** -artefact — headscale captures `/var/lib/headscale` and `/etc/headscale` together, -lnbits captures its data directory and its `.env`. - -Prefer one artefact. A backup should be a consistent snapshot, and two artefacts -written by two runs can drift — you can end up restoring an `.env` that does not -match the database it configures. Pulling a single file back out needs no -unpacking: - -```bash -age -d -i | tar -xzO opt/lnbits/lnbits/.env -``` - -If you genuinely need separate artefacts, call the role twice with different -`backup_source_name`s rather than extending it — but only one call may set -`backup_source_stop_service`, or the service is stopped twice per night. - -The case this shape cannot express is a **database dump plus a file tree** -(`pg_dump` and a media directory, say): you cannot merge those into one stream -without staging plaintext on disk, which is exactly what this design avoids. -None of the current services need it — all are file trees, all stopped for the -dump. A future one that does should use two role calls. - -## Everything here is sqlite, so everything stops - -All five services are sqlite-backed, several in WAL mode (`-wal`/`-shm` files -present). A live copy of a WAL-mode database can be torn or stale, so each is -stopped for the duration. Measured downtime: under a second for headscale and -memos, ~6 s vaultwarden, ~11 s lnbits, and **2m36s for forgejo** — 2.7 G of repos -and database. That last one is the real cost of a consistent snapshot; if it -becomes unacceptable the answer is `sqlite3 .backup` plus an online repo copy, -not skipping the stop. - -## The trap is the reason this role exists - -When `backup_source_stop_service` is set, the script stops the unit and installs -an EXIT trap that starts it again. Without it, a failed dump leaves the service -down until the next timer fires — **every hand-written script this replaced had -that bug**, and it was only ever masked because their `systemctl stop` failed -first, before anything was stopped. - -Verified on spacey: with the dump forced to fail, the log shows -`Stopping → Writing → Restarting`, the script exits 1 (so systemd marks the unit -failed rather than hiding it), and headscale is `active` afterwards. - -If `systemctl stop` itself fails, `set -e` exits *before* the trap is installed — -which is correct, because nothing was stopped. - -## `.partial` - -The dump writes `.partial` and only `mv`s it into place on success, so -a truncated file is never mistaken for a backup. A failure inside the pipeline -does leave one behind, and the prune glob cannot match it (it ends `.partial`, -not `.tar.gz.age`), so the script clears stale partials at the **start** of each -run. Tested by failing mid-pipeline: 1 partial left, 0 after the next run. - -## `backup_source_stop_service` may be a bare name - -`headscale` and `headscale.service` both work. The unit template normalises it, -because systemd rejects a bare name in `After=` with -`Failed to add dependency ... Invalid argument` — which it logs and then ignores, -so the unit appears to work while carrying no ordering at all. - -## Retention is two-tier - -`backup_source_retention_days` is **local** and short — these hosts are -disk-constrained. The long tail lives on `small-backups-box`, which decides its -own retention per source. Losing the local copy is expected and fine. diff --git a/ansible/roles/backup_source/defaults/main.yml b/ansible/roles/backup_source/defaults/main.yml deleted file mode 100644 index b195190..0000000 --- a/ansible/roles/backup_source/defaults/main.yml +++ /dev/null @@ -1,44 +0,0 @@ ---- -# Required -backup_source_name: "" # "headscale" -> headscale_.tar.gz.age -backup_source_description: "" # "Headscale" -backup_source_dump_command: "" # must write the payload to STDOUT - -# Placement -backup_source_dir: "/opt/backups/{{ backup_source_name }}" -backup_source_artifact_suffix: "tar.gz.age" - -# Encryption. Asymmetric: the host holds only the public key and cannot decrypt -# what it produces. -backup_source_recipient: "{{ age_backup_recipient }}" - -# The unprivileged account small-backups-box pulls as. It owns the dump -# directory and nothing else; it deliberately has no sudo. -backup_source_pull_user: backup-pull -backup_source_pull_key: "{{ backup_pull_public_key }}" - -# Safety. Give either a systemd unit, or an explicit pair of commands for -# services that are not systemd-managed (vaultwarden is a docker compose stack). -# Whichever is used, a trap guarantees the restart. -backup_source_stop_service: "" # systemd unit stopped for the dump -backup_source_stop_command: "" # overrides stop_service when set -backup_source_start_command: "" # required alongside stop_command - -# Retention here is LOCAL and short; small-backups-box keeps the long tail. -backup_source_retention_days: 7 - -# Schedule. The box pulls at 04:00, so dumps must land before that. -backup_source_on_calendar: "*-*-* 02:00:00" - -# ── Reporting ──────────────────────────────────────────────────────────────── -# Where to report that this dump ran and produced a plausible artefact. -# Gatus external endpoint: -# POST {url}?success=true|false&error=... -# Authorization: Bearer {token} -# Empty is valid and is not an error: the unit's exit code is still the answer, -# and the STORE will independently notice a stale dump within ~26h because the -# artefact filename carries this dump's timestamp. Reporting here only buys -# earlier detection and tells you it was the DUMP that broke rather than the -# pull. -backup_source_push_url: "" -backup_source_push_token: "" diff --git a/ansible/roles/backup_source/handlers/main.yml b/ansible/roles/backup_source/handlers/main.yml deleted file mode 100644 index 37a7f2b..0000000 --- a/ansible/roles/backup_source/handlers/main.yml +++ /dev/null @@ -1,4 +0,0 @@ ---- -- name: Reload systemd for backup units - ansible.builtin.systemd: - daemon_reload: yes diff --git a/ansible/roles/backup_source/tasks/main.yml b/ansible/roles/backup_source/tasks/main.yml deleted file mode 100644 index a964c5d..0000000 --- a/ansible/roles/backup_source/tasks/main.yml +++ /dev/null @@ -1,110 +0,0 @@ ---- -- name: Assert backup_source parameters are sane - ansible.builtin.assert: - that: - - backup_source_name | length > 0 - - backup_source_description | length > 0 - - backup_source_dump_command | length > 0 - - backup_source_recipient | length > 0 - - backup_source_recipient is match('^age1[0-9a-z]{58}$') - - not (backup_source_stop_command | length > 0 and backup_source_start_command | length == 0) - fail_msg: >- - backup_source: '{{ backup_source_name | default("") }}' needs a name, - description, dump command and a valid age recipient (age1... 62 chars). - backup_source_stop_command must be paired with backup_source_start_command. - quiet: true - -# Declared here rather than assumed. Stage 1 installed it by hand; this is what -# makes a rebuilt host get it too. -# Cache refresh is best-effort on purpose. An unrelated third-party repo with a -# bad signing key (spacey had two: an expired Caddy subkey and a SHA1 nodesource -# key) makes `apt-get update` return warnings, which the apt module treats as a -# hard failure — and that must not stop backups being configured. Installing the -# package is NOT best-effort: if age is genuinely unavailable, the next task fails. -- name: Refresh the apt cache (best effort) - ansible.builtin.apt: - update_cache: yes - cache_valid_time: 3600 - failed_when: false - changed_when: false - -- name: Ensure age is installed - ansible.builtin.apt: - name: - - age - # curl is needed only when backup_source_push_url is set, but installing it - # unconditionally keeps the task idempotent and it is present on every - # Debian host here anyway. - - curl - state: present - -# The pull account: unprivileged, no sudo, exists only so small-backups-box can -# read the dump directory. Trust points one way — the box can read backups, and -# can do nothing else on this host. -- name: "Ensure the {{ backup_source_pull_user }} account exists" - ansible.builtin.user: - name: "{{ backup_source_pull_user }}" - system: yes - shell: /bin/sh # rsync-over-ssh needs a shell; nologin breaks it - home: "/var/lib/{{ backup_source_pull_user }}" - create_home: yes - password: '!' # no password login, ever - when: backup_source_pull_user | length > 0 - -- name: "Authorise the backup box's key for {{ backup_source_pull_user }}" - ansible.posix.authorized_key: - user: "{{ backup_source_pull_user }}" - key: "{{ backup_source_pull_key }}" - key_options: "restrict" # no pty, no forwarding, no user rc - exclusive: yes - state: present - when: backup_source_pull_user | length > 0 - -# The shared container above the per-service directories. It must be traversable -# or the pull account cannot reach its own directory. The script's `mkdir -p` -# runs under `umask 077` and would otherwise create this 0700. -- name: "Ensure {{ backup_source_dir | dirname }} is traversable" - ansible.builtin.file: - path: "{{ backup_source_dir | dirname }}" - state: directory - owner: root - group: root - mode: '0755' - -- name: "Ensure {{ backup_source_dir }} exists" - ansible.builtin.file: - path: "{{ backup_source_dir }}" - state: directory - owner: root - group: "{{ backup_source_pull_user | default('root', true) }}" - mode: '0750' - -- name: "Install the {{ backup_source_name }} backup script" - ansible.builtin.template: - src: backup.sh.j2 - dest: "/usr/local/bin/{{ backup_source_name }}-backup.sh" - owner: root - group: root - mode: '0750' - validate: "bash -n %s" - -# The .service carries the push token in an Environment= line, so it is 0600. -# The .timer holds nothing secret and stays world-readable. -- name: "Install the {{ backup_source_name }}-backup systemd units" - ansible.builtin.template: - src: "backup.{{ item.unit }}.j2" - dest: "/etc/systemd/system/{{ backup_source_name }}-backup.{{ item.unit }}" - owner: root - group: root - mode: "{{ item.mode }}" - loop: - - {unit: service, mode: "0600"} - - {unit: timer, mode: "0644"} - notify: Reload systemd for backup units - -- name: "Enable the {{ backup_source_name }}-backup timer" - ansible.builtin.systemd: - name: "{{ backup_source_name }}-backup.timer" - enabled: yes - state: started - daemon_reload: yes diff --git a/ansible/roles/backup_source/templates/backup.service.j2 b/ansible/roles/backup_source/templates/backup.service.j2 deleted file mode 100644 index 7a1df21..0000000 --- a/ansible/roles/backup_source/templates/backup.service.j2 +++ /dev/null @@ -1,16 +0,0 @@ -[Unit] -Description={{ backup_source_description }} backup -{% if backup_source_stop_service %} -{# systemd rejects a bare name here ("Failed to add dependency ... Invalid - argument"), so normalise to a full unit name. #} -After={{ backup_source_stop_service if '.' in backup_source_stop_service else backup_source_stop_service ~ '.service' }} -{% endif %} - -[Service] -Type=oneshot -ExecStart=/usr/local/bin/{{ backup_source_name }}-backup.sh -Environment=BACKUP_PUSH_URL={{ backup_source_push_url }} -Environment=BACKUP_PUSH_TOKEN={{ backup_source_push_token }} -StandardOutput=journal -StandardError=journal -SyslogIdentifier={{ backup_source_name }}-backup diff --git a/ansible/roles/backup_source/templates/backup.sh.j2 b/ansible/roles/backup_source/templates/backup.sh.j2 deleted file mode 100644 index 018c313..0000000 --- a/ansible/roles/backup_source/templates/backup.sh.j2 +++ /dev/null @@ -1,129 +0,0 @@ -#!/usr/bin/env bash -# {{ backup_source_description }} backup — managed by Ansible (roles/backup_source) -# -# Dumps to stdout, encrypts with age, writes {{ backup_source_dir }}. -# The host holds only the age PUBLIC key, so it cannot read its own backups. -set -euo pipefail -umask 077 - -BACKUP_DIR="{{ backup_source_dir }}" -RETENTION_DAYS={{ backup_source_retention_days }} -RECIPIENT="{{ backup_source_recipient }}" -SUFFIX="{{ backup_source_artifact_suffix }}" -NAME="{{ backup_source_name }}" -{% if backup_source_stop_service or backup_source_stop_command %} -STOP_CMD={{ (backup_source_stop_command or ('systemctl stop ' ~ backup_source_stop_service)) | quote }} -START_CMD={{ (backup_source_start_command or ('systemctl start ' ~ backup_source_stop_service)) | quote }} -SERVICE="{{ backup_source_stop_service or backup_source_description }}" # label for the log only -{% endif %} - -TIMESTAMP=$(date +%Y%m%d_%H%M%S) -ARTIFACT="${BACKUP_DIR}/${NAME}_${TIMESTAMP}.${SUFFIX}" - -die() { echo "FATAL: $*" >&2; exit 1; } -log() { echo "$(date '+%Y-%m-%d %H:%M:%S') $*"; } - -# --- Pre-flight --- -[[ -n "$RECIPIENT" ]] || die "no age recipient configured" -command -v age >/dev/null || die "age is not installed" - -# Mode must agree with what the role sets, or each undoes the other every run. -mkdir -p "$BACKUP_DIR" -{% if backup_source_pull_user %} -chown root:{{ backup_source_pull_user }} "$BACKUP_DIR" -chmod 750 "$BACKUP_DIR" -{% else %} -chmod 700 "$BACKUP_DIR" -{% endif %} - -# A run that died mid-dump leaves a .partial. It is not a backup, and the prune -# glob below cannot match it (it ends .partial, not .${SUFFIX}), so clear them -# here or they accumulate forever. -rm -f "${BACKUP_DIR}/${NAME}_"*.partial - -# --- Reporting ------------------------------------------------------------- -# A dump that exits non-zero, or that produces a zero-byte artefact, is a failed -# backup even though the script "finished". Both are reported as failures. -PUSH_URL="${BACKUP_PUSH_URL:-}" -PUSH_TOKEN="${BACKUP_PUSH_TOKEN:-}" - -report() { - local success="$1" message="$2" - [ -n "$PUSH_URL" ] || return 0 - local encoded - encoded=$(printf '%s' "$message" | sed 's/%/%25/g; s/ /%20/g; s/&/%26/g; s/+/%2B/g; s/#/%23/g') - curl -s -o /dev/null --max-time 15 --retry 2 --retry-delay 3 -X POST \ - -H "Authorization: Bearer ${PUSH_TOKEN}" \ - "${PUSH_URL}?success=${success}&error=${encoded}" 2>/dev/null || true -} - -# Reports on ANY exit path, so a dump that dies halfway still reports rather -# than going quiet. The size of the FINISHED artefact decides success, not -# merely reaching the end of the script. -# -# This is called FROM the single EXIT trap below - it must never register an -# EXIT trap of its own. `trap ... EXIT` REPLACES the existing handler rather -# than adding to it, so a second trap here silently discards the one that -# restarts the service, and a backup run leaves the service stopped. That is -# precisely the failure the restart trap exists to prevent. -report_outcome() { - local rc="$1" - if [ "$rc" -ne 0 ]; then - report "false" "${NAME} dump exited ${rc}" - elif [ ! -s "$ARTIFACT" ]; then - report "false" "${NAME} produced no artefact at ${ARTIFACT}" - else - report "true" "${NAME} $(du -h "$ARTIFACT" | cut -f1)" - fi -} - -# --- One EXIT handler, doing both jobs ------------------------------------- -# bash keeps exactly ONE EXIT trap: `trap ... EXIT` REPLACES the previous -# handler rather than adding to it. Registering a second one here would -# silently discard the service restart and leave the service stopped after -# every backup - which is the exact bug the restart exists to prevent, and it -# is invisible until someone notices the service is down. -on_exit() { - local rc=$? -{% if backup_source_stop_service or backup_source_stop_command %} - log "Restarting ${SERVICE}..." - eval "$START_CMD" || true -{% endif %} - report_outcome "$rc" -} -trap on_exit EXIT - -{% if backup_source_stop_service or backup_source_stop_command %} -# --- Stop the service; the trap above guarantees it comes back ------------- -# The trap is the point: without it a failed dump leaves the service down until -# the next timer fires. Every hand-written script this replaced had that bug. -# It is armed BEFORE the stop, so even a failure during the stop restarts. -log "Stopping ${SERVICE}..." -eval "$STOP_CMD" -{% endif %} - -# --- Dump straight into age; plaintext never touches the disk --- -log "Writing ${ARTIFACT}..." -{{ backup_source_dump_command }} | age -r "$RECIPIENT" -o "${ARTIFACT}.partial" -{% if backup_source_pull_user %} -# Match the final ownership immediately, so even a partial left by a later -# failure is not an unreadable obstacle to the pull. -chown root:{{ backup_source_pull_user }} "${ARTIFACT}.partial" -chmod 640 "${ARTIFACT}.partial" -{% endif %} -mv "${ARTIFACT}.partial" "$ARTIFACT" -{% if backup_source_pull_user %} -# Readable by the pull account and nobody else. The contents are age-encrypted -# regardless, so this is depth rather than the actual protection. -chown root:{{ backup_source_pull_user }} "$ARTIFACT" -chmod 640 "$ARTIFACT" -{% else %} -chmod 600 "$ARTIFACT" -{% endif %} -log "Wrote ${ARTIFACT} ($(du -h "$ARTIFACT" | cut -f1))" - -# --- Prune --- -log "Pruning local artefacts older than ${RETENTION_DAYS} days..." -find "$BACKUP_DIR" -maxdepth 1 -type f -name "${NAME}_*.${SUFFIX}" -mtime +"${RETENTION_DAYS}" -delete - -log "Done." diff --git a/ansible/roles/backup_source/templates/backup.timer.j2 b/ansible/roles/backup_source/templates/backup.timer.j2 deleted file mode 100644 index d9a7e6a..0000000 --- a/ansible/roles/backup_source/templates/backup.timer.j2 +++ /dev/null @@ -1,11 +0,0 @@ -[Unit] -Description={{ backup_source_description }} backup - -[Timer] -OnCalendar={{ backup_source_on_calendar }} -# Persistent: a window missed while the host was down runs on next boot. cron on -# a laptop had no equivalent, which is how two backups went unnoticed for months. -Persistent=true - -[Install] -WantedBy=timers.target diff --git a/ansible/roles/backup_store/README.md b/ansible/roles/backup_store/README.md deleted file mode 100644 index 13d9f5a..0000000 --- a/ansible/roles/backup_store/README.md +++ /dev/null @@ -1,55 +0,0 @@ -# `backup_store` - -Pulls already-encrypted backup artefacts from every source host onto -`small-backups-box`, on a timer, and expires them per source. - -Generalises the hand-written `pull-backups.sh` that had one hardcoded source -(`arbret`). That job's behaviour is preserved exactly: same source path, same -90 days, same destination directory. - -## This host holds no key - -Everything pulled here is ciphertext produced by `backup_source` on the source -host. The box cannot read any of it — the age identity lives only on lapy. That -is deliberate: the machine holding every backup should not also be able to open -them. - -## One failing source must not stop the others - -The script is `set -uo pipefail`, **not** `-e`. Each source runs in its own -function, failures are counted, and the script exits non-zero at the end so -systemd marks the unit failed. A dead host costs you that one source, not the -whole run. - -This is the specific failure the whole plan exists to prevent: the laptop jobs -aborted on first error and then silently produced empty directories for nine -months. - -## Trust points one way - -The box authenticates with `~/.ssh/id_pull` to an unprivileged, dedicated -account on each source (`backup-pull`, or `arbret` on prd-arbret), authorised -with `restrict`. That account can read one directory and do nothing else — no -sudo, no pty, no forwarding. A compromised backup box cannot reach into -production. - -## Addressing: names, never IPs - -Sources are addressed by name. The job this replaced hardcoded spacey's IP; the -droplet was later rebuilt, the address was recycled to a stranger, and the -backup failed silently from 2025-12-01 while the directory listing still looked -healthy. - -Two kinds of name are in play: - -- **Tailnet members** (vipy, memos-box, …) → MagicDNS names. These require a - headscale ACL grant from `tag:small-backups-box` to the source's `:22`; without - it the box cannot even resolve the peer, let alone reach it. -- **spacey** is *not* a tailnet member — it is the headscale control server — so - its backup is pulled over the public internet via `headscale.contrapeso.xyz`, - which follows the host if the droplet is rebuilt. - -## Retention here is the long tail - -Sources keep a few days locally; this box keeps 90 (or whatever the source entry -says). Losing the source's local copy is expected. diff --git a/ansible/roles/backup_store/defaults/main.yml b/ansible/roles/backup_store/defaults/main.yml deleted file mode 100644 index 9032d55..0000000 --- a/ansible/roles/backup_store/defaults/main.yml +++ /dev/null @@ -1,32 +0,0 @@ ---- -backup_store_dir: "{{ ansible_env.HOME }}/backups" -backup_store_ssh_key: "{{ ansible_env.HOME }}/.ssh/id_pull" -backup_store_on_calendar: "*-*-* 04:00:00" - -# One entry per source. `retention_days` is the LONG tail; the source keeps its -# own short local retention. -# - name: headscale -# source: "backup-pull@headscale.contrapeso.xyz:/opt/backups/headscale/" -# retention_days: 90 -backup_store_sources: [] - -# ── Reporting ──────────────────────────────────────────────────────────────── -# check-backups.sh reports one result PER SOURCE plus one for the store itself, -# so the base URL is the endpoints collection and the script appends each key. -# Empty is valid: the script still prints its report and exits 0/1. -backup_store_check_push_base: "" -backup_store_check_push_token: "" - -# Every six hours, offset past the 04:00 pull so the first run of the day sees a -# finished pull. The BACKUPS are daily, but this check is not - it reads the -# source's dump timestamp out of the artefact filename, so running it more often -# catches "the source stopped dumping" within hours rather than a day, and lets -# the Gatus heartbeat be 7h instead of 30h. -# 05:30 then every 6h. Note the syntax: a comma-separated list of full -# times ("05:30:00,11:30:00,...") is NOT valid and systemd rejects the unit -# with "bad unit file setting" - validated with `systemd-analyze calendar`. -backup_store_check_on_calendar: "*-*-* 05/6:30:00" - -# An artefact older than this is stale. Sources dump daily at 02:00-02:30 and the -# pull is at 04:00, so 26h tolerates exactly one missed night before alarming. -backup_store_check_max_age_hours: 26 diff --git a/ansible/roles/backup_store/handlers/main.yml b/ansible/roles/backup_store/handlers/main.yml deleted file mode 100644 index 632be35..0000000 --- a/ansible/roles/backup_store/handlers/main.yml +++ /dev/null @@ -1,5 +0,0 @@ ---- -- name: Reload systemd for pull-backups - ansible.builtin.systemd: - daemon_reload: yes - become: yes diff --git a/ansible/roles/backup_store/tasks/main.yml b/ansible/roles/backup_store/tasks/main.yml deleted file mode 100644 index 55500c6..0000000 --- a/ansible/roles/backup_store/tasks/main.yml +++ /dev/null @@ -1,89 +0,0 @@ ---- -- name: Assert backup_store sources are sane - ansible.builtin.assert: - that: - - backup_store_sources | length > 0 - - backup_store_sources | map(attribute='name') | list | length == backup_store_sources | length - - backup_store_sources | map(attribute='source') | list | length == backup_store_sources | length - - backup_store_sources | map(attribute='retention_days') | list | length == backup_store_sources | length - fail_msg: "backup_store: every source needs name, source and retention_days" - quiet: true - -- name: Ensure rsync is installed - ansible.builtin.apt: - name: rsync - state: present - update_cache: yes - cache_valid_time: 3600 - become: yes - -- name: Ensure the backup store directory exists - ansible.builtin.file: - path: "{{ backup_store_dir }}" - state: directory - mode: '0700' - -- name: Install the pull-backups script - ansible.builtin.template: - src: pull-backups.sh.j2 - dest: /usr/local/bin/pull-backups.sh - owner: root - group: root - mode: '0755' - validate: "bash -n %s" - become: yes - -# An assertion that last night actually worked. Generated from the same source -# list as the puller, so it can never drift out of sync with what is supposed to -# be arriving. Runs on a timer AND is useful by hand. -- name: Install the backup check script - ansible.builtin.template: - src: check-backups.sh.j2 - dest: /usr/local/bin/check-backups.sh - owner: root - group: root - mode: '0755' - validate: "bash -n %s" - become: yes - -# The .service carries the push token, so it is 0600; the .timer is not secret. -- name: Install the check-backups systemd units - ansible.builtin.template: - src: "check-backups.{{ item.unit }}.j2" - dest: "/etc/systemd/system/check-backups.{{ item.unit }}" - owner: root - group: root - mode: "{{ item.mode }}" - loop: - - {unit: service, mode: "0600"} - - {unit: timer, mode: "0644"} - become: yes - -# restarted, not started: `started` is a no-op on an already-active timer, so a -# changed schedule would never be picked up. -- name: Enable the check-backups timer - ansible.builtin.systemd: - name: check-backups.timer - enabled: yes - state: restarted - daemon_reload: yes - become: yes - -- name: Install the pull-backups systemd units - ansible.builtin.template: - src: "pull-backups.{{ item }}.j2" - dest: "/etc/systemd/system/pull-backups.{{ item }}" - owner: root - group: root - mode: '0644' - loop: [service, timer] - become: yes - notify: Reload systemd for pull-backups - -- name: Enable the pull-backups timer - ansible.builtin.systemd: - name: pull-backups.timer - enabled: yes - state: started - daemon_reload: yes - become: yes diff --git a/ansible/roles/backup_store/templates/check-backups.service.j2 b/ansible/roles/backup_store/templates/check-backups.service.j2 deleted file mode 100644 index 3e50b6a..0000000 --- a/ansible/roles/backup_store/templates/check-backups.service.j2 +++ /dev/null @@ -1,16 +0,0 @@ -[Unit] -Description=Verify the nightly backup pull actually worked -After=network-online.target -Wants=network-online.target - -[Service] -Type=oneshot -User={{ ansible_user_id }} -ExecStart=/usr/local/bin/check-backups.sh {{ backup_store_check_max_age_hours }} -Environment=BACKUP_CHECK_PUSH_BASE={{ backup_store_check_push_base }} -Environment=BACKUP_CHECK_PUSH_TOKEN={{ backup_store_check_push_token }} -StandardOutput=journal -StandardError=journal - -[Install] -WantedBy=multi-user.target diff --git a/ansible/roles/backup_store/templates/check-backups.sh.j2 b/ansible/roles/backup_store/templates/check-backups.sh.j2 deleted file mode 100644 index 37c9916..0000000 --- a/ansible/roles/backup_store/templates/check-backups.sh.j2 +++ /dev/null @@ -1,197 +0,0 @@ -#!/usr/bin/env bash -# Assert the nightly backups actually worked. -# -# Run as {{ ansible_user_id }} on this host. Needs no sudo. -# -# What it CANNOT do: verify contents. The age identity lives only on lapy, so -# this host cannot decrypt anything it holds — by design. These are freshness, -# completeness and integrity checks. To verify content, decrypt on lapy: -# ssh {{ ansible_user_id }}@$(hostname) "cat ~/backups//" \ -# | age -d -i ~/.age/counterweight_age | tar -tzf - | head -# -# Exit 0 = everything passed (warnings allowed), 1 = at least one FAIL. -set -uo pipefail - -STORE="{{ backup_store_dir }}" -MAX_AGE_H="${1:-26}" # an artefact older than this is stale -NOW=$(date +%s) -fails=0; warns=0 - -# Colour only when attached to a terminal: this gets piped into files and, later, -# probably into a notification. -if [ -t 1 ]; then R=$'\033[31m'; Y=$'\033[33m'; G=$'\033[32m'; N=$'\033[0m' -else R=''; Y=''; G=''; N=''; fi - -# Per-source verdicts, so each source can be reported independently. A single -# aggregate red light tells you backups are broken; it does not tell you which -# one, which is the thing you need at 3am. -declare -A SRC_FAIL SRC_MSG -CURRENT="" - -red() { printf ' %sFAIL%s %s\n' "$R" "$N" "$*"; fails=$((fails+1)); - [ -n "$CURRENT" ] && { SRC_FAIL[$CURRENT]=1; SRC_MSG[$CURRENT]="${SRC_MSG[$CURRENT]:-}${SRC_MSG[$CURRENT]:+; }$*"; }; } -yell() { printf ' %sWARN%s %s\n' "$Y" "$N" "$*"; warns=$((warns+1)); } -ok() { printf ' %sok%s %s\n' "$G" "$N" "$*"; - [ -n "$CURRENT" ] && SRC_MSG[$CURRENT]="${SRC_MSG[$CURRENT]:-}${SRC_MSG[$CURRENT]:+; }$*"; } - -# --- Reporting ------------------------------------------------------------- -# Each source gets its own Gatus external endpoint, plus one for the store -# itself (the pull unit, the timer, and disk capacity). PUSH_BASE empty means -# report nowhere, which is valid: the exit code is still the whole answer. -PUSH_BASE="${BACKUP_CHECK_PUSH_BASE:-}" -PUSH_TOKEN="${BACKUP_CHECK_PUSH_TOKEN:-}" - -report() { - local key="$1" success="$2" message="$3" - [ -n "$PUSH_BASE" ] || return 0 - local encoded - encoded=$(printf '%s' "$message" | sed 's/%/%25/g; s/ /%20/g; s/&/%26/g; s/+/%2B/g; s/#/%23/g') - curl -s -o /dev/null --max-time 15 --retry 2 --retry-delay 3 -X POST \ - -H "Authorization: Bearer ${PUSH_TOKEN}" \ - "${PUSH_BASE}/${key}/external?success=${success}&error=${encoded}" 2>/dev/null || true -} - -hours_since() { echo $(( (NOW - $1) / 3600 )); } - -# Pull the dump timestamp out of _YYYYmmdd_HHMMSS.. This is when -# the SOURCE produced it, which is the thing that actually matters: a source -# whose timer died still pulls "ok" forever, because yesterday's artefact is -# still sitting there. Checking only the pull would miss exactly that. -dump_epoch() { - local base ts - base=$(basename "$1") - ts=$(echo "$base" | grep -oE '[0-9]{8}_[0-9]{6}' | head -1) || return 1 - [ -n "$ts" ] || return 1 - date -d "${ts:0:4}-${ts:4:2}-${ts:6:2} ${ts:9:2}:${ts:11:2}:${ts:13:2}" +%s 2>/dev/null -} - -check_source() { - local name="$1" keep="$2" dir="$STORE/$1" - printf '\n%s\n' "== $name" - CURRENT="$name" - SRC_FAIL[$name]=0 - SRC_MSG[$name]="" - - [ -d "$dir" ] || { red "$name: no directory $dir"; return; } - - local n; n=$(find "$dir" -maxdepth 1 -type f -name "${name}_*" | wc -l) - [ "$n" -gt 0 ] || { red "$name: no artefacts at all"; return; } - - local partials; partials=$(find "$dir" -maxdepth 1 -name '*.partial' | wc -l) - [ "$partials" -eq 0 ] || red "$name: $partials .partial file(s) pulled — the pull should exclude these" - - local newest; newest=$(ls -t "$dir"/${name}_* 2>/dev/null | head -1) - local prev; prev=$(ls -t "$dir"/${name}_* 2>/dev/null | sed -n 2p) - - # 1. Did the SOURCE dump recently? - local de; de=$(dump_epoch "$newest") - if [ -z "${de:-}" ]; then - yell "$name: cannot parse a dump timestamp from $(basename "$newest")" - else - local dh; dh=$(hours_since "$de") - if [ "$dh" -lt 0 ]; then - # A future-dated artefact would otherwise stay "fresh" forever and the - # staleness check would never fire again — the exact silent failure this - # script exists to catch. - red "$name: newest dump is dated ${dh#-}h in the FUTURE — clock skew on the source?" - elif [ "$dh" -gt "$MAX_AGE_H" ]; then - red "$name: newest dump is ${dh}h old (>${MAX_AGE_H}h) — the source timer did not run" - else - ok "$name: dumped ${dh}h ago" - fi - fi - - # 2. Did the PULL bring it over recently? - local ph; ph=$(hours_since "$(stat -c %Y "$newest")") - if [ "$ph" -gt "$MAX_AGE_H" ]; then - red "$name: newest artefact was pulled ${ph}h ago (>${MAX_AGE_H}h)" - else - ok "$name: pulled ${ph}h ago" - fi - - # 3. Is it plausibly a real backup? - local sz; sz=$(stat -c %s "$newest") - if [ "$sz" -eq 0 ]; then - red "$name: newest artefact is ZERO bytes" - elif [ -n "$prev" ]; then - local psz; psz=$(stat -c %s "$prev") - if [ "$psz" -gt 0 ] && [ "$sz" -lt $(( psz / 2 )) ]; then - # Not automatically wrong: headscale legitimately shrank 297K -> 20K when - # a clean stop checkpointed its write-ahead log into the database. - yell "$name: $(numfmt --to=iec "$sz") is less than half the previous $(numfmt --to=iec "$psz") — check it decrypts to what you expect" - else - ok "$name: $(numfmt --to=iec "$sz") ($n artefacts)" - fi - else - ok "$name: $(numfmt --to=iec "$sz") (first artefact)" - fi - - # 4. Is retention pruning? Allow generous slack for multiple dumps per day. - if [ "$n" -gt $(( keep * 3 + 10 )) ]; then - yell "$name: $n artefacts for a ${keep}-day retention — pruning may not be working" - fi -} - -echo "Backup check on $(hostname) at $(date '+%Y-%m-%d %H:%M:%S %Z')" -echo "Artefacts older than ${MAX_AGE_H}h are treated as stale." - -# --- the pull job itself --- -# Reserved key, reported as backup-store_pull-job. The store's own machinery is -# a different alarm from any one source being stale, and it is a LEADING -# indicator where the per-source checks are lagging ones: those only fire once -# an artefact is >26h stale, i.e. about a day after the fault. A disabled timer, -# a failed pull job or a filling disk are all visible here immediately, and they -# name the cause instead of showing six stale sources with no explanation. -CURRENT="__store" -SRC_FAIL[__store]=0 -SRC_MSG[__store]="" -printf '\n%s\n' "== pull-backups.service" -result=$(systemctl show pull-backups.service -p Result --value 2>/dev/null) -status=$(systemctl show pull-backups.service -p ExecMainStatus --value 2>/dev/null) -when=$(systemctl show pull-backups.service -p ExecMainExitTimestamp --value 2>/dev/null) -[ "$result" = "success" ] && ok "last run result: success" || red "last run result: ${result:-unknown} (exit ${status:-?})" -if [ -n "$when" ]; then - wh=$(hours_since "$(date -d "$when" +%s)") - [ "$wh" -le "$MAX_AGE_H" ] && ok "last ran ${wh}h ago" || red "last ran ${wh}h ago (>${MAX_AGE_H}h) — did the timer fire?" -fi -systemctl is-enabled pull-backups.timer >/dev/null 2>&1 \ - && ok "timer enabled, next $(systemctl show pull-backups.timer -p NextElapseUSecRealtime --value 2>/dev/null)" \ - || red "pull-backups.timer is NOT enabled" - -# --- each source --- -{% for src in backup_store_sources %} -check_source "{{ src.name }}" {{ src.retention_days }} -CURRENT="" -# The store key is what Gatus computes from group+name: sanitize("backup-store") -# + "_" + sanitize("{{ src.name }}"). -report "backup-store_{{ src.name }}" \ - "$([ "${SRC_FAIL[{{ src.name }}]:-1}" -eq 0 ] && echo true || echo false)" \ - "${SRC_MSG[{{ src.name }}]:-no result}" -{% endfor %} - -# --- capacity --- -CURRENT="__store" -printf '\n%s\n' "== disk" -use=$(df --output=pcent "$STORE" | tail -1 | tr -dc '0-9') -avail=$(df -h --output=avail "$STORE" | tail -1 | tr -d ' ') -if [ "$use" -ge 90 ]; then red "store is ${use}% full, ${avail} free" -elif [ "$use" -ge 75 ]; then yell "store is ${use}% full, ${avail} free" -else ok "store is ${use}% full, ${avail} free"; fi - -CURRENT="" -report "backup-store_pull-job" \ - "$([ "${SRC_FAIL[__store]:-1}" -eq 0 ] && echo true || echo false)" \ - "${SRC_MSG[__store]:-no result}" - -printf '\n%s\n' "-----" -if [ "$fails" -gt 0 ]; then - echo "RESULT: $fails failure(s), $warns warning(s)" - echo "Investigate with: journalctl -u pull-backups -n 50 --no-pager" - exit 1 -fi -if [ "$warns" -gt 0 ]; then - echo "RESULT: all checks passed, $warns warning(s)" -else - echo "RESULT: all checks passed" -fi -exit 0 diff --git a/ansible/roles/backup_store/templates/check-backups.timer.j2 b/ansible/roles/backup_store/templates/check-backups.timer.j2 deleted file mode 100644 index ba5c5a8..0000000 --- a/ansible/roles/backup_store/templates/check-backups.timer.j2 +++ /dev/null @@ -1,11 +0,0 @@ -[Unit] -Description=Run the backup verification after the nightly pull -Requires=check-backups.service - -[Timer] -OnCalendar={{ backup_store_check_on_calendar }} -# Run a missed occurrence on the next boot rather than skipping the day. -Persistent=true - -[Install] -WantedBy=timers.target diff --git a/ansible/roles/backup_store/templates/pull-backups.service.j2 b/ansible/roles/backup_store/templates/pull-backups.service.j2 deleted file mode 100644 index c65516c..0000000 --- a/ansible/roles/backup_store/templates/pull-backups.service.j2 +++ /dev/null @@ -1,10 +0,0 @@ -[Unit] -Description=Pull encrypted backups from production - -[Service] -Type=oneshot -User={{ ansible_user_id }} -ExecStart=/usr/local/bin/pull-backups.sh -StandardOutput=journal -StandardError=journal -SyslogIdentifier=pull-backups diff --git a/ansible/roles/backup_store/templates/pull-backups.sh.j2 b/ansible/roles/backup_store/templates/pull-backups.sh.j2 deleted file mode 100644 index 1db7d9c..0000000 --- a/ansible/roles/backup_store/templates/pull-backups.sh.j2 +++ /dev/null @@ -1,48 +0,0 @@ -#!/usr/bin/env bash -# Pull encrypted backups from production — managed by Ansible (roles/backup_store) -# -# Everything here is already ciphertext: this host only moves and expires files, -# and holds no key that can read them. -set -uo pipefail # deliberately NOT -e; see the loop below - -SSH_KEY="{{ backup_store_ssh_key }}" -STORE="{{ backup_store_dir }}" - -log() { echo "$(date '+%Y-%m-%d %H:%M:%S') $*"; } -fail() { echo "$(date '+%Y-%m-%d %H:%M:%S') ERROR: $*" >&2; failures=$((failures + 1)); } - -failures=0 - -# One source failing must not stop the others. The whole point of this box is -# that a single dead host cannot silently take the rest of the backups with it — -# which is exactly how the laptop-based jobs failed unnoticed for nine months. -{% for src in backup_store_sources %} -# --- {{ src.name }} --- -pull_{{ src.name | replace('-', '_') }}() { - local dir="${STORE}/{{ src.name }}" - mkdir -p "$dir" - log "Pulling {{ src.name }} from {{ src.source }}..." - # --exclude '*.partial': a dump that died mid-write leaves one behind, owned - # root:root 0600 because the chown only happens after a successful mv. Without - # this exclude the pull account cannot read it and rsync fails for the WHOLE - # source — so one failed dump would silently block every subsequent pull of - # that service. An incomplete artefact is never worth transferring anyway. - if rsync -az --timeout=120 --exclude '*.partial' \ - -e "ssh -i $SSH_KEY -o StrictHostKeyChecking=accept-new -o ConnectTimeout=15" \ - "{{ src.source }}" "$dir/"; then - log " {{ src.name }}: ok ($(find "$dir" -maxdepth 1 -type f | wc -l) artefacts, $(du -sh "$dir" | cut -f1))" - else - fail "{{ src.name }}: rsync failed" - return 1 - fi - log " {{ src.name }}: pruning older than {{ src.retention_days }} days" - find "$dir" -maxdepth 1 -type f -name '{{ src.name }}_*' -mtime +{{ src.retention_days }} -delete -} -pull_{{ src.name | replace('-', '_') }} || true - -{% endfor %} -if [ "$failures" -gt 0 ]; then - log "FAILED: $failures source(s) did not pull" - exit 1 -fi -log "All sources pulled." diff --git a/ansible/roles/backup_store/templates/pull-backups.timer.j2 b/ansible/roles/backup_store/templates/pull-backups.timer.j2 deleted file mode 100644 index 336e1ca..0000000 --- a/ansible/roles/backup_store/templates/pull-backups.timer.j2 +++ /dev/null @@ -1,9 +0,0 @@ -[Unit] -Description=Daily offsite backup pull - -[Timer] -OnCalendar={{ backup_store_on_calendar }} -Persistent=true - -[Install] -WantedBy=timers.target diff --git a/ansible/roles/bitcoin_knots/README.md b/ansible/roles/bitcoin_knots/README.md deleted file mode 100644 index 8e35900..0000000 --- a/ansible/roles/bitcoin_knots/README.md +++ /dev/null @@ -1,85 +0,0 @@ -# `bitcoin_knots` - -Builds Bitcoin Knots from source with PGP + SHA256 verification of the release -tarball, runs it as a full node on `knots-box`, and keeps a health check on a -systemd timer. The second play in the calling playbook publishes the P2P port -from the edge host via `socket_proxy`. - -Converted from `deploy_bitcoin_knots_playbook.yml` (892 lines) under Plan 6. The -playbook is now 40 lines. - -## The build is guarded; the chain is never touched - -`build.yml` is 32 tasks, every one carrying -`when: not bitcoind_binary_exists.stat.exists`. On a host that already has the -binary the whole download / verify / 30-60 minute compile skips — **including the -two `state: absent` deletions**, which target `/opt/bitcoin-knots/source` and the -extracted build directory. - -The chain lives elsewhere and nothing here touches it: - -| | | -|---|---| -| `bitcoin_knots_dir` | `/opt/bitcoin-knots` — build tree, safe to delete | -| `bitcoin_data_dir` | `/var/lib/bitcoin` — config, logs, wallets | -| `bitcoin_large_data_dir` | `/mnt/knots_data` — **~875 GB of blockchain** | - -The signature-verification tasks are the security control of this role. They are -copied verbatim; do not "simplify" them. - -## ⚠ This node is half of the mining setup - -`bitcoin.conf` carries a DATUM Gateway section that was hand-added on the node -and was **missing from the playbook's template**: - -```ini -blockmaxsize=3985000 -blockmaxweight=3985000 -blocknotify=killall -USR1 datum_gateway -maxmempool=1000 -blockreconstructionextratxn=1000000 -``` - -`blocknotify` is how `datum_gateway` learns a new block landed. Running the old -playbook would have deleted all of it, and solo mining would have carried on -grinding against a stale template — a silent failure that costs money rather -than raising an error. The template now carries it behind -`bitcoin_datum_gateway_enabled`. - -**bitcoin-knots and datum-gateway are one system, not two services.** Changing -either config means thinking about both. - -## The restart handler, and why exactness matters now - -The hand-written `Restart bitcoind` handler carried -`when: uptime_kuma_enabled | default(false)`, so it had been inert since the -decommissioning: `bitcoin.conf` and the systemd unit both notify it and neither -could restart anything. A config change applied to disk, reported success, and -never took effect. - -It is ungated here — which raises the bar for the template. **Any** residual -difference between the template and the live file, down to a trailing newline, -means the task reports `changed` and restarts a Bitcoin node on every run. It -took four rounds of `--check --diff` to reach `changed=0`: the DATUM section, an -explanatory comment that was rendering into the deployed file (now a `{# #}` -Jinja comment), a `# Pruning (optional)` comment the live file had, and one -trailing blank line. - -## `dbcache` - -Computed as 90% of RAM unless `bitcoin_dbcache_mb_override` is set. The live node -was hand-tuned to **200 MB**; the calculation produces 3528. As with fulcrum, -`set_fact` outranks role defaults, so the *calculation* honours the override — a -value pinned only in `defaults/` is silently ignored. - -## Monitoring: one variable, no product knowledge - -The check tests bitcoind's RPC and records the answer in its exit code, which -systemd keeps: `systemctl is-failed bitcoin-knots-healthcheck.service`. Set -`healthcheck_push_url` to report anywhere that accepts an HTTP ping. - -The timer had last fired **2026-08-09** while still reporting `active` and -`enabled` — the same `OnBootSec` + `OnUnitActiveSec` dead chain as fulcrum, where -nothing re-arms it if the service does not run in a given boot. The role runs the -check once after enabling, which both smoke-tests it and supplies the reference -the timer schedules from. diff --git a/ansible/roles/bitcoin_knots/defaults/main.yml b/ansible/roles/bitcoin_knots/defaults/main.yml deleted file mode 100644 index cce3850..0000000 --- a/ansible/roles/bitcoin_knots/defaults/main.yml +++ /dev/null @@ -1,93 +0,0 @@ -# Bitcoin Knots Configuration Variables - -# Version - REQUIRED: Specify exact version/tag to build -# The only version string. There used to be a second, v-prefixed copy -# (bitcoin_knots_version) that nothing read - two hand-maintained copies of one -# fact, with nothing keeping them in step. -bitcoin_knots_version_short: "29.2.knots20251110" - -# Directories -bitcoin_knots_dir: /opt/bitcoin-knots -bitcoin_knots_source_dir: "{{ bitcoin_knots_dir }}/source" -bitcoin_data_dir: /var/lib/bitcoin # Standard location for config, logs, wallets -bitcoin_large_data_dir: /mnt/knots_data # Custom location for blockchain data (blocks, chainstate) -bitcoin_conf_dir: /etc/bitcoin - -# Network -bitcoin_rpc_port: 8332 -# The edge host's socket-proxy/Caddy play needs this too, and a role default is -# invisible outside this role. The authoritative value for the live deployment is -# in host_vars/knots_box_local/main.yml, which outranks this; the value here is the -# protocol standard, so the role still works standalone. -bitcoin_p2p_port: 8333 -bitcoin_rpc_bind: "0.0.0.0" - -# RPC capacity. When a block lands, Fulcrum, mempool and DATUM all fetch it at -# once while validation holds the node busy on a saturated HDD; with the -# defaults the queue overflowed and bitcoind REJECTED calls ("http work queue -# depth exceeded" - 93 times on 2026-10-18), which also made the health probe -# flap. A deeper queue lets those bursts wait instead of failing. Costs a few -# idle threads and a little memory. -bitcoin_rpcworkqueue: 64 -bitcoin_rpcthreads: 8 - -# Build options -bitcoin_build_jobs: 4 # Parallel build jobs (-j flag), adjust based on CPU cores -bitcoin_build_prefix: /usr/local - -# Configuration options -bitcoin_enable_txindex: true # Set to true if transaction index needed (REQUIRED for Electrum servers like Electrs/ElectrumX) -bitcoin_max_connections: 125 -# dbcache will be calculated as 90% of host RAM automatically in playbook - -# ZMQ Configuration -bitcoin_zmq_enabled: true -bitcoin_zmq_bind: "tcp://0.0.0.0" -bitcoin_zmq_port_rawblock: 28332 -bitcoin_zmq_port_rawtx: 28333 -bitcoin_zmq_port_hashblock: 28334 -bitcoin_zmq_port_hashtx: 28335 - -# Service user -bitcoin_user: bitcoin -bitcoin_group: bitcoin - -# --- Health check ---------------------------------------------------------- -# Checks bitcoind RPC and records the answer in its exit code, which systemd -# keeps: `systemctl is-failed bitcoin-knots-healthcheck.service`. -# -# WHERE TO REPORT HEALTH — the one place to plug in monitoring. Empty means -# check, exit honestly, report nowhere. Any endpoint accepting an HTTP ping -# works; nothing here is specific to a monitoring product. -healthcheck_push_url: "" -# Bearer token for the Gatus external endpoint. Required whenever a push URL -# is set: Gatus rejects an unauthenticated push with 401. -healthcheck_push_token: "" - -# --- Logging ---------------------------------------------------------------- -# The live node logs to a file. Set to "" to use printtoconsole=1 (journald). -bitcoin_logfile: "{{ bitcoin_data_dir }}/debug.log" - -# --- dbcache ---------------------------------------------------------------- -# Computed as 90% of RAM unless this is set. As with fulcrum, note that -# set_fact outranks role defaults, so the CALCULATION has to honour this - a -# value pinned only in defaults/ is silently ignored. -# -# 1200, raised from a hand-tuned 200 on 2026-10-18. At 200 MB almost no UTXO -# lookup hit memory: the P2P thread alone read ~4 MB/s off the saturated HDD -# mirror validating relayed transactions, and block validation took a median -# 11 s, p90 103 s, max 344 s - long enough for RPC to stall and the health -# probe to fail. 1200 is what fits in the 4 GB VM (bitcoind ~1.5 GB RSS at -# 200, mempool ~3 MB in practice). Raising the VM to 6 GB would allow ~3000. -bitcoin_dbcache_mb_override: 1200 - -# --- DATUM Gateway ---------------------------------------------------------- -# This node feeds block templates to datum_gateway on knots-box. These settings -# were hand-added to bitcoin.conf and were missing from the template, so a -# playbook run would have removed them and broken the mining setup. -bitcoin_datum_gateway_enabled: true -bitcoin_blockmaxsize: 3985000 -bitcoin_blockmaxweight: 3985000 -bitcoin_blocknotify: "killall -USR1 datum_gateway" -bitcoin_maxmempool: 1000 -bitcoin_blockreconstructionextratxn: 1000000 diff --git a/ansible/roles/bitcoin_knots/handlers/main.yml b/ansible/roles/bitcoin_knots/handlers/main.yml deleted file mode 100644 index 49ac219..0000000 --- a/ansible/roles/bitcoin_knots/handlers/main.yml +++ /dev/null @@ -1,14 +0,0 @@ ---- -# Ungated on purpose. The hand-written handler carried -# when: uptime_kuma_enabled | default(false) -# so it has been inert since the decommissioning. Two tasks notify it — -# bitcoin.conf and the systemd unit — and neither could actually restart -# bitcoind. A configuration change to a Bitcoin node therefore applied to disk, -# reported success, and silently never took effect. -# -# Restarting bitcoind re-opens the chainstate; it does not reindex. -- name: Restart bitcoind - systemd: - name: bitcoind - state: restarted - daemon_reload: yes diff --git a/ansible/roles/bitcoin_knots/tasks/build.yml b/ansible/roles/bitcoin_knots/tasks/build.yml deleted file mode 100644 index 5a1b900..0000000 --- a/ansible/roles/bitcoin_knots/tasks/build.yml +++ /dev/null @@ -1,222 +0,0 @@ ---- -# Every task here is guarded by `when: not bitcoind_binary_exists.stat.exists`, -# so on a host that already has the binary the whole download / verify / build -# sequence skips — including the two `state: absent` deletions, which target -# /opt/bitcoin-knots/{source,bitcoin-} and never the chain data in -# /mnt/knots_data. -- name: Check if bitcoind binary already exists - stat: - path: "{{ bitcoin_build_prefix }}/bin/bitcoind" - register: bitcoind_binary_exists - changed_when: false - -- name: Install gnupg for signature verification - apt: - name: gnupg - state: present - when: not bitcoind_binary_exists.stat.exists - -- name: Import Luke Dashjr's Bitcoin Knots signing key - command: gpg --keyserver hkps://keyserver.ubuntu.com --recv-keys 90C8019E36C2E964 - register: key_import - changed_when: "'already in secret keyring' not in key_import.stdout and 'already in public keyring' not in key_import.stdout" - when: not bitcoind_binary_exists.stat.exists - failed_when: key_import.rc != 0 - -- name: Display imported key fingerprint - command: gpg --fingerprint 90C8019E36C2E964 - register: key_fingerprint - changed_when: false - when: not bitcoind_binary_exists.stat.exists - -- name: Download SHA256SUMS file - get_url: - url: "https://bitcoinknots.org/files/{{ bitcoin_version_major }}.x/{{ bitcoin_knots_version_short }}/SHA256SUMS" - dest: "/tmp/bitcoin-knots-{{ bitcoin_knots_version_short }}-SHA256SUMS" - mode: '0644' - when: not bitcoind_binary_exists.stat.exists - -- name: Download SHA256SUMS.asc signature file - get_url: - url: "https://bitcoinknots.org/files/{{ bitcoin_version_major }}.x/{{ bitcoin_knots_version_short }}/SHA256SUMS.asc" - dest: "/tmp/bitcoin-knots-{{ bitcoin_knots_version_short }}-SHA256SUMS.asc" - mode: '0644' - when: not bitcoind_binary_exists.stat.exists - -- name: Verify PGP signature on SHA256SUMS file - command: gpg --verify /tmp/bitcoin-knots-{{ bitcoin_knots_version_short }}-SHA256SUMS.asc /tmp/bitcoin-knots-{{ bitcoin_knots_version_short }}-SHA256SUMS - register: sha256sums_verification - changed_when: false - failed_when: false # Don't fail here - check for 'Good signature' in next task - when: not bitcoind_binary_exists.stat.exists - - -- name: Display SHA256SUMS verification result - debug: - msg: "{{ sha256sums_verification.stdout_lines + sha256sums_verification.stderr_lines }}" - when: not bitcoind_binary_exists.stat.exists - -- name: Fail if SHA256SUMS signature verification failed - fail: - msg: "SHA256SUMS signature verification failed. Aborting build." - when: not bitcoind_binary_exists.stat.exists and ('Good signature' not in sha256sums_verification.stdout and 'Good signature' not in sha256sums_verification.stderr) - -- name: Remove any existing tarball to force fresh download - file: - path: /tmp/bitcoin-{{ bitcoin_knots_version_short }}.tar.gz - state: absent - when: not bitcoind_binary_exists.stat.exists - -- name: Download Bitcoin Knots source tarball - get_url: - url: "{{ bitcoin_source_tarball_url }}" - dest: "/tmp/bitcoin-{{ bitcoin_knots_version_short }}.tar.gz" - mode: '0644' - validate_certs: yes - force: yes - when: not bitcoind_binary_exists.stat.exists - -- name: Calculate SHA256 checksum of downloaded tarball - command: sha256sum /tmp/bitcoin-{{ bitcoin_knots_version_short }}.tar.gz - register: tarball_checksum - changed_when: false - when: not bitcoind_binary_exists.stat.exists - -- name: Extract expected checksum from SHA256SUMS file - shell: grep "bitcoin-{{ bitcoin_knots_version_short }}.tar.gz" /tmp/bitcoin-knots-{{ bitcoin_knots_version_short }}-SHA256SUMS | awk '{print $1}' - register: expected_checksum - changed_when: false - when: not bitcoind_binary_exists.stat.exists - failed_when: expected_checksum.stdout == "" - -- name: Display checksum comparison - debug: - msg: - - "Expected: {{ expected_checksum.stdout | trim }}" - - "Actual: {{ tarball_checksum.stdout.split()[0] }}" - when: not bitcoind_binary_exists.stat.exists - -- name: Verify tarball checksum matches SHA256SUMS - fail: - msg: "Tarball checksum mismatch! Expected {{ expected_checksum.stdout | trim }}, got {{ tarball_checksum.stdout.split()[0] }}" - when: not bitcoind_binary_exists.stat.exists and expected_checksum.stdout | trim != tarball_checksum.stdout.split()[0] - -- name: Remove existing source directory if it exists (to force fresh extraction) - file: - path: "{{ bitcoin_knots_source_dir }}" - state: absent - when: not bitcoind_binary_exists.stat.exists - -- name: Remove extracted directory if it exists (from previous runs) - file: - path: "{{ bitcoin_knots_dir }}/bitcoin-{{ bitcoin_knots_version_short }}" - state: absent - when: not bitcoind_binary_exists.stat.exists - -- name: Extract verified source tarball - unarchive: - src: /tmp/bitcoin-{{ bitcoin_knots_version_short }}.tar.gz - dest: "{{ bitcoin_knots_dir }}" - remote_src: yes - when: not bitcoind_binary_exists.stat.exists - -- name: Check if extracted directory exists - stat: - path: "{{ bitcoin_knots_dir }}/bitcoin-{{ bitcoin_knots_version_short }}" - register: extracted_dir_stat - changed_when: false - when: not bitcoind_binary_exists.stat.exists - -- name: Rename extracted directory to expected name - command: mv "{{ bitcoin_knots_dir }}/bitcoin-{{ bitcoin_knots_version_short }}" "{{ bitcoin_knots_source_dir }}" - when: not bitcoind_binary_exists.stat.exists and extracted_dir_stat.stat.exists - -- name: Check if CMakeLists.txt exists - stat: - path: "{{ bitcoin_knots_source_dir }}/CMakeLists.txt" - register: cmake_exists - changed_when: false - when: not bitcoind_binary_exists.stat.exists - -- name: Create CMake build directory - file: - path: "{{ bitcoin_knots_source_dir }}/build" - state: directory - mode: '0755' - when: not bitcoind_binary_exists.stat.exists and cmake_exists.stat.exists | default(false) - -- name: Configure Bitcoin Knots build with CMake - command: > - cmake - -DCMAKE_INSTALL_PREFIX={{ bitcoin_build_prefix }} - -DBUILD_BITCOIN_WALLET=OFF - -DCMAKE_BUILD_TYPE=Release - -DWITH_ZMQ=ON - .. - args: - chdir: "{{ bitcoin_knots_source_dir }}/build" - when: not bitcoind_binary_exists.stat.exists and cmake_exists.stat.exists | default(false) - register: configure_result - changed_when: true - -- name: Verify CMake enabled ZMQ - shell: | - set -e - cd "{{ bitcoin_knots_source_dir }}/build" - cmake -LAH .. | grep -iE 'ZMQ|WITH_ZMQ|ENABLE_ZMQ|USE_ZMQ' - when: not bitcoind_binary_exists.stat.exists and cmake_exists.stat.exists | default(false) - register: zmq_check - changed_when: false - -- name: Fail if CMakeLists.txt not found - fail: - msg: "CMakeLists.txt not found in {{ bitcoin_knots_source_dir }}. Cannot build Bitcoin Knots." - when: not bitcoind_binary_exists.stat.exists and not (cmake_exists.stat.exists | default(false)) - -- name: Build Bitcoin Knots with CMake (this may take 30-60+ minutes) - command: cmake --build . -j{{ bitcoin_build_jobs }} - args: - chdir: "{{ bitcoin_knots_source_dir }}/build" - when: not bitcoind_binary_exists.stat.exists and cmake_exists.stat.exists | default(false) - async: 3600 - poll: 0 - register: build_result - changed_when: true - -- name: Check build status - async_status: - jid: "{{ build_result.ansible_job_id }}" - register: build_job_result - until: build_job_result.finished - retries: 120 - delay: 60 - when: not bitcoind_binary_exists.stat.exists and build_result.ansible_job_id is defined - -- name: Fail if build failed - fail: - msg: "Bitcoin Knots build failed: {{ build_job_result.msg }}" - when: not bitcoind_binary_exists.stat.exists and build_result.ansible_job_id is defined and build_job_result.failed | default(false) - -- name: Install Bitcoin Knots binaries - command: cmake --install . - args: - chdir: "{{ bitcoin_knots_source_dir }}/build" - when: not bitcoind_binary_exists.stat.exists and cmake_exists.stat.exists | default(false) - changed_when: true - -- name: Verify bitcoind binary exists - stat: - path: "{{ bitcoin_build_prefix }}/bin/bitcoind" - register: bitcoind_installed - changed_when: false - -- name: Verify bitcoin-cli binary exists - stat: - path: "{{ bitcoin_build_prefix }}/bin/bitcoin-cli" - register: bitcoin_cli_installed - changed_when: false - -- name: Fail if binaries not found - fail: - msg: "Bitcoin Knots binaries not found after installation" - when: not bitcoind_installed.stat.exists or not bitcoin_cli_installed.stat.exists diff --git a/ansible/roles/bitcoin_knots/tasks/configure.yml b/ansible/roles/bitcoin_knots/tasks/configure.yml deleted file mode 100644 index 52fd3fd..0000000 --- a/ansible/roles/bitcoin_knots/tasks/configure.yml +++ /dev/null @@ -1,20 +0,0 @@ ---- -# Ownership copied verbatim from the playbook this replaces; verified -# mechanically against `git show HEAD:` rather than retyped from memory. -- name: Create bitcoin.conf configuration file - ansible.builtin.template: - src: bitcoin.conf.j2 - dest: "{{ bitcoin_conf_dir }}/bitcoin.conf" - owner: "{{ bitcoin_user }}" - group: "{{ bitcoin_group }}" - mode: '0640' - notify: Restart bitcoind - -- name: Create systemd service file for bitcoind - ansible.builtin.template: - src: bitcoind.service.j2 - dest: /etc/systemd/system/bitcoind.service - owner: root - group: root - mode: '0644' - notify: Restart bitcoind diff --git a/ansible/roles/bitcoin_knots/tasks/healthcheck.yml b/ansible/roles/bitcoin_knots/tasks/healthcheck.yml deleted file mode 100644 index c6c8db3..0000000 --- a/ansible/roles/bitcoin_knots/tasks/healthcheck.yml +++ /dev/null @@ -1,56 +0,0 @@ ---- -# Everything here answers "is bitcoind healthy" and records the answer. The -# Uptime Kuma specifics that used to follow — an embedded Python script creating -# monitors over the API, a /tmp credentials file, push-URL extraction and a -# systemd Environment= rewrite — are gone. Where it reports is now one variable, -# healthcheck_push_url. See the role README. -- name: Install curl for health check script - apt: - name: curl - state: present - -- name: Create Bitcoin Knots health check script - ansible.builtin.template: - src: healthcheck.sh.j2 - dest: /usr/local/bin/bitcoin-knots-healthcheck-push.sh - owner: root - group: root - mode: '0755' - validate: "bash -n %s" - -- name: Create systemd service for Bitcoin Knots health check - ansible.builtin.template: - src: healthcheck.service.j2 - dest: /etc/systemd/system/bitcoin-knots-healthcheck.service - owner: root - group: root - mode: "0600" - -- name: Create systemd timer for Bitcoin Knots health check - ansible.builtin.template: - src: healthcheck.timer.j2 - dest: /etc/systemd/system/bitcoin-knots-healthcheck.timer - owner: root - group: root - mode: '0644' - -- name: Reload systemd daemon for health check - systemd: - daemon_reload: yes - -- name: Enable and restart the Bitcoin Knots health check timer - systemd: - name: bitcoin-knots-healthcheck.timer - enabled: yes - state: restarted - daemon_reload: yes - -# Runs the check once, which is both a smoke test and the thing that actually -# arms the timer. This timer is OnBootSec + OnUnitActiveSec with no OnCalendar: -# OnBootSec elapses once, and OnUnitActiveSec needs the SERVICE to have run this -# boot to have anything to schedule from. Restarting the timer does not supply -# that reference; running the service does. The live timer had last fired on -# 2026-08-09 while still reporting `active` and `enabled`. -- name: Run the Bitcoin Knots health check once to arm the timer - command: systemctl start bitcoin-knots-healthcheck.service - changed_when: false diff --git a/ansible/roles/bitcoin_knots/tasks/install.yml b/ansible/roles/bitcoin_knots/tasks/install.yml deleted file mode 100644 index 3003b6e..0000000 --- a/ansible/roles/bitcoin_knots/tasks/install.yml +++ /dev/null @@ -1,104 +0,0 @@ ---- -- name: Calculate dbcache as a share of system RAM - set_fact: - bitcoin_dbcache_mb: "{{ (ansible_memtotal_mb | float * 0.9) | int }}" - when: bitcoin_dbcache_mb_override | string | length == 0 - -- name: Use the explicit dbcache override - set_fact: - bitcoin_dbcache_mb: "{{ bitcoin_dbcache_mb_override }}" - when: bitcoin_dbcache_mb_override | string | length > 0 - changed_when: false - -- name: Display calculated dbcache value - debug: - msg: "Setting dbcache to {{ bitcoin_dbcache_mb }} MB (90% of {{ ansible_memtotal_mb }} MB total RAM)" - - -- name: Install build dependencies - apt: - name: - - build-essential - - libtool - - autotools-dev - - automake - - pkg-config - - bsdmainutils - - python3 - - python3-pip - - libevent-dev - - libboost-system-dev - - libboost-filesystem-dev - - libboost-test-dev - - libboost-thread-dev - - libboost-chrono-dev - - libboost-program-options-dev - - libboost-dev - - libssl-dev - - libdb-dev - - libminiupnpc-dev - - libzmq3-dev - - libnatpmp-dev - - libsqlite3-dev - - git - - curl - - wget - - cmake - state: present - update_cache: yes - -- name: Create bitcoin group - group: - name: "{{ bitcoin_group }}" - system: yes - state: present - -- name: Create bitcoin user - user: - name: "{{ bitcoin_user }}" - group: "{{ bitcoin_group }}" - system: yes - shell: /usr/sbin/nologin - home: "{{ bitcoin_data_dir }}" - create_home: yes - state: present - -- name: Create bitcoin-knots directory - file: - path: "{{ bitcoin_knots_dir }}" - state: directory - owner: root - group: root - mode: '0755' - -- name: Create bitcoin-knots source directory - file: - path: "{{ bitcoin_knots_source_dir }}" - state: directory - owner: root - group: root - mode: '0755' - -- name: Create bitcoin data directory (for config, logs, wallets) - file: - path: "{{ bitcoin_data_dir }}" - state: directory - owner: "{{ bitcoin_user }}" - group: "{{ bitcoin_group }}" - mode: '0750' - -- name: Create bitcoin large data directory (for blockchain) - file: - path: "{{ bitcoin_large_data_dir }}" - state: directory - owner: "{{ bitcoin_user }}" - group: "{{ bitcoin_group }}" - mode: '0750' - -- name: Create bitcoin config directory - file: - path: "{{ bitcoin_conf_dir }}" - state: directory - owner: root - group: root - mode: '0755' diff --git a/ansible/roles/bitcoin_knots/tasks/main.yml b/ansible/roles/bitcoin_knots/tasks/main.yml deleted file mode 100644 index ab869a7..0000000 --- a/ansible/roles/bitcoin_knots/tasks/main.yml +++ /dev/null @@ -1,8 +0,0 @@ ---- -# import_tasks, not include_tasks: static imports stay visible to --list-tasks, -# which is how this conversion was verified against the playbook it replaced. -- ansible.builtin.import_tasks: install.yml -- ansible.builtin.import_tasks: build.yml -- ansible.builtin.import_tasks: configure.yml -- ansible.builtin.import_tasks: service.yml -- ansible.builtin.import_tasks: healthcheck.yml diff --git a/ansible/roles/bitcoin_knots/tasks/service.yml b/ansible/roles/bitcoin_knots/tasks/service.yml deleted file mode 100644 index 6f6149e..0000000 --- a/ansible/roles/bitcoin_knots/tasks/service.yml +++ /dev/null @@ -1,37 +0,0 @@ ---- -- name: Reload systemd daemon - systemd: - daemon_reload: yes - -- name: Enable and start bitcoind service - systemd: - name: bitcoind - enabled: yes - state: started - -- name: Wait for bitcoind RPC to be available - uri: - url: "http://{{ bitcoin_rpc_bind }}:{{ bitcoin_rpc_port }}" - method: POST - body_format: json - body: - jsonrpc: "1.0" - id: "healthcheck" - method: "getblockchaininfo" - params: [] - user: "{{ bitcoin_rpc_user }}" - password: "{{ bitcoin_rpc_password }}" - status_code: 200 - timeout: 10 - register: rpc_check - until: rpc_check.status == 200 - retries: 30 - delay: 5 - ignore_errors: yes - -- name: Display RPC connection status - debug: - msg: "Bitcoin Knots RPC is {{ 'available' if rpc_check.status == 200 else 'not yet available' }}" - -# ═════════════════════════════════════════════════════════════════════════ -# DEPRECATED — Uptime Kuma was decommissioned on 2026-09-11. diff --git a/ansible/roles/bitcoin_knots/templates/bitcoin.conf.j2 b/ansible/roles/bitcoin_knots/templates/bitcoin.conf.j2 deleted file mode 100644 index 4187e24..0000000 --- a/ansible/roles/bitcoin_knots/templates/bitcoin.conf.j2 +++ /dev/null @@ -1,69 +0,0 @@ -# Bitcoin Knots Configuration -# Generated by Ansible - -# Data directory (blockchain storage) -datadir={{ bitcoin_large_data_dir }} - -# RPC Configuration -server=1 -rpcuser={{ bitcoin_rpc_user }} -rpcpassword={{ bitcoin_rpc_password }} -rpcbind={{ bitcoin_rpc_bind }} -rpcport={{ bitcoin_rpc_port }} -rpcallowip=0.0.0.0/0 -rpcworkqueue={{ bitcoin_rpcworkqueue }} -rpcthreads={{ bitcoin_rpcthreads }} - -# Network Configuration -listen=1 -port={{ bitcoin_p2p_port }} -maxconnections={{ bitcoin_max_connections }} - -# Performance -dbcache={{ bitcoin_dbcache_mb }} - -# Transaction Index (optional) -{% if bitcoin_enable_txindex %} -txindex=1 -{% endif %} - -{# The live node carries this comment and the template never produced it, so a - run would have silently deleted it. Harmless in itself, but matching it keeps - this task at `ok` - which means any future `changed` here is a real signal - rather than known noise. #} -# Pruning (optional) - -# Logging -logtimestamps=1 -{% if bitcoin_logfile %} -logfile={{ bitcoin_logfile }} -{% else %} -printtoconsole=1 -{% endif %} - -# ZMQ Configuration -{% if bitcoin_zmq_enabled | default(false) %} -zmqpubrawblock={{ bitcoin_zmq_bind }}:{{ bitcoin_zmq_port_rawblock }} -zmqpubrawtx={{ bitcoin_zmq_bind }}:{{ bitcoin_zmq_port_rawtx }} -zmqpubhashblock={{ bitcoin_zmq_bind }}:{{ bitcoin_zmq_port_hashblock }} -zmqpubhashtx={{ bitcoin_zmq_bind }}:{{ bitcoin_zmq_port_hashtx }} -{% endif %} - -# Security -disablewallet=1 -{% if bitcoin_datum_gateway_enabled %} - -{# These were hand-added on the node and were NOT in this template, so running - the playbook would have stripped them. blocknotify is how datum_gateway - learns a new block landed; without it solo mining keeps grinding on a stale - template - a silent failure that costs money rather than raising an error. - Kept as a Jinja comment so the explanation stays in the repo and out of the - deployed config. #} -# Specific for DATUM gateway -blockmaxsize={{ bitcoin_blockmaxsize }} -blockmaxweight={{ bitcoin_blockmaxweight }} -blocknotify={{ bitcoin_blocknotify }} -maxmempool={{ bitcoin_maxmempool }} -blockreconstructionextratxn={{ bitcoin_blockreconstructionextratxn }} - -{% endif %} diff --git a/ansible/roles/bitcoin_knots/templates/bitcoind.service.j2 b/ansible/roles/bitcoin_knots/templates/bitcoind.service.j2 deleted file mode 100644 index ac1140f..0000000 --- a/ansible/roles/bitcoin_knots/templates/bitcoind.service.j2 +++ /dev/null @@ -1,17 +0,0 @@ -[Unit] -Description=Bitcoin Knots daemon -After=network.target - -[Service] -Type=simple -User={{ bitcoin_user }} -Group={{ bitcoin_group }} -ExecStart={{ bitcoin_build_prefix }}/bin/bitcoind -conf={{ bitcoin_conf_dir }}/bitcoin.conf -Restart=always -RestartSec=10 -TimeoutStopSec=600 -StandardOutput=journal -StandardError=journal - -[Install] -WantedBy=multi-user.target diff --git a/ansible/roles/bitcoin_knots/templates/healthcheck.service.j2 b/ansible/roles/bitcoin_knots/templates/healthcheck.service.j2 deleted file mode 100644 index 9df056e..0000000 --- a/ansible/roles/bitcoin_knots/templates/healthcheck.service.j2 +++ /dev/null @@ -1,15 +0,0 @@ -[Unit] -Description=Bitcoin Knots Health Check -After=network.target bitcoind.service - -[Service] -Type=oneshot -User=root -ExecStart=/usr/local/bin/bitcoin-knots-healthcheck-push.sh -Environment=HEALTHCHECK_PUSH_URL={{ healthcheck_push_url }} -Environment=HEALTHCHECK_PUSH_TOKEN={{ healthcheck_push_token }} -StandardOutput=journal -StandardError=journal - -[Install] -WantedBy=multi-user.target diff --git a/ansible/roles/bitcoin_knots/templates/healthcheck.sh.j2 b/ansible/roles/bitcoin_knots/templates/healthcheck.sh.j2 deleted file mode 100644 index 538b8dd..0000000 --- a/ansible/roles/bitcoin_knots/templates/healthcheck.sh.j2 +++ /dev/null @@ -1,69 +0,0 @@ -#!/bin/bash -# Bitcoin Knots health check — managed by Ansible (roles/bitcoin_knots) -# -# The exit code is the answer and systemd keeps it: -# systemctl is-failed bitcoin-knots-healthcheck.service -# Reporting anywhere else is optional and generic. -# -# - -RPC_HOST="{{ bitcoin_rpc_bind }}" -RPC_PORT={{ bitcoin_rpc_port }} -RPC_USER="{{ bitcoin_rpc_user }}" -RPC_PASSWORD="{{ bitcoin_rpc_password }}" -PUSH_URL="${HEALTHCHECK_PUSH_URL:-}" -PUSH_TOKEN="${HEALTHCHECK_PUSH_TOKEN:-}" - -# Check if bitcoind RPC is responding -check_bitcoind() { - local response - response=$(curl -s --max-time 30 \ - --user "${RPC_USER}:${RPC_PASSWORD}" \ - --data-binary '{"jsonrpc":"1.0","id":"healthcheck","method":"getblockchaininfo","params":[]}' \ - --header 'Content-Type: application/json' \ - "http://${RPC_HOST}:${RPC_PORT}" 2>&1) - - if [ $? -eq 0 ]; then - # Check if response contains a non-null error - # Successful responses have "error": null, failures have "error": {...} - if echo "$response" | grep -q '"error":null\|"error": null'; then - return 0 - else - return 1 - fi - else - return 1 - fi -} - -report() { - local status=$1 - local msg=$2 - - # No push URL is normal, not an error: the exit code below is still a - # complete answer for anything reading unit state. - [ -n "$PUSH_URL" ] || return 0 - - # URL encode spaces in message - local encoded_msg="${msg// /%20}" - - # Gatus external endpoint: a POST with a bearer token, NOT Uptime Kuma's - # GET with ?status=up. The callers still pass up/down, so the mapping is - # done here rather than at every call site. - local _ok=false - [ "${status}" = "up" ] && _ok=true - if ! curl -s --max-time 15 --retry 2 -o /dev/null -X POST \ - -H "Authorization: Bearer ${PUSH_TOKEN}" \ - "${PUSH_URL}?success=${_ok}&error=${encoded_msg}"; then - return 1 - fi -} - -# Main health check -if check_bitcoind; then - report "up" "OK" - exit 0 -else - report "down" "bitcoind RPC not responding" - exit 1 -fi diff --git a/ansible/roles/bitcoin_knots/templates/healthcheck.timer.j2 b/ansible/roles/bitcoin_knots/templates/healthcheck.timer.j2 deleted file mode 100644 index d5857ab..0000000 --- a/ansible/roles/bitcoin_knots/templates/healthcheck.timer.j2 +++ /dev/null @@ -1,11 +0,0 @@ -[Unit] -Description=Bitcoin Knots Health Check Timer -Requires=bitcoind.service - -[Timer] -OnBootSec=1min -OnUnitActiveSec=1min -Persistent=true - -[Install] -WantedBy=timers.target diff --git a/ansible/roles/caddy_site/README.md b/ansible/roles/caddy_site/README.md deleted file mode 100644 index c8e32af..0000000 --- a/ansible/roles/caddy_site/README.md +++ /dev/null @@ -1,118 +0,0 @@ -# `caddy_site` - -Writes one Caddy site file into `{{ caddy_sites_dir }}`, makes sure the main -Caddyfile imports that directory, validates the result, and reloads Caddy once. - -Replaces the four-task block that was copy-pasted into 10 playbooks. - -Runs on any host in the `[caddy]` group — `edge` (vipy), `monitoring` -(watchtower) and `vpn_control` (spacey). - -## Usage - -```yaml -- ansible.builtin.include_role: - name: caddy_site - vars: - caddy_site_name: forgejo # -> forgejo.conf - caddy_site_domain: "{{ forgejo_domain }}" - caddy_site_upstream: "localhost:{{ forgejo_port }}" -``` - -Use `include_role`, not a `roles:` block, so the call stays in task order next -to the tasks it depends on. Variables passed this way are scoped to the include -and do not leak into later calls — so **every call must pass everything it -needs**; nothing carries over. - -## Shapes - -Pick exactly one of `caddy_site_upstream`, `caddy_site_root`, `caddy_site_body`. - -| Want | Set | -|---|---| -| `reverse_proxy host:port` | `caddy_site_upstream` | -| static `root *` + `file_server` | `caddy_site_root` | -| anything else | `caddy_site_body` (raw, indented 4 for you) | - -`caddy_site_upstream` accepts two modifiers, which add a block to the -`reverse_proxy`: - -- `caddy_site_headers_up: {"X-Forwarded-Host": "..."}` -- `caddy_site_resolvers: "100.100.100.100"` — Tailscale MagicDNS - -and `caddy_site_basic_auth` wraps the site in a `basic_auth` block. - -## `caddy_site_basic_auth` is a LIST, not a dict - -```yaml -caddy_site_basic_auth: - - user: "{{ datum_dashboard_username }}" - hash: "{{ datum_dashboard_password_hash }}" -``` - -**Ansible does not template dictionary keys.** With `{ "{{ user }}": "hash" }` -the value is rendered and the key is not, so the literal string -`{{ datum_dashboard_username }}` lands in the config file. Found while building -this role; the `assert` refuses a mapping so it cannot happen again. - -## Secrets and `--diff` - -Rendered site files can carry credentials — `datum-gateway.conf` holds a bcrypt -hash — and `--diff` prints rendered content. The template task therefore sets -`diff: "{{ caddy_site_reveal | bool }}"`, default `false`, so `--diff` runs are -safe everywhere. Pass `-e caddy_site_reveal=true` to see what moved on a site -you know is not secret. - -## Validation - -`validate: "caddy validate --adapter caddyfile --config %s"` runs against the -rendered temp file before it is moved into place. Verified on vipy that a single -site fragment validates cleanly (rc=0, `Valid configuration`) and that a -malformed one is rejected (rc=1, with the syntax error and line number). A -failed validate leaves the live file untouched, so a broken config can no longer -reach a running Caddy. - -What it cannot catch is a conflict with the global `/etc/caddy/Caddyfile`. - -## The reload is a handler - -`Reload caddy` fires **once, at the end of the play**, however many sites -notified it. The code this replaced ran `command: systemctl reload caddy` -immediately, mid-play. If a later task in the same play needs the new config to -be live, flush first: - -```yaml -- ansible.builtin.meta: flush_handlers -``` - -## Known intentional difference - -The `resolvers` block is commented `# Use Tailscale MagicDNS to resolve the -upstream hostname` in every case. `datum-gateway` previously said `# Resolve via -Tailscale MagicDNS`. Migrating it therefore rewrites one comment line, which -Caddy ignores. Every other site renders byte-identical to what its playbook -produced. - -## Sites on the hosts that this role does NOT manage - -Four vhosts exist in `/etc/caddy/sites-enabled/` that no playbook writes. They -were made by hand. The role only ever writes the one file it is told to, so it -leaves them alone — but nothing in the repo records them, and that is why they -are listed here. Checked 2026-09-11: - -| File | Host | Serves | State | -|---|---|---|---| -| `uptime-kuma.conf` | watchtower | `localhost:3001` | **HTTP 302 — still live**, see below | -| `arbretstaging.conf` | vipy | `arbret-staging-box:80` via MagicDNS | HTTP 200 | -| `bitcoininfra.conf` | vipy | static `file_server` from `/var/www/bitcoin-services-home` | HTTP 200 | -| `scriberr.conf` | vipy | `scriberr-box:8080` via MagicDNS | HTTP 502 — upstream down | - -**`uptime-kuma.conf` must not be deleted as dead config.** Uptime Kuma was -"decommissioned" in the repo — its playbooks archived and its credentials pulled -from the vault — but the container is **still running** on watchtower -(`louislam/uptime-kuma:latest`, created 2026-02-07, `restart=unless-stopped`) -and is still reachable at its public subdomain. Only the Ansible code was -retired; the service was not. See `archive/uptime_kuma/`. - -`scriberr` returning 502 is the one that looks like genuine rot: it proxies to a -`scriberr-box` that is not answering, and `scriberr-box` is not in the inventory. diff --git a/ansible/roles/caddy_site/defaults/main.yml b/ansible/roles/caddy_site/defaults/main.yml deleted file mode 100644 index 0a59906..0000000 --- a/ansible/roles/caddy_site/defaults/main.yml +++ /dev/null @@ -1,24 +0,0 @@ ---- -# Required -caddy_site_name: "" # file basename -> .conf -caddy_site_domain: "" # site address line; may hold several, comma separated - -# Pick exactly one shape -caddy_site_upstream: "" # "localhost:3000" -> reverse_proxy -caddy_site_root: "" # filesystem path -> root * + file_server -caddy_site_body: "" # raw escape hatch for one-off sites; wins over both - -# reverse_proxy modifiers -caddy_site_resolvers: "" # "100.100.100.100" for Tailscale MagicDNS -caddy_site_headers_up: {} # {"X-Forwarded-Host": "wallet.example.com"} -# A LIST, not a dict: Ansible does not template dict *keys*, so a Jinja -# expression for the username silently passes through as literal text. -caddy_site_basic_auth: [] # [{user: "{{ x_user }}", hash: "{{ x_hash }}"}] - -# Placement. This is now the only definition of caddy_sites_dir - services_config.yml -# used to carry an identical copy, which was removed as redundant. -caddy_sites_dir: /etc/caddy/sites-enabled - -# Rendered site files can carry credentials (basic_auth hashes), so --diff is -# suppressed by default. Pass -e caddy_site_reveal=true to see what moved. -caddy_site_reveal: false diff --git a/ansible/roles/caddy_site/handlers/main.yml b/ansible/roles/caddy_site/handlers/main.yml deleted file mode 100644 index 4bfa7ff..0000000 --- a/ansible/roles/caddy_site/handlers/main.yml +++ /dev/null @@ -1,12 +0,0 @@ ---- -# Fires once at the end of the play, however many sites notified it. -# Anything later in the same play that needs the new config live must be -# preceded by `- ansible.builtin.meta: flush_handlers`. -# become is explicit because handlers do not inherit it from the task that -# notified them. headscale's play runs become: no and elevates per task, so -# without this the reload would run unprivileged and fail. -- name: Reload caddy - become: true - ansible.builtin.systemd: - name: caddy - state: reloaded diff --git a/ansible/roles/caddy_site/tasks/main.yml b/ansible/roles/caddy_site/tasks/main.yml deleted file mode 100644 index 5ad3976..0000000 --- a/ansible/roles/caddy_site/tasks/main.yml +++ /dev/null @@ -1,46 +0,0 @@ ---- -- name: Assert caddy_site parameters are sane - ansible.builtin.assert: - that: - - caddy_site_name | length > 0 - - caddy_site_domain | length > 0 - - (caddy_site_upstream | length > 0) or (caddy_site_root | length > 0) or (caddy_site_body | length > 0) - - caddy_site_basic_auth is not mapping - fail_msg: >- - caddy_site: '{{ caddy_site_name | default("") }}' needs a name, a domain and - one of caddy_site_upstream / caddy_site_root / caddy_site_body. - caddy_site_basic_auth must be a LIST of {user, hash} — Ansible does not template dict keys. - quiet: true - -- name: Ensure Caddy sites-enabled directory exists - ansible.builtin.file: - path: "{{ caddy_sites_dir }}" - state: directory - owner: root - group: root - mode: '0755' - -- name: Ensure Caddyfile imports sites-enabled - ansible.builtin.lineinfile: - path: /etc/caddy/Caddyfile - line: 'import sites-enabled/*' - insertafter: EOF - state: present - create: yes - mode: '0644' - backup: yes - -# `validate` runs `caddy validate` against the rendered temp file before it is -# moved into place: verified on vipy that a single site fragment validates -# cleanly (rc=0, "Valid configuration") and that a malformed one is rejected -# (rc=1). A failed validate leaves the live file untouched. -- name: "Write Caddy site '{{ caddy_site_name }}'" - ansible.builtin.template: - src: site.conf.j2 - dest: "{{ caddy_sites_dir }}/{{ caddy_site_name }}.conf" - owner: root - group: root - mode: '0644' - validate: "caddy validate --adapter caddyfile --config %s" - diff: "{{ caddy_site_reveal | bool }}" - notify: Reload caddy diff --git a/ansible/roles/caddy_site/templates/site.conf.j2 b/ansible/roles/caddy_site/templates/site.conf.j2 deleted file mode 100644 index 5d34c01..0000000 --- a/ansible/roles/caddy_site/templates/site.conf.j2 +++ /dev/null @@ -1,34 +0,0 @@ -{{ caddy_site_domain }} { -{% if caddy_site_body %} -{{ caddy_site_body | trim | indent(4, first=True) }} -{% else %} -{% if caddy_site_basic_auth %} - basic_auth { -{% for cred in caddy_site_basic_auth %} - {{ cred.user }} {{ cred.hash }} -{% endfor %} - } -{% endif %} -{% if caddy_site_root %} - root * {{ caddy_site_root }} - file_server -{% endif %} -{% if caddy_site_upstream %} -{% if caddy_site_headers_up or caddy_site_resolvers %} - reverse_proxy {{ caddy_site_upstream }} { -{% for key, value in caddy_site_headers_up.items() %} - header_up {{ key }} {{ value }} -{% endfor %} -{% if caddy_site_resolvers %} - # Use Tailscale MagicDNS to resolve the upstream hostname - transport http { - resolvers {{ caddy_site_resolvers }} - } -{% endif %} - } -{% else %} - reverse_proxy {{ caddy_site_upstream }} -{% endif %} -{% endif %} -{% endif %} -} diff --git a/ansible/roles/datum_gateway/README.md b/ansible/roles/datum_gateway/README.md deleted file mode 100644 index cdf35e5..0000000 --- a/ansible/roles/datum_gateway/README.md +++ /dev/null @@ -1,73 +0,0 @@ -# `datum_gateway` - -Builds and runs [DATUM Gateway](https://github.com/OCEAN-xyz/datum_gateway), the -solo/pooled mining gateway, on `knots-box`. The calling playbook adds two more -plays on the edge host: the dashboard via `caddy_site`, and the public Stratum -port via `socket_proxy`. - -Converted from `deploy_datum_gateway_playbook.yml` (802 lines) under Plan 6. The -playbook is now 68 lines and keeps all three plays. - -## ⚠ This is half of a system - -The Bitcoin Knots node on the same host feeds this gateway through -`blocknotify=killall -USR1 datum_gateway` in `bitcoin.conf` — see -`roles/bitcoin_knots/README.md`, where that line was found to be missing from the -template entirely. **Changing either config means thinking about both.** - -Interrupting Stratum costs mining shares. Check before any run that restarts it: - -```bash -ss -tn state established '( sport = :23334 )' -``` - -## Two pieces of drift where the node was right - -The repo and the node had diverged on values that matter, and the deployment -would have applied the repo's: - -| | node (correct) | repo said | -|---|---|---| -| `datum_mining_address` | `bc1qvrj3g84…` | `bc1qdse9dsg…` | -| `pool_pass_workers` / `_full_users` | `false` | `true` | - -The address is the one that would have hurt: **it is where block rewards are -paid**, and unlike fulcrum and bitcoin-knots the `Restart datum-gateway` handler -here was *never* gated, so the change would have applied immediately rather than -sitting inert. Both corrected in the vault and defaults, with notes. - -Verify semantics rather than text when touching `config.json` — render it and -compare parsed JSON, because the live file is single-line and the template is -pretty-printed, so a textual diff is all noise: - -```python -json.load(open('live.json')) == json.load(open('rendered.json')) -``` - -## `config.json` holds real secrets — diff is suppressed - -The file carries `bitcoind.rpcpassword` and `api.admin_password`. `--diff` -prints rendered content, so the task sets `diff: false` by default; pass -`-e datum_reveal_config=true` to opt in. - -Note `pool_pass_workers` / `pool_pass_full_users` are **booleans**, not -passwords, despite the names — they control DATUM's pool-password passthrough. -`mining.pool_address` is a Bitcoin address and public by nature. - -## Expect `changed` on the compile every run - -`Configure cmake build` and `Compile datum_gateway` are bare `command:` tasks -with no `changed_when`, so they always report changed and always re-run. The -build is reproducible — `Install datum_gateway binary` sees identical content and -does not replace it, so the installed binary keeps its original timestamp — but -the compile itself is wasted work on every run. That is the idempotent floor, not -drift. - -## Monitoring: one variable, no product knowledge - -The check tests the gateway API and records the answer in its exit code, which -systemd keeps: `systemctl is-failed datum-gateway-healthcheck.service`. Set -`healthcheck_push_url` to report anywhere accepting an HTTP ping. - -Unlike the other services here, only the health-check *timer* handler was gated -by `uptime_kuma_enabled`; the main deployment restart worked throughout. diff --git a/ansible/roles/datum_gateway/defaults/main.yml b/ansible/roles/datum_gateway/defaults/main.yml deleted file mode 100644 index 0c30aa7..0000000 --- a/ansible/roles/datum_gateway/defaults/main.yml +++ /dev/null @@ -1,61 +0,0 @@ -# DATUM Gateway Configuration Variables -# https://github.com/OCEAN-xyz/datum_gateway - -# Version - pin to a specific tag -datum_gateway_version: "v0.4.1beta" - -# Directories -datum_gateway_dir: /opt/datum-gateway -datum_gateway_source_dir: "{{ datum_gateway_dir }}/source" -datum_gateway_config_dir: /etc/datum-gateway -datum_gateway_log_dir: /var/log/datum-gateway - -# Binary -datum_gateway_bin_path: /usr/local/bin/datum_gateway - -# Ports -# The edge host's socket-proxy/Caddy play needs this too, and a role default is -# invisible outside this role. The authoritative value for the live deployment is -# in host_vars/knots_box_local/main.yml, which outranks this; the value here is the -# protocol standard, so the role still works standalone. -datum_gateway_stratum_port: 23334 -datum_gateway_api_port: 7152 - -# Stratum settings -datum_vardiff_min: 524288 # Minimum share difficulty (must be power of 2; OCEAN floor overrides if higher) - -# Service user -datum_gateway_user: datum -datum_gateway_group: datum - -# Build options -datum_gateway_build_jobs: 4 - -# Bitcoin node connection -# The gateway runs on the same host as Bitcoin Knots so localhost RPC works. -# datum_bitcoin_rpc_url should include http:// and port. -datum_bitcoin_rpc_url: "http://127.0.0.1:8332" -# Note: bitcoin_rpc_user and bitcoin_rpc_password come from group_vars/all/vault.yml - -# Mining config -datum_coinbase_tag_primary: "DATUM" -datum_coinbase_tag_secondary: "BY ORDER OF BIP110" - -# Both false on the node; the vars file said true. Corrected 2026-09-13 to -# match reality, on the same basis as datum_mining_address: the running node -# is authoritative. These control DATUM's pool-password passthrough. -datum_pool_pass_workers: false -datum_pool_pass_full_users: false -datum_pooled_mining_only: true - - -# --- Health check ----------------------------------------------------------- -# Checks the DATUM Gateway API and records the answer in its exit code, which -# systemd keeps: `systemctl is-failed datum-gateway-healthcheck.service`. -# -# WHERE TO REPORT HEALTH — the one place to plug in monitoring. Empty means -# check, exit honestly, report nowhere. -healthcheck_push_url: "" -# Bearer token for the Gatus external endpoint. Required whenever a push URL -# is set: Gatus rejects an unauthenticated push with 401. -healthcheck_push_token: "" diff --git a/ansible/roles/datum_gateway/handlers/main.yml b/ansible/roles/datum_gateway/handlers/main.yml deleted file mode 100644 index b7cfe25..0000000 --- a/ansible/roles/datum_gateway/handlers/main.yml +++ /dev/null @@ -1,15 +0,0 @@ ---- -- name: Restart datum-gateway - systemd: - name: datum-gateway - state: restarted - daemon_reload: yes - -# Ungated. This one carried `when: uptime_kuma_enabled | default(false)` while -# the main Restart datum-gateway handler above did not — so on this service the -# deployment restart worked and only the health-check timer restart was dead. -- name: Restart datum-gateway health check timer - systemd: - name: datum-gateway-healthcheck.timer - state: restarted - daemon_reload: yes diff --git a/ansible/roles/datum_gateway/tasks/configure.yml b/ansible/roles/datum_gateway/tasks/configure.yml deleted file mode 100644 index 463dec0..0000000 --- a/ansible/roles/datum_gateway/tasks/configure.yml +++ /dev/null @@ -1,25 +0,0 @@ ---- -# Ownership copied verbatim from the playbook this replaces and verified -# mechanically against `git show HEAD:`. -- name: Write DATUM Gateway config.json - ansible.builtin.template: - src: config.json.j2 - dest: "{{ datum_gateway_config_dir }}/config.json" - owner: "{{ datum_gateway_user }}" - group: "{{ datum_gateway_group }}" - mode: '0640' - # config.json carries the bitcoind RPC password, the API admin password and - # the pool passwords. `--diff` prints rendered content, so running with --diff - # put all of them on the terminal and into any log capturing it. Suppressed by - # default; pass -e datum_reveal_config=true when you genuinely need the diff. - diff: "{{ datum_reveal_config | default(false) | bool }}" - notify: Restart datum-gateway - -- name: Create datum-gateway systemd service - ansible.builtin.template: - src: datum-gateway.service.j2 - dest: /etc/systemd/system/datum-gateway.service - owner: root - group: root - mode: '0644' - notify: Restart datum-gateway diff --git a/ansible/roles/datum_gateway/tasks/healthcheck.yml b/ansible/roles/datum_gateway/tasks/healthcheck.yml deleted file mode 100644 index b5ccb90..0000000 --- a/ansible/roles/datum_gateway/tasks/healthcheck.yml +++ /dev/null @@ -1,50 +0,0 @@ ---- -# Everything here answers "is DATUM Gateway healthy" and records the answer. The -# Uptime Kuma specifics that used to follow — an embedded Python script creating -# monitors over the API, a /tmp credentials file, a push-URL file read back and -# parsed, and a systemd Environment= rewrite — are gone. Where it reports is now -# one variable, healthcheck_push_url. -- name: Create DATUM Gateway health check script - ansible.builtin.template: - src: healthcheck.sh.j2 - dest: /usr/local/bin/datum-gateway-healthcheck-push.sh - owner: root - group: root - mode: '0755' - validate: "bash -n %s" - -- name: Create datum-gateway health check systemd service - ansible.builtin.template: - src: healthcheck.service.j2 - dest: /etc/systemd/system/datum-gateway-healthcheck.service - owner: root - group: root - mode: "0600" - notify: Restart datum-gateway health check timer - -- name: Create datum-gateway health check systemd timer - ansible.builtin.template: - src: healthcheck.timer.j2 - dest: /etc/systemd/system/datum-gateway-healthcheck.timer - owner: root - group: root - mode: '0644' - notify: Restart datum-gateway health check timer - -- name: Reload systemd daemon after health check units - systemd: - daemon_reload: yes - -# Ungated: enabling a timer is deployment, not monitoring. -- name: Enable and restart the datum-gateway health check timer - systemd: - name: datum-gateway-healthcheck.timer - enabled: yes - state: restarted - daemon_reload: yes - -# Arms the timer and smoke-tests the check. See roles/bitcoin_knots/README.md for -# why restarting the timer alone is not enough with OnBootSec + OnUnitActiveSec. -- name: Run the DATUM Gateway health check once to arm the timer - command: systemctl start datum-gateway-healthcheck.service - changed_when: false diff --git a/ansible/roles/datum_gateway/tasks/install.yml b/ansible/roles/datum_gateway/tasks/install.yml deleted file mode 100644 index 61dcadb..0000000 --- a/ansible/roles/datum_gateway/tasks/install.yml +++ /dev/null @@ -1,74 +0,0 @@ ---- -- name: Install DATUM Gateway build dependencies - apt: - name: - - cmake - - build-essential - - git - - libjansson-dev - - libmicrohttpd-dev - - libsodium-dev - - libcurl4-openssl-dev - # Runtime-only (netcat for health check) - - netcat-openbsd - state: present - update_cache: yes - -# =========================================== -# System User and Directories -# =========================================== -- name: Create datum system user - user: - name: "{{ datum_gateway_user }}" - system: yes - shell: /usr/sbin/nologin - home: "{{ datum_gateway_dir }}" - create_home: no - comment: "DATUM Gateway" - -- name: Create DATUM Gateway directories - file: - path: "{{ item.path }}" - state: directory - owner: "{{ item.owner }}" - group: "{{ datum_gateway_group }}" - mode: "{{ item.mode }}" - loop: - - { path: "{{ datum_gateway_dir }}", owner: root, mode: "0755" } - - { path: "{{ datum_gateway_source_dir }}", owner: root, mode: "0755" } - - { path: "{{ datum_gateway_config_dir }}", owner: "{{ datum_gateway_user }}", mode: "0750" } - - { path: "{{ datum_gateway_log_dir }}", owner: "{{ datum_gateway_user }}", mode: "0750" } - -# =========================================== -# Build from Source -# =========================================== -- name: Clone DATUM Gateway repository at {{ datum_gateway_version }} - git: - repo: https://github.com/OCEAN-xyz/datum_gateway.git - dest: "{{ datum_gateway_source_dir }}" - version: "{{ datum_gateway_version }}" - force: yes - register: git_clone - -- name: Configure cmake build - command: cmake . -DCMAKE_BUILD_TYPE=Release - args: - chdir: "{{ datum_gateway_source_dir }}" - -- name: Compile datum_gateway - command: make -j{{ datum_gateway_build_jobs }} - args: - chdir: "{{ datum_gateway_source_dir }}" - -- name: Install datum_gateway binary - copy: - src: "{{ datum_gateway_source_dir }}/datum_gateway" - dest: "{{ datum_gateway_bin_path }}" - remote_src: yes - owner: root - group: root - mode: "0755" - notify: Restart datum-gateway - -# =========================================== -# Configuration diff --git a/ansible/roles/datum_gateway/tasks/main.yml b/ansible/roles/datum_gateway/tasks/main.yml deleted file mode 100644 index 57d9a45..0000000 --- a/ansible/roles/datum_gateway/tasks/main.yml +++ /dev/null @@ -1,6 +0,0 @@ ---- -# import_tasks, not include_tasks: static imports stay visible to --list-tasks. -- ansible.builtin.import_tasks: install.yml -- ansible.builtin.import_tasks: configure.yml -- ansible.builtin.import_tasks: service.yml -- ansible.builtin.import_tasks: healthcheck.yml diff --git a/ansible/roles/datum_gateway/tasks/service.yml b/ansible/roles/datum_gateway/tasks/service.yml deleted file mode 100644 index c8581fd..0000000 --- a/ansible/roles/datum_gateway/tasks/service.yml +++ /dev/null @@ -1,16 +0,0 @@ ---- -- name: Reload systemd daemon - systemd: - daemon_reload: yes - -- name: Enable and start datum-gateway - systemd: - name: datum-gateway - enabled: yes - state: started - -# =========================================== -# Health Check Script + Systemd Timer -# =========================================== -# ═════════════════════════════════════════════════════════════════════════ -# DEPRECATED — Uptime Kuma was decommissioned on 2026-09-11. diff --git a/ansible/roles/datum_gateway/templates/config.json.j2 b/ansible/roles/datum_gateway/templates/config.json.j2 deleted file mode 100644 index e17a386..0000000 --- a/ansible/roles/datum_gateway/templates/config.json.j2 +++ /dev/null @@ -1,35 +0,0 @@ -{ - "bitcoind": { - "rpcuser": "{{ bitcoin_rpc_user }}", - "rpcpassword": "{{ bitcoin_rpc_password }}", - "rpcurl": "{{ datum_bitcoin_rpc_url }}", - "notify_fallback": true - }, - "stratum": { - "listen_port": {{ datum_gateway_stratum_port }}, - "vardiff_min": {{ datum_vardiff_min }} - }, - "mining": { - "pool_address": "{{ datum_mining_address }}", - "coinbase_tag_primary": "{{ datum_coinbase_tag_primary }}", - "coinbase_tag_secondary": "{{ datum_coinbase_tag_secondary }}" - }, - "api": { - "admin_password": "{{ datum_gateway_admin_password }}", - "listen_port": {{ datum_gateway_api_port }}, - "modify_conf": false - }, - "logger": { - "log_to_console": true, - "log_to_file": true, - "log_file": "{{ datum_gateway_log_dir }}/datum_gateway.log", - "log_rotate_daily": true, - "log_level_console": 2, - "log_level_file": 1 - }, - "datum": { - "pool_pass_workers": {{ datum_pool_pass_workers | lower }}, - "pool_pass_full_users": {{ datum_pool_pass_full_users | lower }}, - "pooled_mining_only": {{ datum_pooled_mining_only | lower }} - } -} diff --git a/ansible/roles/datum_gateway/templates/datum-gateway.service.j2 b/ansible/roles/datum_gateway/templates/datum-gateway.service.j2 deleted file mode 100644 index 18f7a3c..0000000 --- a/ansible/roles/datum_gateway/templates/datum-gateway.service.j2 +++ /dev/null @@ -1,22 +0,0 @@ -[Unit] -Description=DATUM Gateway - Bitcoin Mining Gateway -Documentation=https://github.com/OCEAN-xyz/datum_gateway -After=network.target bitcoind.service -Wants=bitcoind.service - -[Service] -User={{ datum_gateway_user }} -Group={{ datum_gateway_group }} -Type=simple -ExecStart={{ datum_gateway_bin_path }} --config {{ datum_gateway_config_dir }}/config.json -Restart=on-failure -RestartSec=10 -StandardOutput=journal -StandardError=journal - -# Prevent config from being read by other users -ReadWritePaths={{ datum_gateway_log_dir }} -ReadOnlyPaths={{ datum_gateway_config_dir }} - -[Install] -WantedBy=multi-user.target diff --git a/ansible/roles/datum_gateway/templates/healthcheck.service.j2 b/ansible/roles/datum_gateway/templates/healthcheck.service.j2 deleted file mode 100644 index e21672e..0000000 --- a/ansible/roles/datum_gateway/templates/healthcheck.service.j2 +++ /dev/null @@ -1,15 +0,0 @@ -[Unit] -Description=DATUM Gateway Health Check -After=network.target datum-gateway.service - -[Service] -Type=oneshot -User=root -ExecStart=/usr/local/bin/datum-gateway-healthcheck-push.sh -Environment=HEALTHCHECK_PUSH_URL={{ healthcheck_push_url }} -Environment=HEALTHCHECK_PUSH_TOKEN={{ healthcheck_push_token }} -StandardOutput=journal -StandardError=journal - -[Install] -WantedBy=multi-user.target diff --git a/ansible/roles/datum_gateway/templates/healthcheck.sh.j2 b/ansible/roles/datum_gateway/templates/healthcheck.sh.j2 deleted file mode 100644 index 43ec6b2..0000000 --- a/ansible/roles/datum_gateway/templates/healthcheck.sh.j2 +++ /dev/null @@ -1,39 +0,0 @@ -#!/bin/bash -# DATUM Gateway health check — managed by Ansible (roles/datum_gateway) -# -# The exit code is the answer and systemd keeps it: -# systemctl is-failed datum-gateway-healthcheck.service -# Reporting anywhere else is optional and generic. -PUSH_URL="${HEALTHCHECK_PUSH_URL:-}" -PUSH_TOKEN="${HEALTHCHECK_PUSH_TOKEN:-}" -STRATUM_PORT={{ datum_gateway_stratum_port }} - -check_datum() { - # Service must be active and stratum port must be listening - systemctl is-active --quiet datum-gateway && \ - nc -z 127.0.0.1 "${STRATUM_PORT}" -} - -report() { - local status=$1 - local msg=$2 - # No push URL is normal, not an error: the exit code below is still a - # complete answer for anything reading unit state. - [ -n "$PUSH_URL" ] || return 0 - # Gatus external endpoint: a POST with a bearer token, NOT Uptime Kuma's - # GET with ?status=up. The callers still pass up/down, so the mapping is - # done here rather than at every call site. - local _ok=false - [ "${status}" = "up" ] && _ok=true - curl -s --max-time 15 --retry 2 -o /dev/null -X POST \ - -H "Authorization: Bearer ${PUSH_TOKEN}" \ - "${PUSH_URL}?success=${_ok}&error=${msg// /%20}" || true -} - -if check_datum; then - report "up" "OK" - exit 0 -else - report "down" "DATUM Gateway not responding" - exit 1 -fi diff --git a/ansible/roles/datum_gateway/templates/healthcheck.timer.j2 b/ansible/roles/datum_gateway/templates/healthcheck.timer.j2 deleted file mode 100644 index c14d1d2..0000000 --- a/ansible/roles/datum_gateway/templates/healthcheck.timer.j2 +++ /dev/null @@ -1,10 +0,0 @@ -[Unit] -Description=DATUM Gateway Health Check Timer - -[Timer] -OnBootSec=2min -OnUnitActiveSec=1min -Persistent=true - -[Install] -WantedBy=timers.target diff --git a/ansible/roles/forgejo_runner/README.md b/ansible/roles/forgejo_runner/README.md deleted file mode 100644 index 6765e7f..0000000 --- a/ansible/roles/forgejo_runner/README.md +++ /dev/null @@ -1,156 +0,0 @@ -# `forgejo_runner` - -Installs and runs a Forgejo Actions runner, registers it with the Forgejo -instance, and keeps a health check on a systemd timer. - -Converted from `deploy_forgejo_runner_playbook.yml` (409 lines) under Plan 6. -The playbook is now 16 lines. - -## Phases - -`tasks/main.yml` imports five files in order: - -| | | -|---|---| -| `prerequisites.yml` | Docker must be present | -| `install.yml` | binary, system user, working directory | -| `configure.yml` | config file, registration with the instance | -| `service.yml` | systemd unit, start, assert it came up | -| `healthcheck.yml` | check script, unit, timer | - -`import_tasks`, not `include_tasks` — static imports are visible to -`--list-tasks`, which is how the conversion was verified against the playbook it -replaced. - -## Monitoring: one variable, no product knowledge - -This role contains **nothing specific to any monitoring system**. What used to -be here — an ~80-line embedded Python script creating monitors over the Uptime -Kuma API, a `/tmp` credentials file, token extraction, a systemd `Environment=` -rewrite, and 8 `when: uptime_kuma_enabled` guards — is gone. - -What remains answers the actual question, *is this service healthy*, and records -it two ways: - -- **the exit code**, which systemd keeps: `systemctl is-failed - forgejo-runner-healthcheck.service` is a complete answer with no monitoring - system involved at all; -- **a log file** at `{{ healthcheck_log_file }}`. - -To report health somewhere, set one variable: - -```yaml -healthcheck_push_url: "https://example/api/push/TOKEN" -``` - -Any endpoint accepting an HTTP ping works. Empty (the default) means check, log, -exit honestly, report nowhere — which is also the right setting for a *pull*-based -monitor like Prometheus' textfile collector, since that reads unit state instead. - -The push URL is a credential (anyone holding it can forge an "up"), so callers -pass it from the vault rather than committing it. - -## One behaviour change, deliberate - -`Assert runner is running` used to be guarded by `uptime_kuma_enabled`, so it -never ran. It is not a monitoring task — it is the deployment checking its own -work — and the deprecation banner swept it up by mistake. It is ungated here, -which means a runner that fails to start now fails the play instead of -deploying "successfully" in silence. - -## Docker-in-Docker for jobs - -`forgejo_runner_dind_enabled: true` gives Actions jobs a Docker daemon of their -own — a `docker:dind` container on this host's daemon, isolated from it — so -`docker build` and `docker compose` work inside a workflow. - -It follows Forgejo's own [Utilizing Docker within -Actions](https://forgejo.org/docs/latest/admin/actions/docker-access/) → -Docker-in-Docker, which is **three** parts, not one: - -| | what it does | where it applies | -|---|---|---| -| `forgejo-dind.service` | a `docker:dind` container, published on `127.0.0.1:2376` only | the host | -| `container.docker_host` | `tcp://127.0.0.1:2376` — the runner creates job/service/step containers **on dind** | the runner process | -| `runner.envs.DOCKER_HOST` + `container.options` | `tcp://forgejo-dind.docker.internal:2375`, resolved by `--add-host=…:host-gateway` | inside each job container | - -The third row is the one that is easy to miss, and leaving it out fails in a -way that looks identical to not having configured anything. `docker_host` only -decides **where the job container is created**; for a `tcp://` value Forgejo -mounts nothing into that container, so the job still has no daemon. The two -addresses differ because they are used from different places — the runner is a -process on the host and reaches dind on loopback; a job container is *inside* -dind, where `127.0.0.1` is itself. `host-gateway` is expanded by the daemon -that creates the container, which is dind, so it lands on dind's own bridge -gateway where its `dockerd` is listening. - -### What changes for jobs - -- Job containers, service containers and anything a job starts all live on the - dind daemon. A `postgres` service and a `docker compose` stack are siblings - there, so they reach each other by name — which is what makes end-to-end - tests work. -- **Nothing a workflow does can touch this host's Docker daemon any more.** - That is the point of dind over `automount`. -- dind has its own image store. Every image is pulled again the first time it - is needed. It is a named volume (`forgejo-dind-lib`), so it survives the - container being recreated, and `forgejo-dind-prune.timer` drops weekly what - has gone a week unused. -- Bind mounts inside a job's own `docker compose` resolve on the **dind** - filesystem, not the job container's. A compose file that bind-mounts the - checked-out repo will not see it. Named volumes and `build:` contexts are - fine. This is inherent to dind, not to this role. - -### Things to know before enabling - -- **This runner is shared.** `arbret`, `arbret-biz`, `arbretdos` and - `arbret-ops` build here too, and `container.docker_host` is per *runner*, not - per repo — every one of them moves onto dind. Their workflows do not change; - their next run pulls images into a cold cache. -- **`capacity` is 4 and dind is one daemon.** Concurrent jobs can see and touch - each other's containers and images through the Docker CLI. Forgejo's docs - call this out. Fine for a single-tenant runner; set `capacity: 1` if - untrusted code ever runs here. -- **The dind container is `--privileged`**, which is why this belongs on a - dedicated CI VM and not beside anything that matters. `forgejo-runner-box` - runs the runner and nothing else. -- **The daemon has no TLS and no authentication.** It is published on - `127.0.0.1` only, so reaching it means already having a shell on this box — - but it does mean any local user is root in a privileged container. Do not - move `forgejo_runner_dind_bind_address` off loopback. -- **The `runner` user is still in the host `docker` group**, left over from - when the runner used the host daemon. With dind it no longer needs it. Worth - removing, deliberately, as its own change. -- **`cache.host` is autodetected**, and job containers now reach the runner's - cache server across dind's NAT. Autodetect returns this box's LAN address, - which is routable from in there. If `actions/cache` starts silently missing, - pin `forgejo_runner_cache_host` — that is the first place to look. -- **Deploy when the runner is idle.** `shutdown_timeout` is 3h, so restarting - it mid-job waits for the job. - -### Turning it off - -Set `forgejo_runner_dind_enabled: false` and re-run. The role stops and -disables the units, removes them, and puts `container.docker_host` back to -`"-"` — the exact configuration this host ran before. The image and the -`forgejo-dind-lib` volume are left alone: they are a cache, they hold no -secrets, and deleting them on a toggle is not the role's call. - -## `config.yml` is templated now - -It used to be `forgejo-runner generate-config`, run once on first deploy and -never touched again — so the repo had no say in anything inside it, including -the setting this feature needs. It is now `templates/config.yml.j2`. - -Every value in it was transcribed from the file live on `forgejo-runner-box` -and diffed key-by-key against it: with dind disabled, 30 of 30 keys are -identical. Two deliberate differences: - -- `runner.envs` no longer carries `A_TEST_ENV_NAME_1` / `A_TEST_ENV_NAME_2`, - upstream's example placeholders, which were being injected into every job. -- the upstream comment blob is gone; the values are not. - -Because the file is managed, a change to it now restarts the runner -(`handlers/main.yml`). Previously a config change could not happen at all; the -failure mode where it lands on disk and the running process never picks it up -is one this repo has hit before. diff --git a/ansible/roles/forgejo_runner/defaults/main.yml b/ansible/roles/forgejo_runner/defaults/main.yml deleted file mode 100644 index 36f4bc1..0000000 --- a/ansible/roles/forgejo_runner/defaults/main.yml +++ /dev/null @@ -1,137 +0,0 @@ ---- -# Binary -forgejo_runner_version: "6.3.1" -forgejo_runner_arch: "linux-amd64" -forgejo_runner_url: "https://code.forgejo.org/forgejo/runner/releases/download/v{{ forgejo_runner_version }}/forgejo-runner-{{ forgejo_runner_version }}-{{ forgejo_runner_arch }}" -forgejo_runner_bin_path: "/usr/local/bin/forgejo-runner" - -# Runtime -forgejo_runner_user: "runner" -forgejo_runner_dir: "/opt/forgejo-runner" -forgejo_runner_config_path: "{{ forgejo_runner_dir }}/config.yml" -forgejo_runner_labels: "docker:docker://node:20-bookworm,ubuntu-latest:docker://node:20-bookworm,ubuntu-22.04:docker://node:20-bookworm,ubuntu-24.04:docker://node:20-bookworm" - -# --- config.yml contents ---------------------------------------------------- -# These reproduce, exactly, the stock `generate-config` output this host has -# been running since first deploy. They are variables now only because the file -# became a template; changing one of them changes the box. -forgejo_runner_log_level: info -forgejo_runner_log_job_level: info -forgejo_runner_capacity: 4 -forgejo_runner_job_timeout: "3h" -forgejo_runner_shutdown_timeout: "3h" -forgejo_runner_cache_enabled: true -# Empty = autodetect this host's outbound address. See the note in the template -# before changing it; under dind this is the value that decides whether -# actions/cache works. -forgejo_runner_cache_host: "" -forgejo_runner_container_network: "" -forgejo_runner_valid_volumes: [] -# Extra job environment and extra `docker run` options for job containers. -# Both MERGE with what dind adds rather than replacing it. -forgejo_runner_envs: {} -forgejo_runner_container_options: "" - -# The Forgejo instance this runner registers with. -forgejo_instance_url: "https://forgejo.contrapeso.xyz" -# forgejo_runner_registration_token comes from the vault. - -# --- Health check ----------------------------------------------------------- -# The check answers "is this service healthy" and records the answer two ways: -# a log file, and its own exit code. The exit code is the durable artefact — -# systemd stores it, so `systemctl is-failed forgejo-runner-healthcheck.service` -# answers the question with no monitoring system involved at all. -healthcheck_interval_seconds: 60 -healthcheck_script_dir: /opt/forgejo-runner-healthcheck -healthcheck_script_path: "{{ healthcheck_script_dir }}/forgejo_runner_healthcheck.sh" -healthcheck_log_file: "{{ healthcheck_script_dir }}/forgejo_runner_healthcheck.log" -healthcheck_service_name: forgejo-runner-healthcheck - -# WHERE TO REPORT HEALTH — the one place to plug in monitoring. -# -# Empty means "check, log, exit honestly, report nowhere". Set it to any URL -# that accepts an HTTP ping and the check will report there. Nothing in this -# role is specific to a particular monitoring product: the Uptime Kuma API -# calls, monitor creation and token handling that used to live here are gone. -# -# A pull-based monitor (Prometheus node_exporter textfile, say) needs this left -# empty — it reads the systemd unit state instead. -healthcheck_push_url: "" -# Bearer token for the Gatus external endpoint. Required whenever a push URL -# is set: Gatus rejects an unauthenticated push with 401. -healthcheck_push_token: "" - -# --- Docker-in-Docker for job containers ------------------------------------ -# Off by default. The role default is the SAFE state: `container.docker_host` -# stays "-", which is what this host has always run, and jobs get no Docker at -# all. services/forgejo-runner/deploy_forgejo_runner_playbook.yml turns it on. -# -# The shape is the one in Forgejo's own admin docs, "Utilizing Docker within -# Actions" -> Docker-in-Docker, and it is three moving parts, not one: -# -# 1. a docker:dind container on the HOST daemon, published on loopback only; -# 2. container.docker_host, which points the RUNNER PROCESS at that daemon, -# so job/service/step containers are created INSIDE dind; -# 3. runner.envs.DOCKER_HOST plus a container.options --add-host, which is -# what lets the JOB reach the same daemon from inside. -# -# Step 3 is not optional and not a duplicate of step 2. The two addresses are -# different because they are used from different places: the runner talks to -# 127.0.0.1 on the host, and a job container inside dind cannot - its own -# 127.0.0.1 is itself. Setting only docker_host gets you job containers that -# run in dind but still cannot find a daemon, which looks exactly like the bug -# we are fixing. See the README. -forgejo_runner_dind_enabled: false - -forgejo_runner_dind_container: forgejo-dind -# Already present on forgejo-runner-box, and the tag the project's own workflow -# refers to. A floating 28.x picks up patch releases on container recreation. -forgejo_runner_dind_image: "docker:28-dind" -forgejo_runner_dind_service: forgejo-dind - -# Loopback ONLY. This daemon has no TLS and no authentication, so anything that -# can open the port is root inside a privileged container. 127.0.0.1 keeps that -# to processes on this box; do not move it to 0.0.0.0 or the LAN address. -# -# The number is 2376 because the upstream docs use it. 2376 is conventionally -# the TLS port and this is PLAINTEXT - the port number promises nothing. -forgejo_runner_dind_bind_address: "127.0.0.1" -forgejo_runner_dind_host_port: 2376 -forgejo_runner_dind_container_port: 2375 - -# The name a JOB container uses to reach the dind daemon. Resolved by an -# --add-host ... :host-gateway on the job container, which Docker expands in -# the context of the daemon that CREATES the container - dind - so it lands on -# dind's own inner bridge gateway, where its dockerd is listening. That is why -# a name is used here and not an address: the address is only knowable from -# inside dind, at container-creation time. -forgejo_runner_dind_internal_host: "forgejo-dind.docker.internal" - -# Networks for the containers dind creates. act_runner makes a fresh network -# per job, so these multiply, and Docker's stock pool reaches into 192.168.x - -# the same space as this LAN. A job container that allocated 192.168.1.0/24 -# would lose the route to every box on it, including this one. Pinning the pool -# well clear of the LAN removes that entirely. -forgejo_runner_dind_address_pool: "172.20.0.0/14" -forgejo_runner_dind_address_pool_size: 24 - -# How long the dind unit waits for its daemon to answer before failing. Cold -# start pulls the image and initialises /var/lib/docker. -forgejo_runner_dind_ready_timeout: 180 - -# dind keeps its own /var/lib/docker. Without a volume it is wiped every time -# the container is recreated, and every image every workflow uses - the -# playwright image alone is 3.3GB - is pulled again from scratch. -forgejo_runner_dind_volume: forgejo-dind-lib - -# ...which then grows without bound, so it is pruned. `until=168h` keeps -# anything used in the last week, which is the cache that makes the volume -# worth having, and drops the rest. -# -# Volumes are deliberately NOT pruned: `until` does not apply to them, so -# --volumes here would be "delete every volume not currently attached", and a -# workflow's named volume between two jobs is not an obvious thing to destroy -# on a timer. Watch disk and prune those by hand. -forgejo_runner_dind_prune_enabled: true -forgejo_runner_dind_prune_on_calendar: "Sun 04:30" -forgejo_runner_dind_prune_filter: "until=168h" diff --git a/ansible/roles/forgejo_runner/handlers/main.yml b/ansible/roles/forgejo_runner/handlers/main.yml deleted file mode 100644 index e367495..0000000 --- a/ansible/roles/forgejo_runner/handlers/main.yml +++ /dev/null @@ -1,31 +0,0 @@ ---- -- name: Reload systemd - ansible.builtin.systemd: - daemon_reload: yes - -- name: Restart dind - ansible.builtin.systemd: - name: "{{ forgejo_runner_dind_service }}.service" - state: restarted - daemon_reload: yes - # In check mode the unit file was never written, so systemd has never heard - # of it and this fails on "Could not find the requested service" - an - # artefact of the dry run, not a problem with the run. Skipping it keeps - # `--check --diff` able to reach the config.yml diff, which is the part of a - # dry run worth reading. - when: not ansible_check_mode - -# The role had no handlers at all before, because it managed no config: the -# runner's config.yml was generated once on first deploy and never touched -# again, so there was nothing a change could need to restart. Now that -# config.yml is a template, a change to it that did NOT restart the runner -# would be the worst of both - the file on disk right and the running process -# still on the old settings. That failure mode has bitten this repo before. -# -# Note the runner's shutdown_timeout is 3h: a restart while a job is in flight -# waits for it, up to systemd's stop timeout. Deploy when the runner is idle. -- name: Restart forgejo-runner - ansible.builtin.systemd: - name: forgejo-runner - state: restarted - daemon_reload: yes diff --git a/ansible/roles/forgejo_runner/tasks/configure.yml b/ansible/roles/forgejo_runner/tasks/configure.yml deleted file mode 100644 index d7e7944..0000000 --- a/ansible/roles/forgejo_runner/tasks/configure.yml +++ /dev/null @@ -1,51 +0,0 @@ ---- -# config.yml is a TEMPLATE now, not a one-shot `generate-config`. -# -# Before this, the file was generated on first deploy and never touched again, -# so the repo had no say in anything inside it - including container.docker_host, -# which is the setting that decides whether jobs can use Docker at all. The -# alternative was patching keys in place with a regex, which in a YAML file -# with repeated key names under different parents is the kind of thing that -# works until it silently does not. -# -# owner/group/mode are runner:runner 0644, read off the live box, matching what -# `generate-config` plus the old chown produced. Nothing secret is in here; the -# registration token lives in .runner. -- name: Write the forgejo-runner configuration - ansible.builtin.template: - src: config.yml.j2 - dest: "{{ forgejo_runner_config_path }}" - owner: "{{ forgejo_runner_user }}" - group: "{{ forgejo_runner_user }}" - mode: '0644' - # A malformed config is a runner that will not start. Catch it here, while - # the old file is still in place, rather than after it has been replaced. - validate: "python3 -c \"import yaml,sys; yaml.safe_load(open(sys.argv[1]))\" %s" - notify: Restart forgejo-runner - -# ── Register runner ──────────────────────────────────────────────────────── -# Registration is separate from configuration and happens once. The labels are -# frozen into .runner at this point; changing forgejo_runner_labels later does -# not move them, which is why the label set is left exactly as it was. -- name: Check if runner is already registered - stat: - path: "{{ forgejo_runner_dir }}/.runner" - register: runner_stat - -- name: Register runner with Forgejo instance - command: > - {{ forgejo_runner_bin_path }} register --no-interactive - --instance {{ forgejo_instance_url }} - --token {{ forgejo_runner_registration_token }} - --name forgejo-runner-box - --labels "{{ forgejo_runner_labels }}" - args: - chdir: "{{ forgejo_runner_dir }}" - when: not runner_stat.stat.exists - -- name: Set runner registration file ownership - file: - path: "{{ forgejo_runner_dir }}/.runner" - owner: "{{ forgejo_runner_user }}" - group: "{{ forgejo_runner_user }}" - when: not runner_stat.stat.exists diff --git a/ansible/roles/forgejo_runner/tasks/dind.yml b/ansible/roles/forgejo_runner/tasks/dind.yml deleted file mode 100644 index f14c070..0000000 --- a/ansible/roles/forgejo_runner/tasks/dind.yml +++ /dev/null @@ -1,140 +0,0 @@ ---- -# The isolated Docker daemon jobs build against. -# -# Runs BEFORE configure.yml and service.yml on purpose: config.yml names this -# daemon's address, and forgejo-runner.service Requires= this unit. Creating it -# after either would mean one run where the runner points at nothing. - -# ── Disabled: undo it, do not just skip it ───────────────────────────────── -# A flag that only ever adds is not a flag. If dind is switched off the runner -# goes back to container.docker_host "-" a few tasks from now, and a dind still -# running would be a privileged container nothing uses. The volume and the -# image are deliberately left: they are the image cache, they hold no secrets, -# and destroying them on a toggle is not this role's call. -- name: Find the dind unit when dind is disabled - ansible.builtin.stat: - path: "/etc/systemd/system/{{ forgejo_runner_dind_service }}.service" - register: dind_unit_leftover - when: not forgejo_runner_dind_enabled - -- name: Stop and disable dind - ansible.builtin.systemd: - name: "{{ item }}" - state: stopped - enabled: no - loop: - - "{{ forgejo_runner_dind_service }}.service" - - "{{ forgejo_runner_dind_service }}-prune.timer" - failed_when: false - when: - - not forgejo_runner_dind_enabled - - dind_unit_leftover.stat.exists | default(false) - -- name: Remove the dind units - ansible.builtin.file: - path: "/etc/systemd/system/{{ item }}" - state: absent - loop: - - "{{ forgejo_runner_dind_service }}.service" - - "{{ forgejo_runner_dind_service }}-prune.service" - - "{{ forgejo_runner_dind_service }}-prune.timer" - when: not forgejo_runner_dind_enabled - notify: Reload systemd - -# ── Enabled ─────────────────────────────────────────────────────────────── -- name: Pull the dind image - ansible.builtin.command: "docker image pull {{ forgejo_runner_dind_image }}" - register: dind_pull - changed_when: "'Downloaded newer image' in dind_pull.stdout" - when: forgejo_runner_dind_enabled - -# Created explicitly rather than left to `docker run`, so that the name is -# declared here and a later `docker volume ls` has an obvious owner. -- name: Check for the dind image-store volume - ansible.builtin.command: "docker volume inspect {{ forgejo_runner_dind_volume }}" - register: dind_volume - changed_when: false - failed_when: false - check_mode: false - when: forgejo_runner_dind_enabled - -- name: Create the dind image-store volume - ansible.builtin.command: "docker volume create {{ forgejo_runner_dind_volume }}" - when: - - forgejo_runner_dind_enabled - - dind_volume.rc | default(1) != 0 - -- name: Create the dind systemd unit - ansible.builtin.template: - src: forgejo-dind.service.j2 - dest: "/etc/systemd/system/{{ forgejo_runner_dind_service }}.service" - owner: root - group: root - mode: '0644' - when: forgejo_runner_dind_enabled - notify: - - Reload systemd - - Restart dind - -- name: Create the dind prune service - ansible.builtin.template: - src: forgejo-dind-prune.service.j2 - dest: "/etc/systemd/system/{{ forgejo_runner_dind_service }}-prune.service" - owner: root - group: root - mode: '0644' - when: forgejo_runner_dind_enabled and forgejo_runner_dind_prune_enabled - notify: Reload systemd - -- name: Create the dind prune timer - ansible.builtin.template: - src: forgejo-dind-prune.timer.j2 - dest: "/etc/systemd/system/{{ forgejo_runner_dind_service }}-prune.timer" - owner: root - group: root - mode: '0644' - when: forgejo_runner_dind_enabled and forgejo_runner_dind_prune_enabled - notify: Reload systemd - -- name: Apply pending unit changes before starting dind - ansible.builtin.meta: flush_handlers - -- name: Enable and start dind - ansible.builtin.systemd: - name: "{{ forgejo_runner_dind_service }}.service" - enabled: yes - state: started - daemon_reload: yes - when: forgejo_runner_dind_enabled - -- name: Enable and start the dind prune timer - ansible.builtin.systemd: - name: "{{ forgejo_runner_dind_service }}-prune.timer" - enabled: yes - state: started - daemon_reload: yes - when: forgejo_runner_dind_enabled and forgejo_runner_dind_prune_enabled - -# The unit's own ExecStartPost already blocks until the daemon answers, so by -# here this should be instant. It is asserted anyway, from the same address the -# runner will use, because "the unit is active" and "the daemon on that port -# answers" are different claims and it is the second one the runner depends on. -- name: Verify the dind daemon answers on the address the runner will use - ansible.builtin.command: >- - docker -H tcp://{{ forgejo_runner_dind_bind_address }}:{{ forgejo_runner_dind_host_port }} - version --format {{ '{{.Server.Version}}' }} - register: dind_version - changed_when: false - retries: 10 - delay: 3 - until: dind_version.rc == 0 - when: forgejo_runner_dind_enabled and not ansible_check_mode - -- name: Show which daemon the jobs will build on - ansible.builtin.debug: - msg: >- - dind daemon {{ dind_version.stdout }} is up on - {{ forgejo_runner_dind_bind_address }}:{{ forgejo_runner_dind_host_port }} - (host daemon is untouched); jobs reach it as - tcp://{{ forgejo_runner_dind_internal_host }}:{{ forgejo_runner_dind_container_port }} - when: forgejo_runner_dind_enabled and not ansible_check_mode diff --git a/ansible/roles/forgejo_runner/tasks/healthcheck.yml b/ansible/roles/forgejo_runner/tasks/healthcheck.yml deleted file mode 100644 index 42fbaf7..0000000 --- a/ansible/roles/forgejo_runner/tasks/healthcheck.yml +++ /dev/null @@ -1,73 +0,0 @@ ---- -# Everything here answers "is the service healthy" and records the answer. -# The Uptime Kuma specifics that used to surround it — an embedded Python script -# that created monitors over the API, a /tmp credentials file, token extraction, -# and a systemd Environment= rewrite — are gone. What reports where is now one -# variable, healthcheck_push_url. See the role README. -- name: Create healthcheck script directory - ansible.builtin.file: - path: "{{ healthcheck_script_dir }}" - state: directory - owner: root - group: root - mode: '0755' - -- name: Create forgejo-runner healthcheck script - ansible.builtin.template: - src: healthcheck.sh.j2 - dest: "{{ healthcheck_script_path }}" - owner: root - group: root - mode: '0755' - validate: "bash -n %s" - -- name: Create healthcheck systemd service - ansible.builtin.template: - src: healthcheck.service.j2 - dest: "/etc/systemd/system/{{ healthcheck_service_name }}.service" - owner: root - group: root - mode: "0600" - -- name: Create healthcheck systemd timer - ansible.builtin.template: - src: healthcheck.timer.j2 - dest: "/etc/systemd/system/{{ healthcheck_service_name }}.timer" - owner: root - group: root - mode: '0644' - -- name: Reload systemd for healthcheck units - systemd: - daemon_reload: yes - -- name: Enable and start healthcheck timer - systemd: - name: "{{ healthcheck_service_name }}.timer" - enabled: yes - state: started - -- name: Test healthcheck script - command: "{{ healthcheck_script_path }}" - register: healthcheck_test - changed_when: false - -- name: Verify healthcheck script works - assert: - that: - - healthcheck_test.rc == 0 - fail_msg: "Healthcheck script failed to execute properly" - -- name: Display deployment summary - debug: - msg: | - Forgejo Runner deployed successfully! - - Runner Name: forgejo-runner-box - Instance: {{ forgejo_instance_url }} - Working Directory: {{ forgejo_runner_dir }} - Service: forgejo-runner.service ({{ runner_active.stdout }}) - - Healthcheck Monitor: {{ healthcheck_service_name }} - Healthcheck Interval: Every {{ healthcheck_interval_seconds }}s - Reporting to: {{ healthcheck_push_url | default('', true) | regex_replace('/api/push/.*', '/api/push/***') | default('(nowhere - set healthcheck_push_url)', true) }} diff --git a/ansible/roles/forgejo_runner/tasks/install.yml b/ansible/roles/forgejo_runner/tasks/install.yml deleted file mode 100644 index b95ab4c..0000000 --- a/ansible/roles/forgejo_runner/tasks/install.yml +++ /dev/null @@ -1,27 +0,0 @@ ---- -- name: Download forgejo-runner binary - get_url: - url: "{{ forgejo_runner_url }}" - dest: "{{ forgejo_runner_bin_path }}" - mode: '0755' - -# ── 3. Create runner system user ─────────────────────────────────── -- name: Create runner system user - user: - name: "{{ forgejo_runner_user }}" - system: yes - shell: /usr/sbin/nologin - home: "{{ forgejo_runner_dir }}" - create_home: no - groups: docker - append: yes - comment: 'Forgejo Runner' - -# ── 4. Create working directory ──────────────────────────────────── -- name: Create forgejo-runner working directory - file: - path: "{{ forgejo_runner_dir }}" - state: directory - owner: "{{ forgejo_runner_user }}" - group: "{{ forgejo_runner_user }}" - mode: '0750' diff --git a/ansible/roles/forgejo_runner/tasks/main.yml b/ansible/roles/forgejo_runner/tasks/main.yml deleted file mode 100644 index 5023e59..0000000 --- a/ansible/roles/forgejo_runner/tasks/main.yml +++ /dev/null @@ -1,12 +0,0 @@ ---- -# import_tasks, not include_tasks: these are unconditional phases, and static -# imports are visible to `--list-tasks`. That matters because the task list is -# how this refactor was verified against the playbook it replaced. -- ansible.builtin.import_tasks: prerequisites.yml -- ansible.builtin.import_tasks: install.yml -# Before configure.yml and service.yml: config.yml names the dind daemon and -# forgejo-runner.service Requires= its unit, so it has to exist by then. -- ansible.builtin.import_tasks: dind.yml -- ansible.builtin.import_tasks: configure.yml -- ansible.builtin.import_tasks: service.yml -- ansible.builtin.import_tasks: healthcheck.yml diff --git a/ansible/roles/forgejo_runner/tasks/prerequisites.yml b/ansible/roles/forgejo_runner/tasks/prerequisites.yml deleted file mode 100644 index d441709..0000000 --- a/ansible/roles/forgejo_runner/tasks/prerequisites.yml +++ /dev/null @@ -1,12 +0,0 @@ ---- -- name: Check if Docker is installed - command: docker --version - register: docker_check - changed_when: false - failed_when: docker_check.rc != 0 - -- name: Fail if Docker is not available - assert: - that: - - docker_check.rc == 0 - fail_msg: "Docker is required for forgejo-runner but is not installed" diff --git a/ansible/roles/forgejo_runner/tasks/service.yml b/ansible/roles/forgejo_runner/tasks/service.yml deleted file mode 100644 index 11e8784..0000000 --- a/ansible/roles/forgejo_runner/tasks/service.yml +++ /dev/null @@ -1,40 +0,0 @@ ---- -- name: Create forgejo-runner systemd service - ansible.builtin.template: - src: forgejo-runner.service.j2 - dest: /etc/systemd/system/forgejo-runner.service - owner: root - group: root - mode: '0644' - notify: Restart forgejo-runner - -- name: Reload systemd - systemd: - daemon_reload: yes - -- name: Enable and start forgejo-runner service - systemd: - name: forgejo-runner - enabled: yes - state: started - -# Before the assert below, not at the end of the play. A config change queues a -# restart; if that restart ran after the verification, the play would assert -# that the OLD process is healthy and then replace it unverified. -- name: Apply a pending restart before checking the runner - ansible.builtin.meta: flush_handlers - -- name: Verify forgejo-runner is active - command: systemctl is-active forgejo-runner - register: runner_active - changed_when: false - -# Ungated on purpose. This was previously guarded by `uptime_kuma_enabled`, but -# it is not a monitoring task — it is the deployment asserting its own success. -# The deprecation banner swept it up along with the Kuma plumbing, which meant a -# broken runner deployed "successfully" and silently. -- name: Assert runner is running - assert: - that: - - runner_active.stdout == "active" - fail_msg: "forgejo-runner service is not active: {{ runner_active.stdout }}" diff --git a/ansible/roles/forgejo_runner/templates/config.yml.j2 b/ansible/roles/forgejo_runner/templates/config.yml.j2 deleted file mode 100644 index b0a5c1e..0000000 --- a/ansible/roles/forgejo_runner/templates/config.yml.j2 +++ /dev/null @@ -1,122 +0,0 @@ -# Forgejo Runner configuration - MANAGED BY ANSIBLE (roles/forgejo_runner). -# Local edits are overwritten on the next run. Change the role instead. -# -# Until the Docker-in-Docker work this file was NOT managed: configure.yml ran -# `forgejo-runner generate-config` once, on first deploy, and never touched the -# result again. Setting container.docker_host meant either patching YAML in -# place with a regex - the fragile thing this repo has been bitten by before - -# or templating the file. So it is templated. -# -# Every value below was transcribed from the config.yml live on -# forgejo-runner-box, which was still stock generate-config output, so applying -# this with dind disabled is a no-op. Two exceptions, both deliberate: -# -# - runner.envs lost A_TEST_ENV_NAME_1 and A_TEST_ENV_NAME_2. They are -# upstream's example placeholders and were being injected into every job on -# this runner. Nothing can depend on them. -# - the upstream comment blob is gone. The values are unchanged. - -{# - Two values are assembled rather than written straight out, because each has - a dind half and a caller-supplied half and they have to merge, not compete. -#} -{% set _envs = dict(forgejo_runner_envs) %} -{% set _opts = [forgejo_runner_container_options] if forgejo_runner_container_options else [] %} -{% if forgejo_runner_dind_enabled %} -{% set _ = _envs.update({'DOCKER_HOST': 'tcp://' ~ forgejo_runner_dind_internal_host ~ ':' ~ forgejo_runner_dind_container_port}) %} -{% set _ = _opts.insert(0, '--add-host=' ~ forgejo_runner_dind_internal_host ~ ':host-gateway') %} -{% endif %} -log: - level: {{ forgejo_runner_log_level }} - job_level: {{ forgejo_runner_log_job_level }} - -runner: - file: .runner - # Concurrent tasks. NOTE, with dind enabled: all capacity slots share ONE - # Docker daemon, so two jobs running at once can see and touch each other's - # containers, images and volumes through the docker CLI. Forgejo's admin docs - # call this out. It is the accepted trade for a single shared runner; drop - # this to 1 if untrusted code ever runs here. - capacity: {{ forgejo_runner_capacity }} -{% if _envs %} - # Environment for every job. DOCKER_HOST is the half of the dind setup that - # the JOB uses: the runner exports it into each job, service and step - # container, so `docker build` and `docker compose` inside a job find the - # dind daemon instead of a socket that is not there. - envs: -{% for key, value in _envs | dictsort %} - {{ key }}: {{ value }} -{% endfor %} -{% else %} - envs: {} -{% endif %} - env_file: .env - timeout: {{ forgejo_runner_job_timeout }} - # How long a `systemctl stop/restart` waits for in-flight jobs. It is 3h, so - # redeploying while a job is running does NOT return promptly - see the - # README's note on when to deploy. - shutdown_timeout: {{ forgejo_runner_shutdown_timeout }} - insecure: false - fetch_timeout: 5s - fetch_interval: 2s - report_interval: 1s - # Empty: the labels in .runner, fixed at registration, are used. - labels: [] - -cache: - enabled: {{ 'true' if forgejo_runner_cache_enabled else 'false' }} - dir: "" - # The address job containers use to reach the runner's cache server, NOT the - # address it listens on. Empty means autodetect, which picks this host's - # outbound address. - # - # This is the value most likely to need attention under dind. Job containers - # now live inside dind and reach the host by NAT, so the address has to be - # one that is routable from in there - this box's LAN address is, and that is - # what autodetect returns today. If actions/cache starts silently missing - # after enabling dind, look here first: pin forgejo_runner_cache_host to - # {{ ansible_default_ipv4.address | default('this host LAN address') }}. - host: "{{ forgejo_runner_cache_host }}" - port: 0 - proxy_port: 0 - external_server: "" - secret: "" - actions_cache_url_override: "" - -container: - network: "{{ forgejo_runner_container_network }}" - enable_ipv6: false - # Stays false. The dind CONTAINER is privileged; job containers have no need - # to be, because the daemon they talk to is already running elsewhere. - privileged: false -{% if _opts %} - # --add-host resolves the dind hostname inside every job container. Docker - # expands `host-gateway` against the daemon that CREATES the container - - # which is dind - so it lands on dind's own bridge gateway, where its dockerd - # is listening. Without this the DOCKER_HOST above names something no job can - # resolve, and jobs fail exactly as if nothing had been configured. - options: '{{ _opts | join(" ") }}' -{% else %} - options: -{% endif %} - workdir_parent: - valid_volumes: {{ forgejo_runner_valid_volumes | to_json }} -{% if forgejo_runner_dind_enabled %} - # Create job, service and step containers on the dind daemon instead of this - # host's. Reached over the published loopback port, because the runner is a - # plain process on the host, not a container beside dind. - # - # Consequence worth knowing: NOTHING a workflow does now touches the host's - # Docker daemon. Images are pulled fresh into dind the first time each one is - # needed, and service containers declared by a job land in dind alongside the - # job container, which is what makes `docker compose` reach them by name. - docker_host: 'tcp://{{ forgejo_runner_dind_bind_address }}:{{ forgejo_runner_dind_host_port }}' -{% else %} - # "-" - find a daemon for the runner's own use, and do not expose it to jobs. - docker_host: '-' -{% endif %} - force_pull: false - force_rebuild: false - -host: - workdir_parent: diff --git a/ansible/roles/forgejo_runner/templates/forgejo-dind-prune.service.j2 b/ansible/roles/forgejo_runner/templates/forgejo-dind-prune.service.j2 deleted file mode 100644 index 571466e..0000000 --- a/ansible/roles/forgejo_runner/templates/forgejo-dind-prune.service.j2 +++ /dev/null @@ -1,16 +0,0 @@ -# MANAGED BY ANSIBLE - roles/forgejo_runner -[Unit] -Description=Prune unused images and containers inside the Forgejo dind daemon -After={{ forgejo_runner_dind_service }}.service -Requires={{ forgejo_runner_dind_service }}.service - -[Service] -Type=oneshot -# Inside the dind container, against dind's own daemon - NOT the host's. The -# host daemon still holds the dind image itself and is not touched here. -# -# Volumes are excluded on purpose; see forgejo_runner_dind_prune_filter in the -# role defaults for why a timer is the wrong place to delete those. -ExecStart=/usr/bin/docker exec {{ forgejo_runner_dind_container }} docker system prune --all --force --filter "{{ forgejo_runner_dind_prune_filter }}" -StandardOutput=journal -StandardError=journal diff --git a/ansible/roles/forgejo_runner/templates/forgejo-dind-prune.timer.j2 b/ansible/roles/forgejo_runner/templates/forgejo-dind-prune.timer.j2 deleted file mode 100644 index a2c2a7b..0000000 --- a/ansible/roles/forgejo_runner/templates/forgejo-dind-prune.timer.j2 +++ /dev/null @@ -1,12 +0,0 @@ -# MANAGED BY ANSIBLE - roles/forgejo_runner -[Unit] -Description=Weekly prune of the Forgejo dind image and container store - -[Timer] -OnCalendar={{ forgejo_runner_dind_prune_on_calendar }} -# The box may be off at 04:30 on a Sunday; catch up rather than skip a week. -Persistent=true -RandomizedDelaySec=900 - -[Install] -WantedBy=timers.target diff --git a/ansible/roles/forgejo_runner/templates/forgejo-dind.service.j2 b/ansible/roles/forgejo_runner/templates/forgejo-dind.service.j2 deleted file mode 100644 index 2caf419..0000000 --- a/ansible/roles/forgejo_runner/templates/forgejo-dind.service.j2 +++ /dev/null @@ -1,52 +0,0 @@ -# MANAGED BY ANSIBLE - roles/forgejo_runner -# -# The isolated Docker daemon that Forgejo Actions jobs use. It runs ON this -# host's daemon but is a separate daemon: nothing a workflow does here can see -# or touch the host's containers, images or volumes. -[Unit] -Description=Docker-in-Docker daemon for Forgejo Actions jobs -Documentation=https://forgejo.org/docs/latest/admin/actions/docker-access/ -After=docker.service -Requires=docker.service - -[Service] -Type=simple -Restart=always -RestartSec=10 -# dind pulls its image on first start and initialises /var/lib/docker; neither -# is fast on a cold volume, and a deadline here would just flap it. -TimeoutStartSec=0 - -# A container left behind by a hard kill would make `docker run --name` fail. -# Leading "-" so a clean start is not an error. -ExecStartPre=-/usr/bin/docker rm -f {{ forgejo_runner_dind_container }} - -ExecStart=/usr/bin/docker run --rm --name {{ forgejo_runner_dind_container }} \ - --privileged \ - --publish {{ forgejo_runner_dind_bind_address }}:{{ forgejo_runner_dind_host_port }}:{{ forgejo_runner_dind_container_port }} \ - --volume {{ forgejo_runner_dind_volume }}:/var/lib/docker \ - --env DOCKER_TLS_CERTDIR= \ - {{ forgejo_runner_dind_image }} \ - dockerd \ - --host=tcp://0.0.0.0:{{ forgejo_runner_dind_container_port }} \ - --host=unix:///var/run/docker.sock \ - --default-address-pool base={{ forgejo_runner_dind_address_pool }},size={{ forgejo_runner_dind_address_pool_size }} - -# Hold the unit in "activating" until the daemon actually answers, so that -# forgejo-runner.service - which Requires= and After= this one - is never -# started against a socket that is not up yet. The runner does not retry: with -# an explicit container.docker_host it fails the connection and the job with it. -# -# Bounded by `timeout` rather than a counting loop, so a dind that never comes -# up FAILS this unit instead of hanging the boot. That is also the thing that -# catches a bad dockerd flag: the container exits, this never answers, and the -# unit goes red rather than the runner quietly losing Docker. -ExecStartPost=/usr/bin/timeout {{ forgejo_runner_dind_ready_timeout }} /bin/sh -c 'until /usr/bin/docker -H tcp://{{ forgejo_runner_dind_bind_address }}:{{ forgejo_runner_dind_host_port }} version >/dev/null 2>&1; do sleep 1; done' - -ExecStop=/usr/bin/docker stop --time 30 {{ forgejo_runner_dind_container }} - -StandardOutput=journal -StandardError=journal - -[Install] -WantedBy=multi-user.target diff --git a/ansible/roles/forgejo_runner/templates/forgejo-runner.service.j2 b/ansible/roles/forgejo_runner/templates/forgejo-runner.service.j2 deleted file mode 100644 index 6c3e468..0000000 --- a/ansible/roles/forgejo_runner/templates/forgejo-runner.service.j2 +++ /dev/null @@ -1,32 +0,0 @@ -# MANAGED BY ANSIBLE - roles/forgejo_runner -[Unit] -Description=Forgejo Runner -Documentation=https://forgejo.org/docs/latest/admin/actions/ -{% if forgejo_runner_dind_enabled %} -# forgejo-dind is a hard dependency, not a nicety. With an explicit -# container.docker_host the runner does not fall back to anything: if that -# daemon is not answering, every job it picks up fails to create its container. -# Requires= also means a stopped dind stops the runner, so the failure is one -# red unit rather than a runner that looks healthy and fails every job. -# -# The dind unit stays in "activating" until its daemon answers, so After= here -# really does mean "after Docker is usable", not just "after the process -# forked". -After=docker.service {{ forgejo_runner_dind_service }}.service -Requires=docker.service {{ forgejo_runner_dind_service }}.service -{% else %} -After=docker.service -Requires=docker.service -{% endif %} - -[Service] -Type=simple -User={{ forgejo_runner_user }} -Group={{ forgejo_runner_user }} -WorkingDirectory={{ forgejo_runner_dir }} -ExecStart={{ forgejo_runner_bin_path }} daemon --config {{ forgejo_runner_config_path }} -Restart=on-failure -RestartSec=10 - -[Install] -WantedBy=multi-user.target diff --git a/ansible/roles/forgejo_runner/templates/healthcheck.service.j2 b/ansible/roles/forgejo_runner/templates/healthcheck.service.j2 deleted file mode 100644 index 173f42b..0000000 --- a/ansible/roles/forgejo_runner/templates/healthcheck.service.j2 +++ /dev/null @@ -1,15 +0,0 @@ -[Unit] -Description=Forgejo Runner Healthcheck -After=network.target - -[Service] -Type=oneshot -ExecStart={{ healthcheck_script_path }} -Environment=HEALTHCHECK_PUSH_URL={{ healthcheck_push_url }} -Environment=HEALTHCHECK_PUSH_TOKEN={{ healthcheck_push_token }} -User=root -StandardOutput=journal -StandardError=journal - -[Install] -WantedBy=multi-user.target diff --git a/ansible/roles/forgejo_runner/templates/healthcheck.sh.j2 b/ansible/roles/forgejo_runner/templates/healthcheck.sh.j2 deleted file mode 100644 index 181714c..0000000 --- a/ansible/roles/forgejo_runner/templates/healthcheck.sh.j2 +++ /dev/null @@ -1,71 +0,0 @@ -#!/bin/bash -# Forgejo Runner healthcheck — managed by Ansible (roles/forgejo_runner) -# -# Answers "is forgejo-runner healthy" and records it two ways: this log, and the -# exit code. The exit code is the durable artefact — systemd keeps it, so -# systemctl is-failed {{ healthcheck_service_name }}.service -# answers the question with no monitoring system involved. -# -# Reporting is optional and generic: if a push URL is configured it also pings -# it. Nothing here knows or cares which monitoring product is on the other end. - -LOG_FILE="{{ healthcheck_log_file }}" -# Read from the environment rather than templated in, so the unit file is the -# only place the token lives and the script is not secret. -PUSH_URL="${HEALTHCHECK_PUSH_URL:-}" -PUSH_TOKEN="${HEALTHCHECK_PUSH_TOKEN:-}" - -log_message() { - echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE" -} - -# Gatus external endpoint: a POST with a bearer token and success=true|false. -# -# This used to report ONLY success - it exited before pushing when the runner -# was down - so a failure was invisible until the heartbeat window expired. -# Reporting the failure is the whole point of having a check. -report() { - local ok="$1" msg="$2" - [ -n "$PUSH_URL" ] || return 0 - curl -s --max-time 15 --retry 2 -o /dev/null -X POST \ - -H "Authorization: Bearer ${PUSH_TOKEN}" \ - "${PUSH_URL}?success=${ok}&error=${msg// /%20}" || true -} - -main() { - if ! systemctl is-active --quiet forgejo-runner; then - log_message "ERROR: forgejo-runner is not active" - report false "forgejo-runner is not active" - exit 1 - fi -{% if forgejo_runner_dind_enabled %} - - # An active runner is no longer the whole answer. With container.docker_host - # pointing at dind, a dead dind means the runner keeps claiming jobs and - # fails every one of them at container creation - green unit, red CI. So - # check the daemon jobs actually build on, not just that its unit is up: - # the unit can be active while dockerd inside it has died. - if ! systemctl is-active --quiet {{ forgejo_runner_dind_service }}; then - log_message "ERROR: {{ forgejo_runner_dind_service }} is not active" - report false "{{ forgejo_runner_dind_service }} is not active" - exit 1 - fi - - if ! docker -H tcp://{{ forgejo_runner_dind_bind_address }}:{{ forgejo_runner_dind_host_port }} version >/dev/null 2>&1; then - log_message "ERROR: dind daemon not answering on {{ forgejo_runner_dind_bind_address }}:{{ forgejo_runner_dind_host_port }}" - report false "dind daemon not answering" - exit 1 - fi - - log_message "forgejo-runner is active and the dind daemon answers" - report true "active" - exit 0 -{% else %} - - log_message "forgejo-runner is active" - report true "active" - exit 0 -{% endif %} -} - -main diff --git a/ansible/roles/forgejo_runner/templates/healthcheck.timer.j2 b/ansible/roles/forgejo_runner/templates/healthcheck.timer.j2 deleted file mode 100644 index 4cba51a..0000000 --- a/ansible/roles/forgejo_runner/templates/healthcheck.timer.j2 +++ /dev/null @@ -1,11 +0,0 @@ -[Unit] -Description=Run Forgejo Runner Healthcheck every minute -Requires={{ healthcheck_service_name }}.service - -[Timer] -OnBootSec=30sec -OnUnitActiveSec={{ healthcheck_interval_seconds }}sec -Persistent=true - -[Install] -WantedBy=timers.target diff --git a/ansible/roles/fulcrum/README.md b/ansible/roles/fulcrum/README.md deleted file mode 100644 index ddefabb..0000000 --- a/ansible/roles/fulcrum/README.md +++ /dev/null @@ -1,65 +0,0 @@ -# `fulcrum` - -Deploys [Fulcrum](https://github.com/cculianu/Fulcrum), an Electrum server -indexing the Bitcoin Knots node, on `fulcrum-box`. The second play in the -calling playbook publishes its SSL port from the edge host via `socket_proxy`. - -Converted from `deploy_fulcrum_playbook.yml` (685 lines) under Plan 6. The -playbook is now 33 lines. - -## The index is the expensive thing - -`{{ fulcrum_db_dir }}` is ~192 GB and takes days to rebuild. Nothing in this role -touches it beyond `state: directory` with the ownership it already has -(`fulcrum:fulcrum 0755`). Restarting Fulcrum re-opens the database; it does not -reindex. - -## Three things this conversion fixed, all pre-existing - -**`bitcoind` pointed at the wrong machine.** The vars file carried -`bitcoin_rpc_host: "192.168.1.140"` commented "IP of knots_box_local", but `.140` -is **fulcrum-box itself** — knots-box is `.135`. The DHCP leases had reshuffled. -The live config had already been hand-corrected to `knots-box`; running the -playbook would have reverted it and broken indexing. Now addressed by Tailscale -name, like everything else in this repo. - -**The restart handler was inert.** It carried -`when: uptime_kuma_enabled | default(false)`, so the three tasks that notify it -(SSL certificate, `fulcrum.conf`, systemd unit) could not restart anything. A -configuration change would write to disk, report success, and silently never take -effect. Ungated. - -**`db_mem` was about to quadruple.** The role computes a share of RAM; on this -5931 MB host 75% is 4448 MB, leaving ~1.4 GB for the OS and Fulcrum's non-cache -memory. The live value had been hand-tuned to 2048. `fulcrum_db_mem_mb_override` -pins it. Note `set_fact` outranks role defaults, so the *calculation* has to -honour the override — pinning it in `defaults/` alone is silently ignored. - -## The health check timer, and how to read it - -The timer is `OnBootSec` + `OnUnitActiveSec` with no `OnCalendar`. That -combination has a failure mode worth knowing: `OnBootSec` is monotonic and -elapses once; `OnUnitActiveSec` schedules relative to the **service** last being -active. If the service does not run in a given boot, there is no reference to -schedule from and the timer sits `active` and `enabled` doing nothing. That is -exactly what had happened here — last trigger **2026-02-17**, seven months of no -health check, with every surface-level indicator green. - -Restarting the timer does not supply that reference; running the service does. -So the role runs the check once after enabling the timer, which is both the fix -and a smoke test. - -**Diagnosing this is easy to get wrong**: `NextElapseUSecRealtime` is always -empty for a monotonic timer, so it looks broken even when it is fine. Read -`NextElapseUSecMonotonic`, or just use `systemctl list-timers`. - -The timer also no longer carries `Requires=fulcrum.service`. On a timer that -means "stop watching when the watched thing stops", which is backwards for a -health check. - -## Monitoring: one variable, no product knowledge - -The check tests the Electrum TCP port and records the answer in its exit code, -which systemd keeps: `systemctl is-failed fulcrum-healthcheck.service`. To report -elsewhere set `healthcheck_push_url` to any endpoint accepting an HTTP ping. The -Uptime Kuma API calls, monitor creation and token handling are gone. diff --git a/ansible/roles/fulcrum/defaults/main.yml b/ansible/roles/fulcrum/defaults/main.yml deleted file mode 100644 index 3e97fa0..0000000 --- a/ansible/roles/fulcrum/defaults/main.yml +++ /dev/null @@ -1,81 +0,0 @@ -# Fulcrum Configuration Variables - -# Version - Pinned to specific release -fulcrum_version: "2.1.0" # Fulcrum version to install - -# Directories -fulcrum_db_dir: /mnt/fulcrum_data/fulcrum_db # Database directory (heavy data on special mount) -fulcrum_config_dir: /etc/fulcrum # Config file location (standard OS path) -fulcrum_lib_dir: /var/lib/fulcrum # Other data files (banner, etc.) on OS disk -fulcrum_binary_path: /usr/local/bin/Fulcrum - -# Network - Bitcoin RPC connection -# Bitcoin Knots is on a different host (knots_box_local) -# Using RPC user/password authentication (credentials from group_vars/all/vault.yml) -# Addressed by Tailscale name, never a LAN IP. This was -# bitcoin_rpc_host: "192.168.1.140" # IP of knots_box_local -# but .140 is fulcrum-box ITSELF - knots-box is .135. The DHCP leases had -# reshuffled (the same drift that transposed the inventory), so running this -# playbook would have pointed Fulcrum at itself and broken indexing. The live -# config had already been hand-corrected to knots-box; this makes the repo -# agree with it. -bitcoin_rpc_host: "knots-box" -bitcoin_rpc_port: 8332 # Bitcoin Knots RPC port -# Note: bitcoin_rpc_user and bitcoin_rpc_password are loaded from group_vars/all/vault.yml - -# Network - Fulcrum server -fulcrum_tcp_port: 50001 -# The edge host's socket-proxy/Caddy play needs this too, and a role default is -# invisible outside this role. The authoritative value for the live deployment is -# in host_vars/fulcrum_box_local/main.yml, which outranks this; the value here is the -# protocol standard, so the role still works standalone. -fulcrum_ssl_port: 50002 -# Binding address for Fulcrum TCP/SSL server: -# - "127.0.0.1" = localhost only (use when Caddy is on the same box) -# - "0.0.0.0" = all interfaces (use when Caddy is on a different box) -# - Specific IP = bind to specific network interface -fulcrum_tcp_bind: "0.0.0.0" # Default: localhost (change to "0.0.0.0" if Caddy is on different box) -fulcrum_ssl_bind: "0.0.0.0" # Binding address for SSL port -# If Caddy is on a different box, set this to the IP address that Caddy will use to connect - -# SSL/TLS Configuration -fulcrum_ssl_enabled: true -fulcrum_ssl_cert_path: "{{ fulcrum_config_dir }}/fulcrum.crt" -fulcrum_ssl_key_path: "{{ fulcrum_config_dir }}/fulcrum.key" -fulcrum_ssl_cert_days: 3650 # 10 years validity for self-signed cert - - -# Performance -# db_mem will be calculated as 75% of available RAM automatically in playbook -# db_mem is computed as this share of RAM unless fulcrum_db_mem_mb is set -# explicitly. On a 5931 MB host 75% is 4448 MB, which leaves ~1.4 GB for the -# OS and for Fulcrum's non-cache memory; the live config had been hand-tuned -# down to 2048 and that setting is respected below. -fulcrum_db_mem_percent: 0.75 # 75% of RAM for database cache - -# Configuration options -fulcrum_anon_logs: true # Anonymize client IPs and TxIDs in logs -fulcrum_peering: false # Disable peering with other Fulcrum servers -fulcrum_zmq_allow_hashtx: true # Allow ZMQ hashtx notifications - -# Service user -fulcrum_user: fulcrum -fulcrum_group: fulcrum - - -# --- Health check ----------------------------------------------------------- -# Checks the Electrum TCP port and records the answer in its exit code, which -# systemd keeps: `systemctl is-failed fulcrum-healthcheck.service`. -# -# WHERE TO REPORT HEALTH — the one place to plug in monitoring. Empty means -# check, exit honestly, report nowhere. Any endpoint accepting an HTTP ping -# works; nothing here is specific to a monitoring product. -healthcheck_push_url: "" -# Bearer token for the Gatus external endpoint. Required whenever a push URL -# is set: Gatus rejects an unauthenticated push with 401. -healthcheck_push_token: "" - -# Explicit db_mem in MB. When set it wins over fulcrum_db_mem_percent; empty -# means compute from RAM. Set here because the live host had been hand-tuned to -# 2048 and a silent jump to 4448 is not something a refactor should do. -fulcrum_db_mem_mb_override: 2048 diff --git a/ansible/roles/fulcrum/handlers/main.yml b/ansible/roles/fulcrum/handlers/main.yml deleted file mode 100644 index c0d9cc4..0000000 --- a/ansible/roles/fulcrum/handlers/main.yml +++ /dev/null @@ -1,15 +0,0 @@ ---- -# Ungated on purpose. The hand-written handler carried -# when: uptime_kuma_enabled | default(false) -# so it has been inert since the decommissioning: three tasks notify it (the SSL -# certificate, fulcrum.conf and the systemd unit), and none of them could -# actually restart Fulcrum. A configuration change therefore applied to disk and -# silently never took effect — the worst kind of quiet failure, because the -# playbook reports success and the running service keeps its old settings. -# -# Restarting Fulcrum re-opens its database; it does not reindex. -- name: Restart fulcrum - systemd: - name: fulcrum - state: restarted - daemon_reload: yes diff --git a/ansible/roles/fulcrum/tasks/healthcheck.yml b/ansible/roles/fulcrum/tasks/healthcheck.yml deleted file mode 100644 index 983bffb..0000000 --- a/ansible/roles/fulcrum/tasks/healthcheck.yml +++ /dev/null @@ -1,62 +0,0 @@ ---- -# Everything here answers "is Fulcrum healthy" and records the answer. The -# Uptime Kuma specifics that used to follow — an embedded Python script creating -# monitors over the API, a /tmp credentials file, push-URL extraction and a -# systemd Environment= rewrite — are gone. Where it reports is now one variable, -# healthcheck_push_url. See the role README. -- name: Create Fulcrum health check script - ansible.builtin.template: - src: healthcheck.sh.j2 - dest: /usr/local/bin/fulcrum-healthcheck-push.sh - owner: root - group: root - mode: '0755' - validate: "bash -n %s" - -- name: Create systemd service for Fulcrum health check - ansible.builtin.template: - src: healthcheck.service.j2 - dest: /etc/systemd/system/fulcrum-healthcheck.service - owner: root - group: root - mode: "0600" - -- name: Create systemd timer for Fulcrum health check - ansible.builtin.template: - src: healthcheck.timer.j2 - dest: /etc/systemd/system/fulcrum-healthcheck.timer - owner: root - group: root - mode: '0644' - -- name: Reload systemd daemon for health check - systemd: - daemon_reload: yes - -# state: restarted, not started. The hand-written timer had got itself stuck -# `active` with no next elapse and had not fired since 2026-02-17; `started` on -# an already-active timer is a no-op and would have left it stuck. Restarting -# re-arms it. See the note in healthcheck.timer.j2. -- name: Enable and restart the Fulcrum health check timer - systemd: - name: fulcrum-healthcheck.timer - enabled: yes - state: restarted - daemon_reload: yes - -# Run the check once, which is both a smoke test and the thing that actually -# arms the timer. -# -# This timer is OnBootSec + OnUnitActiveSec with no OnCalendar. OnBootSec is -# monotonic and had long since elapsed; OnUnitActiveSec schedules relative to the -# SERVICE last being active, and the service had not run since 2026-02-17 — so -# there was no reference to schedule from and the timer sat `active` and -# `enabled` with NextElapseUSecMonotonic=infinity. Restarting the timer alone -# does not supply that reference; running the service does. -# -# (Diagnosing this is easy to get wrong: NextElapseUSecRealtime is always empty -# for a monotonic timer, so it looks broken even when it is fine. Read -# NextElapseUSecMonotonic, or just use `systemctl list-timers`.) -- name: Run the Fulcrum health check once to arm the timer - command: systemctl start fulcrum-healthcheck.service - changed_when: false diff --git a/ansible/roles/fulcrum/tasks/install.yml b/ansible/roles/fulcrum/tasks/install.yml deleted file mode 100644 index 1216a96..0000000 --- a/ansible/roles/fulcrum/tasks/install.yml +++ /dev/null @@ -1,146 +0,0 @@ ---- -- name: Calculate db_mem as a share of system RAM - set_fact: - fulcrum_db_mem_mb: "{{ (ansible_memtotal_mb | float * fulcrum_db_mem_percent) | int }}" - when: fulcrum_db_mem_mb_override | string | length == 0 - -- name: Use the explicit db_mem override - set_fact: - fulcrum_db_mem_mb: "{{ fulcrum_db_mem_mb_override }}" - when: fulcrum_db_mem_mb_override | string | length > 0 - changed_when: false - -- name: Display calculated db_mem value - debug: - msg: "Setting db_mem to {{ fulcrum_db_mem_mb }} MB ({{ (fulcrum_db_mem_percent * 100) | int }}% of {{ ansible_memtotal_mb }} MB total RAM)" - -- name: Display Fulcrum version to install - debug: - msg: "Installing Fulcrum version {{ fulcrum_version }}" - -- name: Install required packages - apt: - name: - - curl - - wget - - openssl - state: present - update_cache: yes - -- name: Create fulcrum group - group: - name: "{{ fulcrum_group }}" - system: yes - state: present - -- name: Create fulcrum user - user: - name: "{{ fulcrum_user }}" - group: "{{ fulcrum_group }}" - system: yes - shell: /usr/sbin/nologin - home: /home/{{ fulcrum_user }} - create_home: yes - state: present - -- name: Create Fulcrum database directory (heavy data on special mount) - file: - path: "{{ fulcrum_db_dir }}" - state: directory - owner: "{{ fulcrum_user }}" - group: "{{ fulcrum_group }}" - mode: '0755' - -- name: Create Fulcrum config directory - file: - path: "{{ fulcrum_config_dir }}" - state: directory - owner: root - group: "{{ fulcrum_group }}" - mode: '0755' - -- name: Create Fulcrum lib directory (for banner and other data files) - file: - path: "{{ fulcrum_lib_dir }}" - state: directory - owner: "{{ fulcrum_user }}" - group: "{{ fulcrum_group }}" - mode: '0755' - -# =========================================== -# SSL Certificate Generation -# =========================================== -- name: Check if SSL certificate already exists - stat: - path: "{{ fulcrum_ssl_cert_path }}" - register: fulcrum_ssl_cert_exists - when: fulcrum_ssl_enabled | default(false) - -- name: Generate self-signed SSL certificate for Fulcrum - command: > - openssl req -x509 -newkey rsa:4096 - -keyout {{ fulcrum_ssl_key_path }} - -out {{ fulcrum_ssl_cert_path }} - -sha256 -days {{ fulcrum_ssl_cert_days }} - -nodes - -subj "/C=XX/ST=Decentralized/L=Bitcoin/O=Fulcrum/OU=Electrum/CN=fulcrum.local" - args: - creates: "{{ fulcrum_ssl_cert_path }}" - when: fulcrum_ssl_enabled | default(false) - notify: Restart fulcrum - -- name: Set SSL certificate permissions - file: - path: "{{ fulcrum_ssl_cert_path }}" - owner: "{{ fulcrum_user }}" - group: "{{ fulcrum_group }}" - mode: '0644' - when: fulcrum_ssl_enabled | default(false) and fulcrum_ssl_cert_exists.stat.exists | default(false) or fulcrum_ssl_enabled | default(false) - -- name: Set SSL key permissions - file: - path: "{{ fulcrum_ssl_key_path }}" - owner: "{{ fulcrum_user }}" - group: "{{ fulcrum_group }}" - mode: '0600' - when: fulcrum_ssl_enabled | default(false) - -- name: Check if Fulcrum binary already exists - stat: - path: "{{ fulcrum_binary_path }}" - register: fulcrum_binary_exists - changed_when: false - -- name: Download Fulcrum binary tarball - get_url: - url: "https://github.com/cculianu/Fulcrum/releases/download/v{{ fulcrum_version }}/Fulcrum-{{ fulcrum_version }}-x86_64-linux.tar.gz" - dest: "/tmp/Fulcrum-{{ fulcrum_version }}-x86_64-linux.tar.gz" - mode: '0644' - when: not fulcrum_binary_exists.stat.exists - -- name: Extract Fulcrum binary - unarchive: - src: "/tmp/Fulcrum-{{ fulcrum_version }}-x86_64-linux.tar.gz" - dest: "/tmp" - remote_src: yes - when: not fulcrum_binary_exists.stat.exists - -- name: Install Fulcrum binary - copy: - src: "/tmp/Fulcrum-{{ fulcrum_version }}-x86_64-linux/Fulcrum" - dest: "{{ fulcrum_binary_path }}" - owner: root - group: root - mode: '0755' - remote_src: yes - when: not fulcrum_binary_exists.stat.exists - -- name: Verify Fulcrum binary installation - command: "{{ fulcrum_binary_path }} --version" - register: fulcrum_version_check - changed_when: false - -- name: Display Fulcrum version - debug: - msg: "{{ fulcrum_version_check.stdout_lines }}" - diff --git a/ansible/roles/fulcrum/tasks/main.yml b/ansible/roles/fulcrum/tasks/main.yml deleted file mode 100644 index bc7ff05..0000000 --- a/ansible/roles/fulcrum/tasks/main.yml +++ /dev/null @@ -1,6 +0,0 @@ ---- -# import_tasks, not include_tasks: static imports stay visible to --list-tasks, -# which is how this conversion was verified against the playbook it replaced. -- ansible.builtin.import_tasks: install.yml -- ansible.builtin.import_tasks: service.yml -- ansible.builtin.import_tasks: healthcheck.yml diff --git a/ansible/roles/fulcrum/tasks/service.yml b/ansible/roles/fulcrum/tasks/service.yml deleted file mode 100644 index 716b44d..0000000 --- a/ansible/roles/fulcrum/tasks/service.yml +++ /dev/null @@ -1,54 +0,0 @@ ---- -- name: Create Fulcrum banner file - ansible.builtin.template: - src: banner.txt.j2 - dest: "{{ fulcrum_lib_dir }}/fulcrum-banner.txt" - owner: "{{ fulcrum_user }}" - group: "{{ fulcrum_group }}" - mode: '0644' - -- name: Create Fulcrum configuration file - ansible.builtin.template: - src: fulcrum.conf.j2 - dest: "{{ fulcrum_config_dir }}/fulcrum.conf" - owner: "{{ fulcrum_user }}" - group: "{{ fulcrum_group }}" - mode: '0640' - notify: Restart fulcrum - -- name: Create systemd service file for Fulcrum - ansible.builtin.template: - src: fulcrum.service.j2 - dest: /etc/systemd/system/fulcrum.service - owner: root - group: root - mode: '0644' - notify: Restart fulcrum - -- name: Reload systemd daemon - systemd: - daemon_reload: yes - -- name: Enable and start Fulcrum service - systemd: - name: fulcrum - enabled: yes - state: started - -- name: Wait for Fulcrum to start - wait_for: - port: "{{ fulcrum_tcp_port }}" - host: "{{ fulcrum_tcp_bind }}" - delay: 5 - timeout: 30 - ignore_errors: yes - -- name: Check Fulcrum service status - systemd: - name: fulcrum - register: fulcrum_service_status - changed_when: false - -- name: Display Fulcrum service status - debug: - msg: "Fulcrum service is {{ 'running' if fulcrum_service_status.status.ActiveState == 'active' else 'not running' }}" diff --git a/ansible/roles/fulcrum/templates/banner.txt.j2 b/ansible/roles/fulcrum/templates/banner.txt.j2 deleted file mode 100644 index 1ce4ced..0000000 --- a/ansible/roles/fulcrum/templates/banner.txt.j2 +++ /dev/null @@ -1,3 +0,0 @@ -counterinfra - -PER ASPERA AD ASTRA diff --git a/ansible/roles/fulcrum/templates/fulcrum.conf.j2 b/ansible/roles/fulcrum/templates/fulcrum.conf.j2 deleted file mode 100644 index 69f072c..0000000 --- a/ansible/roles/fulcrum/templates/fulcrum.conf.j2 +++ /dev/null @@ -1,29 +0,0 @@ -# Fulcrum Configuration -# Generated by Ansible - -# Bitcoin Core/Knots RPC settings -bitcoind = {{ bitcoin_rpc_host }}:{{ bitcoin_rpc_port }} -rpcuser = {{ bitcoin_rpc_user }} -rpcpassword = {{ bitcoin_rpc_password }} - -# Fulcrum server general settings -datadir = {{ fulcrum_db_dir }} -tcp = {{ fulcrum_tcp_bind }}:{{ fulcrum_tcp_port }} -peering = {{ 'true' if fulcrum_peering else 'false' }} -zmq_allow_hashtx = {{ 'true' if fulcrum_zmq_allow_hashtx else 'false' }} - -# SSL/TLS Configuration -{% if fulcrum_ssl_enabled | default(false) %} -ssl = {{ fulcrum_ssl_bind }}:{{ fulcrum_ssl_port }} -cert = {{ fulcrum_ssl_cert_path }} -key = {{ fulcrum_ssl_key_path }} -{% endif %} - -# Anonymize client IP addresses and TxIDs in logs -anon_logs = {{ 'true' if fulcrum_anon_logs else 'false' }} - -# Max RocksDB Memory in MiB -db_mem = {{ fulcrum_db_mem_mb }}.0 - -# Banner -banner = {{ fulcrum_lib_dir }}/fulcrum-banner.txt diff --git a/ansible/roles/fulcrum/templates/fulcrum.service.j2 b/ansible/roles/fulcrum/templates/fulcrum.service.j2 deleted file mode 100644 index 02f4aa5..0000000 --- a/ansible/roles/fulcrum/templates/fulcrum.service.j2 +++ /dev/null @@ -1,24 +0,0 @@ -# MiniBolt: systemd unit for Fulcrum -# /etc/systemd/system/fulcrum.service - -[Unit] -Description=Fulcrum -After=network.target - -StartLimitBurst=2 -StartLimitIntervalSec=20 - -[Service] -ExecStart={{ fulcrum_binary_path }} {{ fulcrum_config_dir }}/fulcrum.conf - -User={{ fulcrum_user }} -Group={{ fulcrum_group }} - -# Process management -#################### -Type=simple -KillSignal=SIGINT -TimeoutStopSec=300 - -[Install] -WantedBy=multi-user.target diff --git a/ansible/roles/fulcrum/templates/healthcheck.service.j2 b/ansible/roles/fulcrum/templates/healthcheck.service.j2 deleted file mode 100644 index b519808..0000000 --- a/ansible/roles/fulcrum/templates/healthcheck.service.j2 +++ /dev/null @@ -1,15 +0,0 @@ -[Unit] -Description=Fulcrum Health Check -After=network.target fulcrum.service - -[Service] -Type=oneshot -User=root -ExecStart=/usr/local/bin/fulcrum-healthcheck-push.sh -Environment=HEALTHCHECK_PUSH_URL={{ healthcheck_push_url }} -Environment=HEALTHCHECK_PUSH_TOKEN={{ healthcheck_push_token }} -StandardOutput=journal -StandardError=journal - -[Install] -WantedBy=multi-user.target diff --git a/ansible/roles/fulcrum/templates/healthcheck.sh.j2 b/ansible/roles/fulcrum/templates/healthcheck.sh.j2 deleted file mode 100644 index 4449fed..0000000 --- a/ansible/roles/fulcrum/templates/healthcheck.sh.j2 +++ /dev/null @@ -1,41 +0,0 @@ -#!/bin/bash -# Fulcrum health check — managed by Ansible (roles/fulcrum) -# -# Checks that Fulcrum's Electrum TCP port is accepting connections, and records -# the answer in the exit code, which systemd keeps: -# systemctl is-failed fulcrum-healthcheck.service -# That is a complete answer with no monitoring system involved. Reporting -# elsewhere is optional and generic — set healthcheck_push_url. - -FULCRUM_HOST="{{ fulcrum_tcp_bind }}" -FULCRUM_PORT={{ fulcrum_tcp_port }} -PUSH_URL="${HEALTHCHECK_PUSH_URL:-}" -PUSH_TOKEN="${HEALTHCHECK_PUSH_TOKEN:-}" - -check_fulcrum() { - timeout 5 bash -c "echo > /dev/tcp/${FULCRUM_HOST}/${FULCRUM_PORT}" 2>/dev/null -} - -report() { - local status=$1 msg=$2 - # No push URL is normal, not an error: the exit code below is still a - # complete answer for anything reading unit state. - [ -n "$PUSH_URL" ] || return 0 - # Gatus external endpoint: a POST with a bearer token, NOT Uptime Kuma's - # GET with ?status=up. The callers still pass up/down, so the mapping is - # done here rather than at every call site. - local _ok=false - [ "${status}" = "up" ] && _ok=true - curl -s --max-time 15 --retry 2 -o /dev/null -X POST \ - -H "Authorization: Bearer ${PUSH_TOKEN}" \ - "${PUSH_URL}?success=${_ok}&error=${msg// /%20}" || true -} - -if check_fulcrum; then - report "up" "OK" - exit 0 -else - echo "Fulcrum TCP port ${FULCRUM_PORT} not responding" - report "down" "Fulcrum TCP port not responding" - exit 1 -fi diff --git a/ansible/roles/fulcrum/templates/healthcheck.timer.j2 b/ansible/roles/fulcrum/templates/healthcheck.timer.j2 deleted file mode 100644 index df16ff3..0000000 --- a/ansible/roles/fulcrum/templates/healthcheck.timer.j2 +++ /dev/null @@ -1,17 +0,0 @@ -[Unit] -Description=Fulcrum Health Check Timer -# NOTE: this deliberately does NOT carry `Requires=fulcrum.service`, which the -# hand-written unit had. Requires on a timer means the timer is stopped when the -# required unit stops — i.e. "if the thing I am watching goes down, stop -# watching it", which is backwards for a health check and leaves nothing to -# re-arm the timer when the service returns. The live timer had been `active` -# and `enabled` with NextElapseUSecMonotonic=infinity and a last trigger of -# 2026-02-17: seven months with no health check and no outward sign of it. - -[Timer] -OnBootSec=1min -OnUnitActiveSec=1min -Persistent=true - -[Install] -WantedBy=timers.target diff --git a/ansible/roles/gatus/README.md b/ansible/roles/gatus/README.md deleted file mode 100644 index 129057c..0000000 --- a/ansible/roles/gatus/README.md +++ /dev/null @@ -1,110 +0,0 @@ -# gatus - -Deploys [Gatus](https://github.com/TwiN/gatus) — health checks, a status page, -and alerting — **as the upstream container image**, on the -`observability` group. - -## Why the container - -Upstream publishes **no binary release assets**. The image is the only artefact -they ship and therefore the only one they test, so it is what `docker run` in -their README gets you, and it is what this role deploys. - -Building from source is entirely possible — their Dockerfile is a bare -`CGO_ENABLED=0 go build`, the Vue dashboard is compiled in via `//go:embed -static` in `web/static.go`, and the sqlite driver is pure-Go -`modernc.org/sqlite` so nothing needs linking. This role did that at first. The -reasons it doesn't now: - -- It produces a binary upstream never ran. -- It means compiling the AWS SDK, gRPC and the Google API libraries on the - smallest box in the estate. On this VPS that load was heavy enough that - unrelated Ansible tasks timed out while it ran. - -The cost of the container is a daemon on the machine whose job is to notice -when everything else breaks. That is a real trade, made deliberately. - -## Pinned by digest, not by tag - -```yaml -gatus_image_digest: "sha256:c5f210d0…" -gatus_image: "ghcr.io/twin/gatus@{{ gatus_image_digest }}" -``` - -A tag is mutable — `v5.36.0` can be repushed — so pinning the tag alone is a -weaker promise than it looks. The digest is a content address: if it resolves, -it is byte-for-byte the image reviewed here, and `docker compose pull` either -fetches exactly that or fails. `gatus_version` is kept alongside it purely so a -human can read which release it is; **the two must be updated together.** - -## What `FROM scratch` means for running it - -The image has no `/etc/passwd`, so there is no user to drop to by name and the -default is root. The compose file runs it by numeric id (`10001:10001`) and the -data directory on the host is owned to match. Everything else is locked down to -approximate what the systemd unit used to do natively: - -| systemd | compose | -|---|---| -| `ProtectSystem=strict` | `read_only: true` | -| `NoNewPrivileges=true` | `security_opt: [no-new-privileges:true]` | -| `CapabilityBoundingSet=` | `cap_drop: [ALL]` | -| `AmbientCapabilities=CAP_NET_RAW` | `cap_add: [NET_RAW]` (for `icmp://`) | - -## Configuration is a directory, not a file - -`GATUS_CONFIG_PATH` points at `/opt/gatus/config`, and Gatus merges every `*.yaml` -underneath it — maps deep-merge, lists append. This role owns exactly one file: - -``` -/opt/gatus/config/00-base.yaml web, storage, ui, alerting, security (this role) -/opt/gatus/config/endpoints/*.yaml one file per service (gatus_endpoint) -``` - -**A primitive defined in two files is ambiguous and upstream refuses it.** So -anything that is not a list belongs in `00-base.yaml` and nowhere else. Endpoints -are lists, so each service's file appends cleanly — the same shape as -`caddy_site`, where each service contributes its own vhost. - -## Pull and push - -Gatus polls. For anything with a reachable HTTP or TCP surface that is the -better check, because it tests the path a user actually takes. The monitoring -host joins the headscale mesh via `infra/920`, so internal boxes are reachable -by MagicDNS name and can be polled directly rather than having to report in. - -For state with no pollable surface — ZFS pool health, UPS mains status, disk -usage, backup freshness — Gatus has **external endpoints**, a push API: - -``` -POST /api/v1/endpoints/{group}_{name}/external?success=true&error=&duration= -Authorization: Bearer -``` - -with `heartbeat.interval` to alert when nothing reports in. That is the same -shape as the generic `healthcheck_push_url` already wired into every service -role, so those scripts need a URL, a POST, and an auth header — not a rewrite. - -## Variables - -See `defaults/main.yml`. The ones that matter: - -| Variable | Default | Note | -|---|---|---| -| `gatus_version` / `gatus_image_digest` | `v5.36.0` / `sha256:c5f210d0…` | must move together | -| `gatus_bind_address` | `127.0.0.1` | **never bind publicly** — the push API shares this listener | -| `gatus_storage_type` | `sqlite` | `memory` loses all history on restart | -| `gatus_alerting` | `{}` | pass-through; any provider Gatus supports | -| `gatus_allow_icmp` | `true` | adds back `NET_RAW` for `icmp://` checks | - -`gatus_alerting` empty is valid and is the current state: every condition is -still evaluated and recorded, there is just nowhere to shout yet. - -## Verifying - -```bash -docker ps --filter name=gatus -docker logs gatus --tail 50 -curl -s localhost:8080/health -ls /opt/gatus/config/endpoints/ -``` diff --git a/ansible/roles/gatus/defaults/main.yml b/ansible/roles/gatus/defaults/main.yml deleted file mode 100644 index 6f6ec09..0000000 --- a/ansible/roles/gatus/defaults/main.yml +++ /dev/null @@ -1,118 +0,0 @@ ---- -# Gatus, deployed the way upstream distributes it: the container image. -# -# Upstream publishes NO binary release assets - the image is the only artefact -# they ship, and therefore the only artefact they test. Building from source is -# possible (`go build` alone is enough; the Vue dashboard is compiled in via -# `//go:embed static`, and CGO_ENABLED=0 works because the sqlite driver is -# pure-Go modernc.org/sqlite) but it produces a binary upstream never ran, and -# it means a full compile of the AWS SDK and gRPC on the smallest box in the -# estate. - -# ── Image ──────────────────────────────────────────────────────────────────── -gatus_version: "v5.36.0" -# Pinned by DIGEST, not by tag. A tag is mutable - `v5.36.0` can be repushed - -# so pinning the tag alone is a weaker promise than it looks. The digest is the -# content address: if it resolves, it is byte-for-byte the image reviewed here. -# Both must be updated together; the tag is kept only so humans can read it. -gatus_image_digest: "sha256:c5f210d095fa78e6efaa20ffeb14803f2ba4f10615e16a6d12087697149617f0" -gatus_image: "ghcr.io/twin/gatus@{{ gatus_image_digest }}" - -# ── Paths (host side) ──────────────────────────────────────────────────────── -gatus_dir: /opt/gatus -gatus_config_dir: "{{ gatus_dir }}/config" -gatus_data_dir: "{{ gatus_dir }}/data" - -# Gatus merges every *.yaml under GATUS_CONFIG_PATH and its subdirectories: -# maps deep-merge, lists append. That is why this role ships a config DIRECTORY -# rather than one file - each service contributes its own endpoint file, the -# same way each service contributes a vhost through `caddy_site`. -# -# Primitives must be defined exactly once across all files or the merge is -# ambiguous, so everything that is not a list lives in the base file and -# nowhere else. -gatus_base_config_file: "00-base.yaml" -gatus_endpoints_dir: "{{ gatus_config_dir }}/endpoints" - -# ── Identity ───────────────────────────────────────────────────────────────── -# The image is FROM scratch, so it has no /etc/passwd and no user to drop to by -# name. Run it by numeric uid/gid instead, and own the data volume to match. -gatus_uid: 10001 -gatus_gid: 10001 - -# ── Web ────────────────────────────────────────────────────────────────────── -gatus_port: 8080 -# HOST-side address the container's port is published on. Gatus itself always -# binds 0.0.0.0 inside the container - see the note in config.yaml.j2. Never -# publish this on 0.0.0.0: the external-endpoint push API shares the dashboard's -# listener, and Caddy is what should be in front of both. -gatus_bind_address: "127.0.0.1" -# Pass-through, the same shape as gatus_alerting: whatever is set here is -# rendered verbatim under `ui:`, so every option upstream supports is reachable -# without adding a variable per key. See config/ui/ui.go for the full set - -# title, description, header, dashboard-heading/subheading, logo, link, favicon, -# buttons, custom-css, dark-mode, default-sort-by, default-filter-by. -gatus_ui: - title: "Status" - header: "Status" - # Group by default. The dashboard's own groupByGroup toggle starts OFF and - # remembers per browser in localStorage, so without this the ten groups render - # as one flat list for anyone who has not clicked it. - default-sort-by: group - -# ── Storage ────────────────────────────────────────────────────────────────── -# sqlite, not memory: history has to survive a restart, or the dashboard lies -# about uptime after every deploy. Path is INSIDE the container. -gatus_storage_type: sqlite -gatus_storage_path: "/data/gatus.db" -gatus_storage_caching: true - -# Per-endpoint row caps. Gatus bounds the database by COUNT, not by time, and -# trims inline on insert (storage/store/sql/sql.go, InsertEndpointResult) - so -# there is no retention job to write and no way for this to fill a disk. -# -# History depth is therefore a function of check frequency, not of days: -# 900 results is ~3 days of a 5-minute liveness check, and ~2.5 years of a daily -# disk check. Upstream's default is 100, which would have been 8 hours of -# liveness - not enough to still see a weekend incident on Monday. -# -# Note the uptime table is separate and its 30-day retention is hard-coded -# upstream (uptimeRetention), so uptime percentages top out at 30 days whatever -# this is set to. -gatus_storage_max_results: 900 -gatus_storage_max_events: 50 - -# ── Alerting ───────────────────────────────────────────────────────────────── -# Pass-through: rendered verbatim under `alerting:`, so any provider Gatus -# supports works without touching this role. Empty means "check and record, -# alert nowhere" - valid, and the default until a provider is chosen. -gatus_alerting: {} -gatus_default_alerts: [] - -# ── Security ───────────────────────────────────────────────────────────────── -# gatus_basic_auth: {username: admin, password-bcrypt-base64: "..."} -gatus_basic_auth: {} - -gatus_maintenance: {} - -# Keep serving the previous config if a contributed endpoint file is malformed, -# instead of panicking. See the note in config.yaml.j2. -gatus_skip_invalid_config_update: true -gatus_log_level: INFO - -# ── Self-check ─────────────────────────────────────────────────────────────── -# Gatus panics on a config with no endpoints, so the role always ships one. -# Turning this off is only safe once another file in endpoints/ provides one. -gatus_self_check: true - -# ── ICMP ───────────────────────────────────────────────────────────────────── -# Gatus supports icmp:// endpoints. Raw ICMP needs CAP_NET_RAW, which the -# container would get free only if it ran as root; it does not. Set false if -# you never use icmp:// checks and want the capability dropped entirely. -gatus_allow_icmp: true - -# ── Shared network ─────────────────────────────────────────────────────────── -# Gatus runs in a container, so the HOST's loopback is not reachable from it. -# Anything Gatus must talk to locally - the Signal API that sends its alerts - -# has to be on a shared docker network and addressed by service name. -gatus_network: monitoring diff --git a/ansible/roles/gatus/handlers/main.yml b/ansible/roles/gatus/handlers/main.yml deleted file mode 100644 index 25ad865..0000000 --- a/ansible/roles/gatus/handlers/main.yml +++ /dev/null @@ -1,5 +0,0 @@ ---- -- name: Restart gatus - ansible.builtin.command: - cmd: docker compose up -d --force-recreate - chdir: "{{ gatus_dir }}" diff --git a/ansible/roles/gatus/tasks/configure.yml b/ansible/roles/gatus/tasks/configure.yml deleted file mode 100644 index ef5d7e7..0000000 --- a/ansible/roles/gatus/tasks/configure.yml +++ /dev/null @@ -1,64 +0,0 @@ ---- -- name: Assert Docker is available - ansible.builtin.command: docker --version - register: gatus_docker_check - changed_when: false - failed_when: gatus_docker_check.rc != 0 - -# Created explicitly rather than by either compose file, so neither the gatus -# stack nor the signal-api stack has to be deployed before the other. -- name: Ensure the shared monitoring network exists - ansible.builtin.command: "docker network create {{ gatus_network }}" - register: gatus_net - changed_when: "'already exists' not in gatus_net.stderr" - failed_when: - - gatus_net.rc != 0 - - "'already exists' not in gatus_net.stderr" - -- name: Create the gatus directories - ansible.builtin.file: - path: "{{ item.path }}" - state: directory - owner: "{{ item.owner }}" - group: "{{ item.group }}" - mode: "{{ item.mode }}" - loop: - - {path: "{{ gatus_dir }}", owner: root, group: root, mode: "0755"} - # Config is root-owned and world-unreadable: it holds external-endpoint - # tokens. The container mounts it read-only and reads it as gatus_uid, so - # that id needs group access - hence the group ownership below. - - {path: "{{ gatus_config_dir }}", owner: root, group: "{{ gatus_gid }}", mode: "0750"} - - {path: "{{ gatus_endpoints_dir }}", owner: root, group: "{{ gatus_gid }}", mode: "0750"} - # Data is the one path the container writes to, so it must be owned by the - # numeric id the container runs as. `read_only: true` makes everything else - # in the container immutable. - - {path: "{{ gatus_data_dir }}", owner: "{{ gatus_uid }}", group: "{{ gatus_gid }}", mode: "0750"} - -- name: Write the base gatus configuration - ansible.builtin.template: - src: config.yaml.j2 - dest: "{{ gatus_config_dir }}/{{ gatus_base_config_file }}" - owner: root - group: "{{ gatus_gid }}" - mode: "0640" - notify: Restart gatus - -# Without this the container crash-loops on an empty config. See the template. -- name: Write the gatus self-check endpoint - ansible.builtin.template: - src: endpoint-self.yaml.j2 - dest: "{{ gatus_endpoints_dir }}/00-self.yaml" - owner: root - group: "{{ gatus_gid }}" - mode: "0640" - when: gatus_self_check | bool - notify: Restart gatus - -- name: Write the docker compose file - ansible.builtin.template: - src: docker-compose.yml.j2 - dest: "{{ gatus_dir }}/docker-compose.yml" - owner: root - group: root - mode: "0644" - notify: Restart gatus diff --git a/ansible/roles/gatus/tasks/main.yml b/ansible/roles/gatus/tasks/main.yml deleted file mode 100644 index e1b67bf..0000000 --- a/ansible/roles/gatus/tasks/main.yml +++ /dev/null @@ -1,5 +0,0 @@ ---- -# import, never include: dynamic includes are opaque to --list-tasks, which is -# the primary verification tool in this repo. -- ansible.builtin.import_tasks: configure.yml -- ansible.builtin.import_tasks: service.yml diff --git a/ansible/roles/gatus/tasks/service.yml b/ansible/roles/gatus/tasks/service.yml deleted file mode 100644 index bfcca3b..0000000 --- a/ansible/roles/gatus/tasks/service.yml +++ /dev/null @@ -1,35 +0,0 @@ ---- -# Pulling by digest means this either fetches exactly the reviewed image or -# fails. There is no "latest wins" path. -- name: Pull the pinned gatus image - ansible.builtin.command: - cmd: docker compose pull - chdir: "{{ gatus_dir }}" - register: gatus_pull - changed_when: "'Downloaded newer image' in gatus_pull.stderr or 'Pull complete' in gatus_pull.stderr" - -- name: Start gatus - ansible.builtin.command: - cmd: docker compose up -d --remove-orphans - chdir: "{{ gatus_dir }}" - register: gatus_up - changed_when: "'Started' in gatus_up.stderr or 'Created' in gatus_up.stderr or 'Recreated' in gatus_up.stderr" - -- name: Flush handlers so a config change is live before it is verified - ansible.builtin.meta: flush_handlers - -- name: Wait for gatus to answer - ansible.builtin.uri: - url: "http://{{ gatus_bind_address }}:{{ gatus_port }}/health" - status_code: [200, 404] - register: gatus_health - until: gatus_health.status in [200, 404] - retries: 12 - delay: 5 - -- name: Assert gatus is running - ansible.builtin.assert: - that: - - gatus_health.status in [200, 404] - fail_msg: "gatus did not come up on {{ gatus_bind_address }}:{{ gatus_port }} - check `docker logs gatus`" - success_msg: "gatus is answering on {{ gatus_bind_address }}:{{ gatus_port }}" diff --git a/ansible/roles/gatus/templates/config.yaml.j2 b/ansible/roles/gatus/templates/config.yaml.j2 deleted file mode 100644 index faf61fb..0000000 --- a/ansible/roles/gatus/templates/config.yaml.j2 +++ /dev/null @@ -1,64 +0,0 @@ -# {{ gatus_base_config_file }} — managed by Ansible (roles/gatus) -# -# BASE CONFIGURATION ONLY. -# -# Gatus merges every *.yaml under GATUS_CONFIG_PATH: maps are deep-merged and -# lists are appended, but a primitive defined in two files is ambiguous and -# upstream refuses it. So `web`, `storage`, `ui`, `alerting` and `security` are -# set here and MUST NOT appear in any other file in this directory. -# -# Endpoints are lists, so they append cleanly. Each service drops its own file -# into endpoints/ via the gatus_endpoint role - the same shape as caddy_site. - -web: - # 0.0.0.0 is the CONTAINER's interface, not the host's. This must not be - # 127.0.0.1: that is the container's own loopback, which docker-proxy cannot - # reach, and gatus comes up healthy while the published port refuses every - # connection. - # - # The isolation comes from the port mapping in docker-compose.yml, which - # publishes to {{ gatus_bind_address }} on the host. Caddy fronts that, and it - # matters because the external-endpoint push API shares this listener with the - # dashboard. - address: 0.0.0.0 - port: {{ gatus_port }} - -# Gatus reloads when its config changes. If the NEW config fails to parse it -# calls panic() - unless this is set, in which case it logs the error and keeps -# running on the old config. Endpoint files are contributed by other playbooks, -# so one malformed file would otherwise take the monitor down, which is the -# worst possible time to lose it. -skip-invalid-config-update: {{ gatus_skip_invalid_config_update | bool | lower }} - -storage: - type: {{ gatus_storage_type }} -{% if gatus_storage_type != 'memory' %} - path: {{ gatus_storage_path }} -{% endif %} - caching: {{ gatus_storage_caching | bool | lower }} - maximum-number-of-results: {{ gatus_storage_max_results }} - maximum-number-of-events: {{ gatus_storage_max_events }} - -ui: -{{ gatus_ui | to_nice_yaml(indent=2) | indent(2, true) }} -{% if gatus_alerting %} - -alerting: -{{ gatus_alerting | to_nice_yaml(indent=2) | indent(2, true) }} -{% else %} - -# No alerting provider is configured yet. Gatus still evaluates every condition -# and records every result; it simply has nowhere to shout. Setting -# `gatus_alerting` is the single change needed to wire one up. -{% endif %} -{% if gatus_basic_auth %} - -security: - basic: -{{ gatus_basic_auth | to_nice_yaml(indent=2) | indent(4, true) }} -{% endif %} -{% if gatus_maintenance %} - -maintenance: -{{ gatus_maintenance | to_nice_yaml(indent=2) | indent(2, true) }} -{% endif %} diff --git a/ansible/roles/gatus/templates/docker-compose.yml.j2 b/ansible/roles/gatus/templates/docker-compose.yml.j2 deleted file mode 100644 index 512c9c5..0000000 --- a/ansible/roles/gatus/templates/docker-compose.yml.j2 +++ /dev/null @@ -1,53 +0,0 @@ -# Managed by Ansible (roles/gatus) -services: - gatus: - image: {{ gatus_image }} - container_name: gatus - restart: unless-stopped - - # The image is FROM scratch: no /etc/passwd, so there is no user to drop to - # by name and the default is root. Run it by numeric id instead. - user: "{{ gatus_uid }}:{{ gatus_gid }}" - - ports: - # Loopback on purpose. Caddy fronts this, and the external-endpoint push - # API is served from the same listener as the dashboard - publishing - # 0.0.0.0 would put both straight on the public internet. - - "{{ gatus_bind_address }}:{{ gatus_port }}:{{ gatus_port }}" - - environment: - GATUS_CONFIG_PATH: /config - GATUS_LOG_LEVEL: "{{ gatus_log_level }}" - - volumes: - - {{ gatus_config_dir }}:/config:ro - - {{ gatus_data_dir }}:/data - - # Hardening. The systemd unit this replaced got most of it from - # ProtectSystem/NoNewPrivileges/etc; these are the container equivalents. - read_only: true - security_opt: - - no-new-privileges:true - cap_drop: - - ALL -{% if gatus_allow_icmp %} - cap_add: - # icmp:// endpoints need raw sockets. Dropped above with ALL, added back - # explicitly so the grant is visible rather than inherited from root. - - NET_RAW -{% endif %} - - networks: - # Shared with signal-api, so alerts can be delivered by service name. - # 127.0.0.1 inside this container is the container, not the host. - - {{ gatus_network }} - - logging: - driver: json-file - options: - max-size: "10m" - max-file: "3" - -networks: - {{ gatus_network }}: - external: true diff --git a/ansible/roles/gatus/templates/endpoint-self.yaml.j2 b/ansible/roles/gatus/templates/endpoint-self.yaml.j2 deleted file mode 100644 index 197ae56..0000000 --- a/ansible/roles/gatus/templates/endpoint-self.yaml.j2 +++ /dev/null @@ -1,25 +0,0 @@ -# 00-self.yaml — managed by Ansible (roles/gatus) -# -# Gatus refuses to start with no endpoints at all: -# panic: error parsing config: configuration should contain at least one -# endpoint or suite -# -# So the role ships one. Checking its own listener is not as circular as it -# looks: it proves the config directory parsed, the container is serving, and -# the storage backend accepted a write. If this row is missing from the -# dashboard, the dashboard is not telling you the truth about anything else. -# -# It is also what keeps `gatus` deployable before any service has contributed -# an endpoint file of its own. - -endpoints: - - name: gatus - group: infrastructure - url: "http://localhost:{{ gatus_port }}/health" - interval: 60s - conditions: - - "[STATUS] == 200" -{% if gatus_default_alerts %} - alerts: -{{ gatus_default_alerts | to_nice_yaml(indent=2) | indent(6, true) }} -{% endif %} diff --git a/ansible/roles/gatus_endpoint/defaults/main.yml b/ansible/roles/gatus_endpoint/defaults/main.yml deleted file mode 100644 index 4da927a..0000000 --- a/ansible/roles/gatus_endpoint/defaults/main.yml +++ /dev/null @@ -1,59 +0,0 @@ ---- -# One invocation writes ONE file into Gatus's endpoints directory. Gatus merges -# every *.yaml under GATUS_CONFIG_PATH and appends lists, so each caller owns -# its own file and they compose without coordinating - the same shape as -# caddy_site, where each service contributes its own vhost. - -# Filename stem: .yaml -gatus_endpoint_name: "" - -# PULLED endpoints - Gatus makes the request and evaluates conditions. -# - {name, group, url, interval, conditions: [...], alerts: [...]} -# -# A DNS check adds `dns: {query-type, query-name}` - and note that for those, -# `url` is the RESOLVER to ask, not the name being looked up. -# An API behind a credential adds `headers: {Authorization: ...}`. -# A domain-expiry check is just `url: ` with a [DOMAIN_EXPIRATION] -# condition; it uses WHOIS/RDAP and needs no scheme. -gatus_endpoint_pulled: [] - -# EXTERNAL endpoints - the host pushes its own result. Gatus never reaches out, -# which is what makes this work for machines behind NAT and for state that has -# no pollable surface at all (disk usage, ZFS health, UPS mains). -# -# - {name, group, token, heartbeat, alerts: [...]} -# -# `heartbeat` is the important one: if nothing reports within that window Gatus -# alerts. That is what makes a push check detect its own failure - a dead timer -# looks exactly like a dead host, which is the correct reading. -gatus_endpoint_external: [] - -# Where the files live. Matches roles/gatus. -gatus_config_dir: /opt/gatus/config -gatus_endpoints_dir: "{{ gatus_config_dir }}/endpoints" -gatus_gid: 10001 - -# Alerts attached to every endpoint in this file that does not specify its own. -# -# Gatus's provider-level `default-alert` only supplies DEFAULTS - an endpoint -# still has to opt in with `alerts: - type: signal` or it alerts on nothing at -# all. With ~90 endpoints that cannot be written by hand, so it is applied here. -# -# failure-threshold is set by the CALLER, because the right value depends on the -# check's cadence and there is no single correct default. See the note in -# infra/400_host_monitoring.yml. -gatus_endpoint_default_alerts: [] - -# UI options applied to every PULLED endpoint that does not set its own. -# -# Only pulled endpoints can carry this - an external (push) endpoint has no `ui` -# field at all, because it has no conditions to display. -# -# Note that every hide-* option already defaults to false upstream, so there is -# nothing to un-hide. The one setting that genuinely shows MORE is -# resolve-successful-conditions: by default a failing check displays the real -# value - "[STATUS] (502) == 200" - while a passing one drops it and shows only -# "[STATUS] == 200". Turning it on resolves both, so a healthy DNS check shows -# the IP it actually resolved rather than just the assertion. -gatus_endpoint_default_ui: - resolve-successful-conditions: true diff --git a/ansible/roles/gatus_endpoint/tasks/main.yml b/ansible/roles/gatus_endpoint/tasks/main.yml deleted file mode 100644 index 40f3448..0000000 --- a/ansible/roles/gatus_endpoint/tasks/main.yml +++ /dev/null @@ -1,22 +0,0 @@ ---- -- name: Assert gatus_endpoint parameters are sane - ansible.builtin.assert: - that: - - gatus_endpoint_name | length > 0 - - gatus_endpoint_pulled | length > 0 or gatus_endpoint_external | length > 0 - fail_msg: >- - gatus_endpoint needs a name and at least one of gatus_endpoint_pulled or - gatus_endpoint_external. An empty file would contribute nothing, and a - config with no endpoints at all makes Gatus refuse to start. - -- name: "Write Gatus endpoints '{{ gatus_endpoint_name }}'" - ansible.builtin.template: - src: endpoints.yaml.j2 - dest: "{{ gatus_endpoints_dir }}/{{ gatus_endpoint_name }}.yaml" - owner: root - group: "{{ gatus_gid }}" - mode: "0640" - # No handler. Gatus polls its own config every 30s - # (main.listenToConfigurationFileChanges) and reloads itself, so writing the - # file IS the deploy. A handler here would also fail whenever this role is - # used without roles/gatus loaded. diff --git a/ansible/roles/gatus_endpoint/templates/endpoints.yaml.j2 b/ansible/roles/gatus_endpoint/templates/endpoints.yaml.j2 deleted file mode 100644 index 34e63e4..0000000 --- a/ansible/roles/gatus_endpoint/templates/endpoints.yaml.j2 +++ /dev/null @@ -1,64 +0,0 @@ -# {{ gatus_endpoint_name }}.yaml — managed by Ansible (roles/gatus_endpoint) -# -# Contributed by a playbook, not hand-edited. Gatus appends the lists in every -# *.yaml under its config directory, so this file adds to whatever else is -# registered without knowing about it. -{% if gatus_endpoint_external %} - -external-endpoints: -{% for e in gatus_endpoint_external %} - - name: {{ e.name }} - group: {{ e.group }} - token: "{{ e.token }}" -{% if e.heartbeat is defined %} - heartbeat: - interval: {{ e.heartbeat }} -{% endif %} -{% set _alerts = e.alerts | default(gatus_endpoint_default_alerts) %} -{% if _alerts %} - alerts: -{{ _alerts | to_nice_yaml(indent=2) | indent(6, true) }} -{% endif %} -{# external endpoints have no `ui` field upstream - they carry no conditions #} -{% endfor %} -{% endif %} -{% if gatus_endpoint_pulled %} - -endpoints: -{% for e in gatus_endpoint_pulled %} - - name: {{ e.name }} - group: {{ e.group }} - url: "{{ e.url }}" - interval: {{ e.interval | default('60s') }} -{% if e.dns is defined %} - # A DNS endpoint: `url` is the RESOLVER to ask, not the thing being asked - # about. The name being queried lives in query-name, and [BODY] holds the - # resolved record for the conditions below. - dns: - query-type: {{ e.dns['query-type'] }} - query-name: {{ e.dns['query-name'] }} -{% endif %} -{% if e.client is defined %} - client: -{{ e.client | to_nice_yaml(indent=2) | indent(6, true) }} -{% endif %} -{% if e.headers is defined %} - headers: -{{ e.headers | to_nice_yaml(indent=2) | indent(6, true) }} -{% endif %} - conditions: -{% for c in e.conditions %} - - "{{ c }}" -{% endfor %} -{% set _alerts = e.alerts | default(gatus_endpoint_default_alerts) %} -{% if _alerts %} - alerts: -{{ _alerts | to_nice_yaml(indent=2) | indent(6, true) }} -{% endif %} -{% set _ui = e.ui | default(gatus_endpoint_default_ui) %} -{% if _ui %} - ui: -{{ _ui | to_nice_yaml(indent=2) | indent(6, true) }} -{% endif %} -{% endfor %} -{% endif %} diff --git a/ansible/roles/healthcheck/defaults/main.yml b/ansible/roles/healthcheck/defaults/main.yml deleted file mode 100644 index 51534c2..0000000 --- a/ansible/roles/healthcheck/defaults/main.yml +++ /dev/null @@ -1,54 +0,0 @@ ---- -# One health check: a script, a systemd service, a timer, and an optional push. -# -# The exit code is the answer and systemd keeps it: -# systemctl is-failed -healthcheck.service -# Reporting anywhere else is optional and generic. Point healthcheck_push_url at -# Gatus, or at whatever replaces it, or at nothing. - -healthcheck_name: "" # e.g. disk-usage -> disk-usage-healthcheck -healthcheck_description: "" - -# The check itself. Pick ONE: -# healthcheck_check: a template under templates/checks/ (without .sh.j2) -# healthcheck_command: a shell one-liner that exits 0 for healthy -healthcheck_check: "" -healthcheck_command: "" - -# systemd timer. OnUnitActiveSec unless healthcheck_on_calendar is set. -healthcheck_interval: "5min" -healthcheck_on_calendar: "" -healthcheck_boot_delay: "2min" - -# ── Reporting ──────────────────────────────────────────────────────────────── -# Gatus external endpoints: -# POST {url}?success=true|false&error=... -# Authorization: Bearer {token} -# Empty url = check and log only, which is a valid state and not an error. -healthcheck_push_url: "" -healthcheck_push_token: "" - -# Some checks report MORE THAN ONE result - a host with four systemd services -# needs four endpoints, or a single red light cannot tell you which one died. -# Those set healthcheck_push_base to the endpoints COLLECTION and the check body -# appends each key itself, the same way check-backups.sh reports per source. -healthcheck_push_base: "" - -# Units for the systemd-units check. Each becomes its own Gatus endpoint. -healthcheck_units: [] -# Prefix for the per-unit endpoint keys, e.g. "services_vipy" -> services_vipy-caddy. -healthcheck_units_key_prefix: "" - -healthcheck_script_dir: /usr/local/bin -healthcheck_log_dir: /var/log/healthchecks - -# Per-check knobs, consumed by the templates under checks/ -healthcheck_disk_threshold: 85 # percent -healthcheck_cpu_temp_threshold: 80 # celsius -healthcheck_zfs_pool: "" -# ZFS degrades badly once a pool passes roughly 80% - allocation gets slow and -# fragmentation becomes hard to undo, and unlike a normal filesystem you cannot -# simply delete your way back to good performance. So this alarms well before -# the pool is actually out of space. -healthcheck_zfs_capacity_threshold: 80 -healthcheck_ups_name: "" diff --git a/ansible/roles/healthcheck/tasks/main.yml b/ansible/roles/healthcheck/tasks/main.yml deleted file mode 100644 index f37a0d5..0000000 --- a/ansible/roles/healthcheck/tasks/main.yml +++ /dev/null @@ -1,90 +0,0 @@ ---- -- name: "Assert healthcheck '{{ healthcheck_name }}' is fully specified" - ansible.builtin.assert: - that: - - healthcheck_name | length > 0 - - healthcheck_description | length > 0 - - (healthcheck_check | length > 0) != (healthcheck_command | length > 0) - - not (healthcheck_push_url | length > 0) or (healthcheck_push_token | length > 0) - fail_msg: >- - healthcheck needs a name, a description, exactly one of healthcheck_check - or healthcheck_command, and a token whenever a push URL is set. A push URL - with no token would report to Gatus and be rejected 401 on every run. - -# Deduplicated across the whole play run. This role is included once PER CHECK, -# and a host with several checks was otherwise running apt several times to -# install a curl that was already there - 29 apt transactions estate-wide, and -# the slowest thing in the deploy by a wide margin. The fact below remembers -# what has already been ensured on this host. -- name: Install healthcheck dependencies - ansible.builtin.package: - name: "{{ healthcheck_wanted_packages }}" - state: present - vars: - healthcheck_wanted_packages: >- - {{ (healthcheck_packages | default(['curl'])) - | difference(healthcheck_installed_packages | default([])) }} - when: healthcheck_wanted_packages | length > 0 - -- name: Remember which dependencies this host already has - ansible.builtin.set_fact: - healthcheck_installed_packages: >- - {{ (healthcheck_installed_packages | default([])) - | union(healthcheck_packages | default(['curl'])) }} - -- name: Create the healthcheck log directory - ansible.builtin.file: - path: "{{ healthcheck_log_dir }}" - state: directory - owner: root - group: root - mode: "0750" - -- name: "Install the {{ healthcheck_name }} check script" - ansible.builtin.template: - src: healthcheck.sh.j2 - dest: "{{ healthcheck_script_dir }}/{{ healthcheck_name }}-healthcheck.sh" - owner: root - group: root - mode: "0755" - -# The token is in this unit file, so it must not be world-readable. -- name: "Install the {{ healthcheck_name }} systemd service" - ansible.builtin.template: - src: healthcheck.service.j2 - dest: "/etc/systemd/system/{{ healthcheck_name }}-healthcheck.service" - owner: root - group: root - mode: "0600" - -- name: "Install the {{ healthcheck_name }} systemd timer" - ansible.builtin.template: - src: healthcheck.timer.j2 - dest: "/etc/systemd/system/{{ healthcheck_name }}-healthcheck.timer" - owner: root - group: root - mode: "0644" - -# `restarted`, not `started`: started is a no-op on an already-active timer, so -# a changed interval or a stuck timer would never be picked up. -- name: "Enable and start the {{ healthcheck_name }} timer" - ansible.builtin.systemd: - name: "{{ healthcheck_name }}-healthcheck.timer" - enabled: yes - state: restarted - daemon_reload: yes - -- name: "Run the {{ healthcheck_name }} check once now" - ansible.builtin.command: "{{ healthcheck_script_dir }}/{{ healthcheck_name }}-healthcheck.sh" - environment: - HEALTHCHECK_PUSH_URL: "{{ healthcheck_push_url }}" - HEALTHCHECK_PUSH_TOKEN: "{{ healthcheck_push_token }}" - register: healthcheck_first_run - changed_when: false - failed_when: false - -- name: "Report the first {{ healthcheck_name }} result" - ansible.builtin.debug: - msg: >- - {{ healthcheck_name }}: {{ 'HEALTHY' if healthcheck_first_run.rc == 0 - else 'UNHEALTHY (rc=' ~ healthcheck_first_run.rc ~ ')' }} diff --git a/ansible/roles/healthcheck/templates/checks/cpu-temp.sh.j2 b/ansible/roles/healthcheck/templates/checks/cpu-temp.sh.j2 deleted file mode 100644 index cf57509..0000000 --- a/ansible/roles/healthcheck/templates/checks/cpu-temp.sh.j2 +++ /dev/null @@ -1,28 +0,0 @@ - # Hottest core across every thermal zone and hwmon sensor lm-sensors knows - # about. Reading the hottest rather than an average is deliberate: one core - # throttling is a real problem that an average hides. - local threshold={{ healthcheck_cpu_temp_threshold }} - local hottest=0 label="" - - while read -r t; do - [ -z "$t" ] && continue - t=${t%.*} - if [ "$t" -gt "$hottest" ]; then hottest=$t; fi - done < <(sensors -u 2>/dev/null | awk '/_input:/ && /temp/ {print $2}') - - # Fall back to the kernel thermal zones if lm-sensors reports nothing. - if [ "$hottest" -eq 0 ]; then - for z in /sys/class/thermal/thermal_zone*/temp; do - [ -r "$z" ] || continue - local milli; milli=$(cat "$z" 2>/dev/null) || continue - local c=$((milli / 1000)) - if [ "$c" -gt "$hottest" ]; then hottest=$c; label=$(cat "${z%/temp}/type" 2>/dev/null); fi - done - fi - - if [ "$hottest" -eq 0 ]; then - MESSAGE="no temperature sensors readable" - return 1 - fi - MESSAGE="${hottest}C${label:+ (${label})}" - [ "$hottest" -lt "$threshold" ] diff --git a/ansible/roles/healthcheck/templates/checks/disk-usage.sh.j2 b/ansible/roles/healthcheck/templates/checks/disk-usage.sh.j2 deleted file mode 100644 index 86fd3a6..0000000 --- a/ansible/roles/healthcheck/templates/checks/disk-usage.sh.j2 +++ /dev/null @@ -1,35 +0,0 @@ - # Every real filesystem must be under the threshold. The exclusions are all - # pseudo-filesystems whose "usage" is not a disk-space fact an operator can - # act on: tmpfs and devtmpfs are RAM, squashfs is read-only, overlay is a - # container layer, and efivarfs is UEFI variable storage - which sits near - # 90% full on a perfectly healthy machine and would page every single day. - # - # NOTE: no -P. `df -P --output=...` fails with "options -P and --output are - # mutually exclusive", and an earlier version of this check had both - so df - # errored, the loop read nothing, and it reported "max 0% on /" and exited 0 - # on every host regardless of real usage. A disk check that always says 0% - # is worse than no disk check, so the guard below treats "no filesystems - # read" as a failure rather than as health. - local threshold={{ healthcheck_disk_threshold }} - local worst=0 worst_mount="" over="" seen=0 - - while read -r pct mount; do - pct=${pct%\%} - [ -z "$pct" ] && continue - case "$pct" in (*[!0-9]*) continue;; esac - seen=$((seen + 1)) - if [ "$pct" -gt "$worst" ]; then worst=$pct; worst_mount=$mount; fi - if [ "$pct" -ge "$threshold" ]; then over="${over}${over:+, }${mount} ${pct}%"; fi - done < <(df -x tmpfs -x devtmpfs -x squashfs -x overlay -x efivarfs -x ramfs --output=pcent,target 2>/dev/null | tail -n +2) - - if [ "$seen" -eq 0 ]; then - MESSAGE="df returned no filesystems - the check itself is broken" - return 1 - fi - - if [ -n "$over" ]; then - MESSAGE="over ${threshold}%: ${over}" - return 1 - fi - MESSAGE="max ${worst}% on ${worst_mount:-/} (${seen} filesystems)" - return 0 diff --git a/ansible/roles/healthcheck/templates/checks/liveness.sh.j2 b/ansible/roles/healthcheck/templates/checks/liveness.sh.j2 deleted file mode 100644 index 141e478..0000000 --- a/ansible/roles/healthcheck/templates/checks/liveness.sh.j2 +++ /dev/null @@ -1,6 +0,0 @@ - # Liveness has no test to run: the fact that this script executed at all is - # the signal. What proves the host is alive is the PUSH arriving at Gatus, - # and what detects the host being dead is the heartbeat window expiring with - # no push. So this always succeeds - the reporting is the check. - MESSAGE="up since $(uptime -p 2>/dev/null || echo unknown)" - return 0 diff --git a/ansible/roles/healthcheck/templates/checks/systemd-units.sh.j2 b/ansible/roles/healthcheck/templates/checks/systemd-units.sh.j2 deleted file mode 100644 index 90c6193..0000000 --- a/ansible/roles/healthcheck/templates/checks/systemd-units.sh.j2 +++ /dev/null @@ -1,33 +0,0 @@ - # One result PER UNIT, not one for the host. A host running four services - # needs four endpoints: a single red light would tell you "something on vipy - # is down" without saying which, and that is the question you actually have. - # - # Keys are host-qualified because unit names collide - caddy runs on four - # machines. Gatus builds the key as sanitize(group)_sanitize(name), so - # group "services" + name "vipy/caddy" becomes services_vipy-caddy. - local prefix="{{ healthcheck_units_key_prefix }}" - local down="" - - for unit in {{ healthcheck_units | join(' ') }}; do - local state substate - state=$(systemctl is-active "$unit" 2>/dev/null || true) - substate=$(systemctl show -p SubState --value "$unit" 2>/dev/null || true) - - if [ "$state" = "active" ]; then - report_key "${prefix}-${unit}" "true" "${unit} active (${substate:-running})" - else - # `failed` and `inactive` are different stories: one crashed, one was - # stopped. Both are down, but the message should say which. - local detail="${unit} is ${state:-unknown}" - [ "$state" = "failed" ] && detail="${detail} (${substate:-failed})" - report_key "${prefix}-${unit}" "false" "$detail" - down="${down}${down:+, }${unit}=${state:-unknown}" - fi - done - - if [ -n "$down" ]; then - MESSAGE="down: ${down}" - return 1 - fi - MESSAGE="all {{ healthcheck_units | length }} units active" - return 0 diff --git a/ansible/roles/healthcheck/templates/checks/ups-status.sh.j2 b/ansible/roles/healthcheck/templates/checks/ups-status.sh.j2 deleted file mode 100644 index 508c860..0000000 --- a/ansible/roles/healthcheck/templates/checks/ups-status.sh.j2 +++ /dev/null @@ -1,18 +0,0 @@ - # OL means on line power. Anything else - OB (on battery), LB (low battery), - # or no answer at all - is a failure worth waking up for, because the - # hypervisor has a finite number of minutes left. - local ups="{{ healthcheck_ups_name }}" - local status charge runtime load - - status=$(upsc "${ups}@localhost" ups.status 2>/dev/null) - if [ -z "$status" ]; then - MESSAGE="cannot reach UPS ${ups} via upsd" - return 1 - fi - - charge=$(upsc "${ups}@localhost" battery.charge 2>/dev/null) - runtime=$(upsc "${ups}@localhost" battery.runtime 2>/dev/null) - load=$(upsc "${ups}@localhost" ups.load 2>/dev/null) - - MESSAGE="status=${status} charge=${charge}% runtime=${runtime}s load=${load}%" - [[ "$status" == *"OL"* ]] diff --git a/ansible/roles/healthcheck/templates/checks/zfs-health.sh.j2 b/ansible/roles/healthcheck/templates/checks/zfs-health.sh.j2 deleted file mode 100644 index ebe88b9..0000000 --- a/ansible/roles/healthcheck/templates/checks/zfs-health.sh.j2 +++ /dev/null @@ -1,63 +0,0 @@ - # Five conditions, all of which have to hold. Ported from the check that - # infra/nodito/32_zfs_pool_setup_playbook.yml deployed, which was correct - - # only its reporting was tied to Uptime Kuma. - local pool="{{ healthcheck_zfs_pool }}" - local json issues="" - - json=$(zpool status -j "$pool" 2>&1) || { MESSAGE="zpool status failed: ${json}"; return 1; } - - # 1. pool state - local state - state=$(echo "$json" | jq -r --arg p "$pool" '.pools[$p].state') - [ "$state" = "ONLINE" ] || issues="${issues}${issues:+; }pool ${state}" - - # 2. every vdev and device ONLINE - local bad - bad=$(echo "$json" | jq -r --arg p "$pool" ' - .pools[$p].vdevs[] | .. | objects - | select(.state? and .state != "ONLINE") - | "\(.name // "unknown"):\(.state)"' 2>/dev/null | paste -sd, -) - [ -z "$bad" ] || issues="${issues}${issues:+; }devices ${bad}" - - # 3. resilver in progress - local fn st - fn=$(echo "$json" | jq -r --arg p "$pool" '.pools[$p].scan_stats.function // "NONE"') - st=$(echo "$json" | jq -r --arg p "$pool" '.pools[$p].scan_stats.state // "NONE"') - if [ "$fn" = "RESILVER" ] && [ "$st" = "SCANNING" ]; then - issues="${issues}${issues:+; }resilvering" - fi - - # 4. read/write/checksum errors. ZFS reports these as strings. - local errs - errs=$(echo "$json" | jq -r --arg p "$pool" ' - .pools[$p].vdevs[] | .. | objects - | select(.name? and ((.read_errors // "0" | tonumber) > 0 - or (.write_errors // "0" | tonumber) > 0 - or (.checksum_errors // "0" | tonumber) > 0)) - | "\(.name) r=\(.read_errors) w=\(.write_errors) c=\(.checksum_errors)"' 2>/dev/null | paste -sd, -) - [ -z "$errs" ] || issues="${issues}${issues:+; }errors ${errs}" - - # 5. errors from the last scrub - local scan_err - scan_err=$(echo "$json" | jq -r --arg p "$pool" '.pools[$p].scan_stats.errors // "0"') - if [ -n "$scan_err" ] && [ "$scan_err" != "0" ] && [ "$scan_err" != "null" ]; then - issues="${issues}${issues:+; }scan errors ${scan_err}" - fi - - # 6. Capacity. Not an error condition in `zpool status` - a 95% full pool is - # reported perfectly ONLINE - so it has to be read separately, and it is - # the failure you get warning of rather than the one you discover. - local capacity - capacity=$(zpool list -H -o capacity "$pool" 2>/dev/null | tr -dc '0-9') - if [ -z "$capacity" ]; then - issues="${issues}${issues:+; }cannot read capacity" - elif [ "$capacity" -ge {{ healthcheck_zfs_capacity_threshold }} ]; then - issues="${issues}${issues:+; }pool ${capacity}% full (>={{ healthcheck_zfs_capacity_threshold }}%)" - fi - - if [ -n "$issues" ]; then MESSAGE="$issues"; return 1; fi - - local scrub - scrub=$(echo "$json" | jq -r --arg p "$pool" '.pools[$p].scan_stats.start_time // "never"') - MESSAGE="${pool} ONLINE, ${capacity}% full, last scrub ${scrub}" - return 0 diff --git a/ansible/roles/healthcheck/templates/healthcheck.service.j2 b/ansible/roles/healthcheck/templates/healthcheck.service.j2 deleted file mode 100644 index 3ddd8aa..0000000 --- a/ansible/roles/healthcheck/templates/healthcheck.service.j2 +++ /dev/null @@ -1,17 +0,0 @@ -[Unit] -Description={{ healthcheck_description }} -After=network-online.target -Wants=network-online.target - -[Service] -Type=oneshot -User=root -ExecStart={{ healthcheck_script_dir }}/{{ healthcheck_name }}-healthcheck.sh -Environment=HEALTHCHECK_PUSH_URL={{ healthcheck_push_url }} -Environment=HEALTHCHECK_PUSH_TOKEN={{ healthcheck_push_token }} -Environment=HEALTHCHECK_PUSH_BASE={{ healthcheck_push_base }} -StandardOutput=journal -StandardError=journal - -[Install] -WantedBy=multi-user.target diff --git a/ansible/roles/healthcheck/templates/healthcheck.sh.j2 b/ansible/roles/healthcheck/templates/healthcheck.sh.j2 deleted file mode 100644 index 6667878..0000000 --- a/ansible/roles/healthcheck/templates/healthcheck.sh.j2 +++ /dev/null @@ -1,82 +0,0 @@ -#!/bin/bash -# {{ healthcheck_name }} — {{ healthcheck_description }} -# Managed by Ansible (roles/healthcheck). Do not edit on the host. -# -# The exit code is the real answer; systemd stores it: -# systemctl is-failed {{ healthcheck_name }}-healthcheck.service -# The push below is an optional extra, and having no URL is normal. - -set -uo pipefail - -LOG_FILE="{{ healthcheck_log_dir }}/{{ healthcheck_name }}.log" -PUSH_URL="${HEALTHCHECK_PUSH_URL:-}" -PUSH_TOKEN="${HEALTHCHECK_PUSH_TOKEN:-}" -# Set only by checks that report several results; see healthcheck_push_base. -PUSH_BASE="${HEALTHCHECK_PUSH_BASE:-}" - -log() { echo "$(date '+%Y-%m-%d %H:%M:%S') - $*" >> "$LOG_FILE"; } - -# Report to Gatus as an external endpoint. Note this is a POST with a bearer -# token, not a GET with a query string - it is not the shape Uptime Kuma used. -report() { - local success="$1" message="$2" - - # No push URL is normal, not an error: the exit code below is a complete - # answer for anything reading unit state. - [ -n "$PUSH_URL" ] || return 0 - - local encoded - encoded=$(printf '%s' "$message" | sed 's/%/%25/g; s/ /%20/g; s/&/%26/g; s/+/%2B/g; s/#/%23/g') - - local code - code=$(curl -s -o /dev/null -w '%{http_code}' -X POST \ - --max-time 15 --retry 2 --retry-delay 3 \ - -H "Authorization: Bearer ${PUSH_TOKEN}" \ - "${PUSH_URL}?success=${success}&error=${encoded}" 2>/dev/null) - - if [ "$code" = "200" ]; then - log "reported success=${success}" - else - log "ERROR: report failed (HTTP ${code})" - return 1 - fi -} - -# Report to an arbitrary endpoint key under PUSH_BASE. Used by checks that -# produce one result per item rather than a single verdict. -report_key() { - local key="$1" success="$2" message="$3" - [ -n "$PUSH_BASE" ] || return 0 - local encoded - encoded=$(printf '%s' "$message" | sed 's/%/%25/g; s/ /%20/g; s/&/%26/g; s/+/%2B/g; s/#/%23/g') - curl -s -o /dev/null --max-time 15 --retry 2 --retry-delay 3 -X POST \ - -H "Authorization: Bearer ${PUSH_TOKEN}" \ - "${PUSH_BASE}/${key}/external?success=${success}&error=${encoded}" 2>/dev/null || true -} - -# ── the check ──────────────────────────────────────────────────────────────── -# -# The blank line before the closing brace below is load-bearing. Jinja strips an -# included template's trailing newline, and trim_blocks (on by default in -# Ansible) then eats the newline after {% raw %}{% endif %}{% endraw %} - so without it the brace lands -# on the same line as the check body's last statement, producing `return 0}` and -# a script that dies with "syntax error: unexpected end of file". -check() { -{% if healthcheck_check %} -{% include 'checks/' ~ healthcheck_check ~ '.sh.j2' %} -{% else %} - {{ healthcheck_command }} -{% endif %} - -} - -MESSAGE="" -if check; then - log "OK${MESSAGE:+ - $MESSAGE}" - report "true" "${MESSAGE:-ok}" - exit 0 -else - log "FAILED${MESSAGE:+ - $MESSAGE}" - report "false" "${MESSAGE:-check failed}" - exit 1 -fi diff --git a/ansible/roles/healthcheck/templates/healthcheck.timer.j2 b/ansible/roles/healthcheck/templates/healthcheck.timer.j2 deleted file mode 100644 index 0271735..0000000 --- a/ansible/roles/healthcheck/templates/healthcheck.timer.j2 +++ /dev/null @@ -1,18 +0,0 @@ -[Unit] -Description=Run {{ healthcheck_description }} -Requires={{ healthcheck_name }}-healthcheck.service - -[Timer] -OnBootSec={{ healthcheck_boot_delay }} -{% if healthcheck_on_calendar %} -OnCalendar={{ healthcheck_on_calendar }} -{% else %} -OnUnitActiveSec={{ healthcheck_interval }} -{% endif %} -# Run a missed occurrence on the next boot rather than silently skipping it. -Persistent=true -# Spread the pushes out so twelve hosts do not all report in the same second. -RandomizedDelaySec={{ healthcheck_randomized_delay | default('30') }} - -[Install] -WantedBy=timers.target diff --git a/ansible/roles/jar_lid_archive/README.md b/ansible/roles/jar_lid_archive/README.md deleted file mode 100644 index 8f2e641..0000000 --- a/ansible/roles/jar_lid_archive/README.md +++ /dev/null @@ -1,121 +0,0 @@ -# `jar_lid_archive` - -Runs [Jar Lid Archive](https://forgejo.contrapeso.xyz/counterweight/jar-lid-archive) -as a docker compose stack: Django under gunicorn (`web`), Postgres 18 (`db`) and -SeaweedFS as the S3 photo store (`storage`). The same three services as the app -repo's own `compose.yaml`, so staging rehearses what dev and CI run. - -Deployed by `services/jar-lid-archive/deploy_jar_lid_archive_playbook.yml` to -every host in `[jar_lid_archive]`. Today that is `jar_lid_staging_local` -(jar-lid-archive-staging-box, nodito VM 3300), public at -`jars-staging.contrapeso.xyz` through vipy's Caddy. - -Per-host values are in `host_vars//main.yml`, and secrets are in `vault.yml` -beside it. A production host would be a second inventory entry with its own -`jar_lid_archive_instance`, domain and vault, not a second playbook. - -## Updates: follows `:master` - -CI pushes `:master` on every green build of master. Every run of the playbook -runs `docker compose pull` and `up -d --wait`, and compose recreates `web` only -if the digest moved. There is no watchtower, so **a merge reaches staging when -the playbook runs**, not on its own. The run ends by printing `/health/`, which -includes the deployed version. - -`up --wait` only returns once every container's healthcheck passes. `web`'s is -`/health/`, which runs `SELECT 1` and touches the bucket, so a green run means -the app actually serves. - -## Networking: why `web` is on the host network - -Docker's published ports are DNATed in the `nat` table, before UFW's `INPUT` -rules ever see the packet. A `ports: "8000:8000"` would answer on the LAN as well -as the tailnet, whatever UFW says. Binding the publish to the tailnet IP avoids -that, but at boot Docker races tailscaled for the address and loses. - -So `web` uses `network_mode: host`, and gunicorn's port is an ordinary host -port: - -- **LAN:** UFW default-deny blocks it. -- **Tailnet:** tailscaled's `ts-input` chain accepts `tailscale0` ahead of UFW - (checked on mempool-box), so the headscale ACL is the real control. It grants - `tag:prd-vipy` exactly `tag:jar-lid-archive-staging-box:8000`. -- `db` and `storage` publish on `127.0.0.1` only, which is how `web` reaches - them from the host network. - -This is load-bearing for HTTPS. The app is told to trust `X-Forwarded-Proto` -(`DJANGO_BEHIND_TLS_PROXY=1`), which is only safe while Caddy is the only -thing that can reach gunicorn. Caddy overwrites whatever value the client sent. - -## Secrets and seed users - -The Django secret key, the Postgres password, the S3 secret and the three seed -passwords were generated into `host_vars/jar_lid_staging_local/vault.yml`. They -reach the box as `/opt/jar-lid-archive/.env` (0600, root), which compose -interpolates. `docker-compose.yml` itself has no secrets. - -The seed users are alice, bob and admin (superuser), with the fixture's usernames -and **vault** passwords. The fixture's `*-test-pass` passwords are published in -the app repo and this site is public. `seed_users` runs on every deploy and -resets each listed user's password, so rotating one means editing the vault and -re-running the playbook. - -```sh -ansible-vault view host_vars/jar_lid_staging_local/vault.yml # from ansible/ -``` - -## Backups - -`services/jar-lid-archive/setup_backup_jar_lid_archive.yml` uses -`backup_source`. At 02:20 it stops `web` only, and -`/usr/local/bin/jar-lid-archive-dump.sh` writes one tar with: - -- `db.dump`: `pg_dump -Fc` -- `seaweedfs/`: the photo store's data directory, as files - -`web` is the only writer to the database and the photo store. With it stopped, -both are quiet and are captured at the same instant. `storage` is deliberately -left running: on this disk it can take minutes to restart, and the first test run -that stopped it left the app down for 26 minutes. Uploads fail while `web` is -down. - -The tar is not gzipped, because the bulk is JPEGs. Three days are kept locally, -and small-backups-box keeps three as well. Staging data is disposable. - -### Restore - -```sh -age -d -i ~/.age/counterweight_age jar-lid-archive-staging_.tar.age | tar -xf - -C /tmp/restore -# on the box, stack down: -docker compose -f /opt/jar-lid-archive/docker-compose.yml down -rm -rf /opt/jar-lid-archive/seaweedfs && cp -a /tmp/restore/seaweedfs /opt/jar-lid-archive/ -docker compose -f /opt/jar-lid-archive/docker-compose.yml up -d db -docker compose -f /opt/jar-lid-archive/docker-compose.yml exec -T db \ - pg_restore -U jarlid -d jarlid --clean --if-exists < /tmp/restore/db.dump -docker compose -f /opt/jar-lid-archive/docker-compose.yml up -d --wait -``` - -## Monitoring - -| What | Where | -|---|---| -| Host liveness and disk | infra/400 (automatic for `managed`) | -| `/health/` from the box, every 5 min | `probe_jar-lid-archive-staging` (403) | -| `/health/` through DNS, Caddy, TLS and the tailnet | `jars-staging` public site (402) | -| Dump ran / artefact arrived | `backup-dump_` / `backup-store_jar-lid-archive-staging` | - -infra/401 has nothing to watch here, because no systemd units of ours run on -this box. - -## Size and disk - -The VM's disk is on `proxmox-tank-1`, nodito's mirror of two HDDs. knots-box's -random reads keep that mirror close to saturated, and this box has measured -**~96% "full" I/O pressure** with nothing of its own running. Everything is slow -here, but especially container starts. The healthcheck `start_period`s are sized -for that. Moving the disk to `local-zfs` (NVMe) would fix it. - - -1 core, 1 GB RAM and 10 GB disk, by choice, for staging. The disk holds Debian, -the images, the photos and three days of local backups, and each backup is -about the size of the whole photo store. infra/400's disk check alarms at 85%. diff --git a/ansible/roles/jar_lid_archive/defaults/main.yml b/ansible/roles/jar_lid_archive/defaults/main.yml deleted file mode 100644 index da488d2..0000000 --- a/ansible/roles/jar_lid_archive/defaults/main.yml +++ /dev/null @@ -1,57 +0,0 @@ ---- -# Jar Lid Archive: Django (gunicorn) + Postgres + SeaweedFS (S3) as one compose -# stack. Source and CI live in forgejo.contrapeso.xyz/counterweight/jar-lid-archive. -# -# Per-deployment values (domain, port, retention) live in the host's host_vars, -# because the Caddy play on the edge host and the backup plays read them too and -# a role default cannot reach a second play. - -jar_lid_archive_dir: /opt/jar-lid-archive - -# ── Images ─────────────────────────────────────────────────────────────────── -# The app follows a moving tag on purpose: every run pulls it, and compose -# recreates `web` only when the digest actually changed. CI pushes :master on -# every green build of master, plus :. -jar_lid_archive_image: forgejo.contrapeso.xyz/counterweight/jar-lid-archive -jar_lid_archive_image_tag: master - -# Pinned to what the app's own compose.yaml and CI run, so staging rehearses the -# same stack. Bump them together with the app repo, not separately. -jar_lid_archive_postgres_image: "postgres:18" -jar_lid_archive_seaweedfs_image: "chrislusf/seaweedfs:4.47" - -# Registry login. Empty = pull anonymously, which works today: the package is -# owner-scoped in Forgejo and public even though the repository is private. -jar_lid_archive_registry: forgejo.contrapeso.xyz -jar_lid_archive_registry_user: "" -jar_lid_archive_registry_token: "" - -# ── Networking ─────────────────────────────────────────────────────────────── -# gunicorn listens on the HOST network on this port. See the README for why host -# networking rather than a published port: Docker's published ports bypass UFW. -jar_lid_archive_port: 8000 -# Loopback-only ports for the two backing services, so the web container (on the -# host network) can reach them and nothing else can. -jar_lid_archive_postgres_port: 5432 -jar_lid_archive_s3_port: 8333 - -jar_lid_archive_domain: "" # e.g. jars-staging.contrapeso.xyz - -# ── App settings ───────────────────────────────────────────────────────────── -jar_lid_archive_postgres_db: jarlid -jar_lid_archive_postgres_user: jarlid -jar_lid_archive_s3_access_key: jarlid -jar_lid_archive_s3_bucket: photos - -# Secrets: pass from the vault. The assert in tasks/deploy.yml refuses empties. -jar_lid_archive_django_secret_key: "" -jar_lid_archive_postgres_password: "" -jar_lid_archive_s3_secret_key: "" - -# ── Seed users ─────────────────────────────────────────────────────────────── -# Loaded with `manage.py seed_users -` on every run. That command RESETS each -# listed user's password and flags, so the vault stays the source of truth. -# Same usernames as the app's tests/fixtures/users.json, never its passwords: -# this deployment is reachable from the internet. -# - {username: alice, password: "...", email: "...", is_superuser: false} -jar_lid_archive_seed_users: [] diff --git a/ansible/roles/jar_lid_archive/tasks/deploy.yml b/ansible/roles/jar_lid_archive/tasks/deploy.yml deleted file mode 100644 index 4a6e783..0000000 --- a/ansible/roles/jar_lid_archive/tasks/deploy.yml +++ /dev/null @@ -1,100 +0,0 @@ ---- -- name: Assert the deployment is fully specified - ansible.builtin.assert: - that: - - jar_lid_archive_domain | length > 0 - - jar_lid_archive_django_secret_key | length >= 50 - - jar_lid_archive_postgres_password | length > 0 - - jar_lid_archive_s3_secret_key | length > 0 - fail_msg: >- - jar_lid_archive needs a domain and its three secrets from the vault. The - app falls back to development defaults for every one of them, so an empty - value here would not fail - it would deploy with a published secret key. - no_log: true - -- name: Create the deployment directory - ansible.builtin.file: - path: "{{ jar_lid_archive_dir }}" - state: directory - owner: root - group: root - mode: '0755' - -# Existence only, no owner: the postgres and seaweedfs entrypoints chown their -# own data directories to the uid inside the container. Declaring an owner here -# would fight them on every run - the mempool MariaDB lesson. -- name: Ensure the data directories exist - ansible.builtin.file: - path: "{{ jar_lid_archive_dir }}/{{ item }}" - state: directory - loop: - - postgres - - seaweedfs - -# diff: false, because --diff prints rendered content and this is all secrets. -- name: Write the compose environment file - ansible.builtin.template: - src: env.j2 - dest: "{{ jar_lid_archive_dir }}/.env" - owner: root - group: root - mode: '0600' - diff: false - -- name: Write docker-compose.yml - ansible.builtin.template: - src: docker-compose.yml.j2 - dest: "{{ jar_lid_archive_dir }}/docker-compose.yml" - owner: root - group: root - mode: '0644' - -- name: Log in to the container registry - ansible.builtin.command: - cmd: "docker login {{ jar_lid_archive_registry }} -u {{ jar_lid_archive_registry_user }} --password-stdin" - stdin: "{{ jar_lid_archive_registry_token }}" - when: jar_lid_archive_registry_token | length > 0 - changed_when: false - no_log: true - -# Pulled on every run: staging follows :master. `up` below recreates `web` only -# when the pulled digest differs from the running one. -- name: Pull images - ansible.builtin.command: docker compose pull --quiet - args: - chdir: "{{ jar_lid_archive_dir }}" - changed_when: false - -# --wait blocks until every service's healthcheck passes, and `web`'s is -# /health/, which itself checks the database and the bucket - so this task -# succeeding means the app actually serves, not merely that containers exist. -- name: Start the stack and wait for it to be healthy - ansible.builtin.command: docker compose up -d --remove-orphans --wait --wait-timeout 300 - args: - chdir: "{{ jar_lid_archive_dir }}" - register: jar_lid_archive_up - changed_when: jar_lid_archive_up.stderr is search('Created|Recreated|Started|Removed') - -- name: Read the deployed version - ansible.builtin.uri: - url: "http://127.0.0.1:{{ jar_lid_archive_port }}/health/" - return_content: true - register: jar_lid_archive_health - # Under --check the stack was never started, so there is nothing to ask. - when: not ansible_check_mode - -- name: Show the deployed version - ansible.builtin.debug: - msg: "{{ jar_lid_archive_health.json }}" - when: not ansible_check_mode - -# Used by the backup (services/jar-lid-archive/setup_backup_jar_lid_archive.yml). -# Lives here because what to dump is a fact about this stack's layout. -- name: Install the dump script - ansible.builtin.template: - src: dump.sh.j2 - dest: /usr/local/bin/jar-lid-archive-dump.sh - owner: root - group: root - mode: '0755' - validate: "bash -n %s" diff --git a/ansible/roles/jar_lid_archive/tasks/docker.yml b/ansible/roles/jar_lid_archive/tasks/docker.yml deleted file mode 100644 index 2eee8dc..0000000 --- a/ansible/roles/jar_lid_archive/tasks/docker.yml +++ /dev/null @@ -1,74 +0,0 @@ ---- -# Same Docker CE install as roles/mempool/tasks/docker.yml. infra/910_docker -# targets every managed host, so each Docker role installs its own engine. -- name: Remove old Docker-related packages - apt: - name: - - docker.io - - docker-doc - - docker-compose - - podman-docker - - containerd - - runc - state: absent - purge: yes - autoremove: yes - -- name: Update apt cache - apt: - update_cache: yes - -- name: Install prerequisites - apt: - name: - - ca-certificates - - curl - state: present - -- name: Create directory for Docker GPG key - file: - path: /etc/apt/keyrings - state: directory - mode: '0755' - -- name: Download Docker GPG key - get_url: - url: https://download.docker.com/linux/debian/gpg - dest: /etc/apt/keyrings/docker.asc - mode: '0644' - -- name: Get Debian architecture - command: dpkg --print-architecture - register: deb_arch - changed_when: false - -- name: Add Docker repository - apt_repository: - repo: "deb [arch={{ deb_arch.stdout }} signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian {{ ansible_distribution_release }} stable" - filename: docker - state: present - update_cache: yes - -- name: Install Docker packages - apt: - name: - - docker-ce - - docker-ce-cli - - containerd.io - - docker-buildx-plugin - - docker-compose-plugin - state: present - update_cache: yes - -- name: Ensure Docker is started and enabled - systemd: - name: docker - enabled: yes - state: started - -- name: Add user to docker group - user: - name: "{{ ansible_user }}" - groups: docker - append: yes - diff --git a/ansible/roles/jar_lid_archive/tasks/main.yml b/ansible/roles/jar_lid_archive/tasks/main.yml deleted file mode 100644 index c16a270..0000000 --- a/ansible/roles/jar_lid_archive/tasks/main.yml +++ /dev/null @@ -1,6 +0,0 @@ ---- -# import_tasks, not include_tasks: static imports stay visible to --list-tasks, -# which is the main verification tool in this repo. -- ansible.builtin.import_tasks: docker.yml -- ansible.builtin.import_tasks: deploy.yml -- ansible.builtin.import_tasks: seed.yml diff --git a/ansible/roles/jar_lid_archive/tasks/seed.yml b/ansible/roles/jar_lid_archive/tasks/seed.yml deleted file mode 100644 index 6b87c31..0000000 --- a/ansible/roles/jar_lid_archive/tasks/seed.yml +++ /dev/null @@ -1,13 +0,0 @@ ---- -# seed_users is idempotent but RESETS each listed user's password and flags, so -# re-running it is how a rotated vault password reaches the app. It reports -# "N created, M updated"; only a creation counts as a change. -- name: Seed users - ansible.builtin.command: - cmd: docker compose exec -T web python manage.py seed_users - - chdir: "{{ jar_lid_archive_dir }}" - stdin: "{{ jar_lid_archive_seed_users | to_json }}" - register: jar_lid_archive_seed - changed_when: "jar_lid_archive_seed.stdout is not search('Seeded users: 0 created')" - when: jar_lid_archive_seed_users | length > 0 - no_log: true diff --git a/ansible/roles/jar_lid_archive/templates/docker-compose.yml.j2 b/ansible/roles/jar_lid_archive/templates/docker-compose.yml.j2 deleted file mode 100644 index cc634f7..0000000 --- a/ansible/roles/jar_lid_archive/templates/docker-compose.yml.j2 +++ /dev/null @@ -1,123 +0,0 @@ -# Managed by Ansible - roles/jar_lid_archive. Do not edit on the host. -# -# Same three services as the app repo's compose.yaml, with three differences: -# - `web` runs the CI-built image instead of `build: .` -# - `web` is on the HOST network, so UFW and the tailnet ACL decide who reaches -# gunicorn. A published port would be DNATed by Docker straight past UFW and -# answer on the LAN too. See the role README. -# - data is bind-mounted under {{ jar_lid_archive_dir }} rather than in named -# volumes, so the backup tars a path it can name. -# -# Secrets are interpolated from .env (0600) beside this file; this file has none. -name: jar-lid-archive - -x-logging: &logging - # Docker's default json-file driver never rotates, on a 10 GB disk. - driver: json-file - options: - max-size: "10m" - max-file: "3" - -x-postgres-env: &postgres-env - POSTGRES_DB: "{{ jar_lid_archive_postgres_db }}" - POSTGRES_USER: "{{ jar_lid_archive_postgres_user }}" - POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?} - -x-s3-env: &s3-env - S3_ACCESS_KEY: "{{ jar_lid_archive_s3_access_key }}" - S3_SECRET_KEY: ${S3_SECRET_KEY:?} - -services: - web: - image: "{{ jar_lid_archive_image }}:{{ jar_lid_archive_image_tag }}" - # gunicorn binds 0.0.0.0:{{ jar_lid_archive_port }} - the port is fixed by the image's CMD. - network_mode: host - environment: - <<: [*postgres-env, *s3-env] - POSTGRES_HOST: 127.0.0.1 - POSTGRES_PORT: "{{ jar_lid_archive_postgres_port }}" - S3_ENDPOINT_URL: "http://127.0.0.1:{{ jar_lid_archive_s3_port }}" - S3_BUCKET: "{{ jar_lid_archive_s3_bucket }}" - DJANGO_DEBUG: "0" - DJANGO_SECRET_KEY: ${DJANGO_SECRET_KEY:?} - # 127.0.0.1 for the container healthcheck and the probe, which both call - # /health/ directly rather than through Caddy. - DJANGO_ALLOWED_HOSTS: "{{ jar_lid_archive_domain }},127.0.0.1,localhost" - # Caddy terminates TLS and forwards plain HTTP with X-Forwarded-Proto. The - # app must trust that header or every POST fails CSRF and share links come - # out http://. Safe only because nothing but Caddy can reach gunicorn. - DJANGO_BEHIND_TLS_PROXY: "1" - DJANGO_CSRF_TRUSTED_ORIGINS: "https://{{ jar_lid_archive_domain }}" - depends_on: - db: - condition: service_healthy - storage: - condition: service_healthy - healthcheck: - test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:{{ jar_lid_archive_port }}/health/')"] - interval: 30s - timeout: 5s - retries: 5 - start_period: 180s - logging: *logging - restart: unless-stopped - - db: - image: "{{ jar_lid_archive_postgres_image }}" - environment: *postgres-env - ports: - - "127.0.0.1:{{ jar_lid_archive_postgres_port }}:5432" - volumes: - - ./postgres:/var/lib/postgresql - healthcheck: - test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"] - interval: 5s - timeout: 3s - retries: 15 - start_period: 120s - logging: *logging - restart: unless-stopped - - storage: - image: "{{ jar_lid_archive_seaweedfs_image }}" - environment: *s3-env - # Copied from the app's compose.yaml: write the S3 identity from the - # environment, then hand over to the image's entrypoint, which drops root - # and runs master + volume + filer + S3 gateway. - # - # The `rm -f` is ours, and load-bearing. On a RESTART of the same container - # (a reboot, a backup's stop/start) /tmp/s3.json survives, owned by seaweed - # from the chown below. Root cannot then re-create it: /tmp is sticky and - # the kernel's protected_regular refuses O_CREAT on another user's file - # there. The result was "can't create /tmp/s3.json: Permission denied", an - # unhealthy storage container, and web never starting - found on the first - # backup run. - entrypoint: ["sh", "-c"] - command: - - | - rm -f /tmp/s3.json - umask 077 - cat > /tmp/s3.json < pg_restore -d jarlid db.dump -# seaweedfs/ the photo store, as files -# -# Called with `web` stopped (the backup playbook arranges it). web is the only -# writer to db and storage, so both are quiescent and agree with each other: no -# row points at a photo the tar does not have. storage keeps running; with no -# writes arriving its volume files do not change under tar. -# -# The pg_dump lands in a temp dir on the same disk as the live database, which -# is already plaintext; it is gone before this script exits. -# -# tar, not tar.gz: the bulk is JPEGs, which do not compress, and gzip on one -# core would only make the window longer. -set -euo pipefail - -cd {{ jar_lid_archive_dir }} -tmp=$(mktemp -d {{ jar_lid_archive_dir }}/.dump.XXXXXX) -trap 'rm -rf "$tmp"' EXIT - -docker compose exec -T db pg_dump -U {{ jar_lid_archive_postgres_user }} -d {{ jar_lid_archive_postgres_db }} -Fc > "$tmp/db.dump" -tar -cf - -C "$tmp" db.dump -C {{ jar_lid_archive_dir }} seaweedfs diff --git a/ansible/roles/jar_lid_archive/templates/env.j2 b/ansible/roles/jar_lid_archive/templates/env.j2 deleted file mode 100644 index 7027cae..0000000 --- a/ansible/roles/jar_lid_archive/templates/env.j2 +++ /dev/null @@ -1,5 +0,0 @@ -# Managed by Ansible - roles/jar_lid_archive. Do not edit on the host. -# Read by docker compose to interpolate docker-compose.yml. 0600: secrets. -POSTGRES_PASSWORD={{ jar_lid_archive_postgres_password }} -S3_SECRET_KEY={{ jar_lid_archive_s3_secret_key }} -DJANGO_SECRET_KEY={{ jar_lid_archive_django_secret_key }} diff --git a/ansible/roles/mempool/README.md b/ansible/roles/mempool/README.md deleted file mode 100644 index d8282cc..0000000 --- a/ansible/roles/mempool/README.md +++ /dev/null @@ -1,69 +0,0 @@ -# `mempool` - -Deploys the [Mempool](https://mempool.space) block explorer as a three-container -Docker Compose stack — MariaDB, backend, frontend — on `mempool-box`, and keeps a -health check on each. - -Converted from `deploy_mempool_playbook.yml` (745 lines) under Plan 6. The -playbook is now 37 lines: this role, plus a second play that publishes the -frontend through Caddy on the edge host. - -## Phases - -| | | -|---|---| -| `docker.yml` | Docker engine: repo, key, packages, service | -| `deploy.yml` | directories, `docker-compose.yml`, pull, up, wait-for-healthy | -| `healthcheck.yml` | three check scripts, three services, three timers | - -## Three health checks, not one - -Mempool is three moving parts and knowing *which* one is down is the point, so -each gets its own check, unit and timer, driven by the `mempool_healthchecks` -list: - -| | checks | -|---|---| -| `mariadb` | `docker inspect` health status of `mempool-db` | -| `backend` | `GET /api/v1/backend-info` | -| `frontend` | `GET /` | - -Each records its answer in its exit code, which systemd keeps: -`systemctl is-failed mempool-backend-healthcheck.service`. Reporting elsewhere -is one field per check, `push_url`, and is the plug-in point for whatever -monitoring exists. Empty means check, exit honestly, report nowhere. The URLs -are credentials, so callers pass them from the vault. - -Nothing here is specific to a monitoring product. The embedded Python that -created monitors over the Uptime Kuma API, the `/tmp` credentials file, the -push-URL file read back and parsed, and three systemd `Environment=` rewrites -are gone. - -## MariaDB owns its own data directory - -`{{ mempool_mysql_dir }}` is bind-mounted into the container, which runs as uid -**999** and must create files there. The playbook this replaced declared -`owner: "{{ ansible_user }}"` (1000) on it, which had drifted from reality ever -since the containers were created — unnoticed, because the playbook had not been -run since. - -That was not academic. The first real run of this role pulled a newer -`mariadb:10.11` and recreated `mempool-db`; had the chown still been in place, -MariaDB would have come back to a directory it could not write. The role now -ensures the directory exists and leaves ownership to the container. - -## `mempool_frontend_port` lives in `services_config.yml` - -Two hosts need it: this role deploys the frontend on `mempool-box`, and the Caddy -play proxies to it from the edge host. A role default is invisible to the second -play, so the value lives in `service_settings.mempool.frontend_port` and the role -default derives from it. - -## Expect `changed=2` on a converged host - -`Pull Mempool images` and `Deploy Mempool containers with docker compose` are -bare `command:` tasks with no `changed_when`, so they always report changed. -That is the idempotent floor, not drift. Everything else reports `ok`. - -**`mariadb:10.11` is a moving tag**, so a run can pull a newer patch release and -recreate the database container. Pin it if that is not what you want. diff --git a/ansible/roles/mempool/defaults/main.yml b/ansible/roles/mempool/defaults/main.yml deleted file mode 100644 index 9ce8f6a..0000000 --- a/ansible/roles/mempool/defaults/main.yml +++ /dev/null @@ -1,59 +0,0 @@ -# Mempool Configuration Variables - -# Version - Pinned to specific release -mempool_version: "v3.2.1" - -# Directories -mempool_dir: /opt/mempool -mempool_data_dir: "{{ mempool_dir }}/data" -mempool_mysql_dir: "{{ mempool_dir }}/mysql" - -# Network - Bitcoin Core/Knots connection (via Tailnet Magic DNS) -bitcoin_host: "knots-box" -bitcoin_rpc_port: 8332 -# Note: bitcoin_rpc_user and bitcoin_rpc_password are loaded from group_vars/all/vault.yml - -# Network - Fulcrum Electrum server (via Tailnet Magic DNS) -fulcrum_host: "fulcrum-box" -fulcrum_port: 50001 -fulcrum_tls: "false" - -# Mempool network mode -mempool_network: "mainnet" - -# Container ports (internal) -# The edge host's socket-proxy/Caddy play needs this too, and a role default is -# invisible outside this role. The authoritative value for the live deployment is -# in host_vars/mempool_box_local/main.yml, which outranks this; the value here is the -# protocol standard, so the role still works standalone. -mempool_frontend_port: 8080 -mempool_backend_port: 8999 - -# MariaDB settings -mariadb_database: "mempool" -mariadb_user: "mempool" -# Note: mariadb_mempool_password is loaded from group_vars/all/vault.yml - - - -# --- Health checks ---------------------------------------------------------- -# Three independent checks, because Mempool is three moving parts and knowing -# WHICH one is down is the whole point. Each records its answer in its exit -# code, which systemd keeps: -# systemctl is-failed mempool-backend-healthcheck.service -# -# push_url is where to report, and is the single plug-in point for whatever -# monitoring exists. Empty means check, exit honestly, report nowhere. -# The URLs are credentials, so callers pass them from the vault. -healthcheck_push_token: "" - -mempool_healthchecks: - - name: mariadb - label: MariaDB - push_url: "" - - name: backend - label: Backend - push_url: "" - - name: frontend - label: Frontend - push_url: "" diff --git a/ansible/roles/mempool/tasks/deploy.yml b/ansible/roles/mempool/tasks/deploy.yml deleted file mode 100644 index d98df7e..0000000 --- a/ansible/roles/mempool/tasks/deploy.yml +++ /dev/null @@ -1,75 +0,0 @@ ---- -- name: Create mempool directories - file: - path: "{{ item }}" - state: directory - owner: "{{ ansible_user }}" - group: "{{ ansible_user }}" - mode: '0755' - loop: - - "{{ mempool_dir }}" - - "{{ mempool_data_dir }}" - -# MariaDB owns its own data directory. The container runs as uid 999 and has to -# create files in there; this playbook declared owner: {{ ansible_user }} (1000), -# which had been drifting from reality ever since the containers were created and -# would have broken MariaDB the first time it needed a new file. It went -# unnoticed only because the playbook had not been run since. -# -# So: ensure the directory exists, and let the container own it. On a fresh -# install the mariadb image's entrypoint sets ownership itself. -- name: Ensure the MariaDB data directory exists - file: - path: "{{ mempool_mysql_dir }}" - state: directory - -- name: Create docker-compose.yml for Mempool - ansible.builtin.template: - src: docker-compose.yml.j2 - dest: "{{ mempool_dir }}/docker-compose.yml" - owner: "{{ ansible_user }}" - group: "{{ ansible_user }}" - mode: '0644' -- name: Pull Mempool images - command: docker compose pull - args: - chdir: "{{ mempool_dir }}" - -- name: Deploy Mempool containers with docker compose - command: docker compose up -d - args: - chdir: "{{ mempool_dir }}" - -- name: Wait for MariaDB to be healthy - command: docker inspect --format='{{ '{{' }}.State.Health.Status{{ '}}' }}' mempool-db - register: mariadb_health - until: mariadb_health.stdout == 'healthy' - retries: 30 - delay: 10 - changed_when: false - -- name: Wait for Mempool backend to start - uri: - url: "http://localhost:{{ mempool_backend_port }}/api/v1/backend-info" - method: GET - status_code: 200 - timeout: 10 - register: backend_check - until: backend_check.status == 200 - retries: 30 - delay: 10 - ignore_errors: yes - -- name: Wait for Mempool frontend to be available - uri: - url: "http://localhost:{{ mempool_frontend_port }}" - method: GET - status_code: 200 - timeout: 10 - register: frontend_check - until: frontend_check.status == 200 - retries: 20 - delay: 5 - ignore_errors: yes - -# ═════════════════════════════════════════════════════════════════════════ diff --git a/ansible/roles/mempool/tasks/docker.yml b/ansible/roles/mempool/tasks/docker.yml deleted file mode 100644 index b97e63e..0000000 --- a/ansible/roles/mempool/tasks/docker.yml +++ /dev/null @@ -1,72 +0,0 @@ ---- -- name: Remove old Docker-related packages - apt: - name: - - docker.io - - docker-doc - - docker-compose - - podman-docker - - containerd - - runc - state: absent - purge: yes - autoremove: yes - -- name: Update apt cache - apt: - update_cache: yes - -- name: Install prerequisites - apt: - name: - - ca-certificates - - curl - state: present - -- name: Create directory for Docker GPG key - file: - path: /etc/apt/keyrings - state: directory - mode: '0755' - -- name: Download Docker GPG key - get_url: - url: https://download.docker.com/linux/debian/gpg - dest: /etc/apt/keyrings/docker.asc - mode: '0644' - -- name: Get Debian architecture - command: dpkg --print-architecture - register: deb_arch - changed_when: false - -- name: Add Docker repository - apt_repository: - repo: "deb [arch={{ deb_arch.stdout }} signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian {{ ansible_distribution_release }} stable" - filename: docker - state: present - update_cache: yes - -- name: Install Docker packages - apt: - name: - - docker-ce - - docker-ce-cli - - containerd.io - - docker-buildx-plugin - - docker-compose-plugin - state: present - update_cache: yes - -- name: Ensure Docker is started and enabled - systemd: - name: docker - enabled: yes - state: started - -- name: Add user to docker group - user: - name: "{{ ansible_user }}" - groups: docker - append: yes - diff --git a/ansible/roles/mempool/tasks/healthcheck.yml b/ansible/roles/mempool/tasks/healthcheck.yml deleted file mode 100644 index 1e1ac28..0000000 --- a/ansible/roles/mempool/tasks/healthcheck.yml +++ /dev/null @@ -1,58 +0,0 @@ ---- -# Three checks, one per moving part. The Uptime Kuma specifics that used to -# follow — an embedded Python script creating monitors over the API, a /tmp -# credentials file, a push-URL file read back and parsed, and three systemd -# Environment= rewrites — are gone. Where each reports is now hc.push_url. -- name: Create Mempool health check scripts - ansible.builtin.template: - src: "healthcheck-{{ hc.name }}.sh.j2" - dest: "/usr/local/bin/mempool-{{ hc.name }}-healthcheck-push.sh" - owner: root - group: root - mode: '0755' - validate: "bash -n %s" - loop: "{{ mempool_healthchecks }}" - loop_control: - loop_var: hc - label: "{{ hc.name }}" - -- name: Create systemd services for health checks - ansible.builtin.template: - src: healthcheck.service.j2 - dest: "/etc/systemd/system/mempool-{{ hc.name }}-healthcheck.service" - owner: root - group: root - mode: "0600" - loop: "{{ mempool_healthchecks }}" - loop_control: - loop_var: hc - label: "{{ hc.name }}" - -- name: Create systemd timers for health checks - ansible.builtin.template: - src: healthcheck.timer.j2 - dest: "/etc/systemd/system/mempool-{{ hc.name }}-healthcheck.timer" - owner: root - group: root - mode: '0644' - loop: "{{ mempool_healthchecks }}" - loop_control: - loop_var: hc - label: "{{ hc.name }}" - -- name: Reload systemd daemon - systemd: - daemon_reload: yes - -# Ungated on purpose: enabling a timer is deployment, not monitoring. The -# deprecation banner swept this up with the push plumbing, so Ansible stopped -# managing three timers that are in fact running on the host. -- name: Enable and start health check timers - systemd: - name: "mempool-{{ hc.name }}-healthcheck.timer" - enabled: yes - state: started - loop: "{{ mempool_healthchecks }}" - loop_control: - loop_var: hc - label: "{{ hc.name }}" diff --git a/ansible/roles/mempool/tasks/main.yml b/ansible/roles/mempool/tasks/main.yml deleted file mode 100644 index 9feb652..0000000 --- a/ansible/roles/mempool/tasks/main.yml +++ /dev/null @@ -1,6 +0,0 @@ ---- -# import_tasks, not include_tasks: static imports stay visible to --list-tasks, -# which is how this conversion was verified against the playbook it replaced. -- ansible.builtin.import_tasks: docker.yml -- ansible.builtin.import_tasks: deploy.yml -- ansible.builtin.import_tasks: healthcheck.yml diff --git a/ansible/roles/mempool/templates/docker-compose.yml.j2 b/ansible/roles/mempool/templates/docker-compose.yml.j2 deleted file mode 100644 index fda2ad1..0000000 --- a/ansible/roles/mempool/templates/docker-compose.yml.j2 +++ /dev/null @@ -1,75 +0,0 @@ -# All containers use host network for Tailscale MagicDNS resolution -services: - mariadb: - image: mariadb:10.11 - container_name: mempool-db - restart: unless-stopped - network_mode: host - environment: - MYSQL_DATABASE: "{{ mariadb_database }}" - MYSQL_USER: "{{ mariadb_user }}" - MYSQL_PASSWORD: "{{ mariadb_mempool_password }}" - MYSQL_ROOT_PASSWORD: "{{ mariadb_mempool_password }}" - volumes: - - {{ mempool_mysql_dir }}:/var/lib/mysql - healthcheck: - test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"] - interval: 10s - timeout: 5s - retries: 5 - start_period: 30s - - mempool-backend: - image: mempool/backend:{{ mempool_version }} - container_name: mempool-backend - restart: unless-stopped - network_mode: host - environment: - # Database (localhost since all containers share host network) - DATABASE_ENABLED: "true" - DATABASE_HOST: "127.0.0.1" - DATABASE_DATABASE: "{{ mariadb_database }}" - DATABASE_USERNAME: "{{ mariadb_user }}" - DATABASE_PASSWORD: "{{ mariadb_mempool_password }}" - # Bitcoin Core/Knots (via Tailnet MagicDNS) - CORE_RPC_HOST: "{{ bitcoin_host }}" - CORE_RPC_PORT: "{{ bitcoin_rpc_port }}" - CORE_RPC_USERNAME: "{{ bitcoin_rpc_user }}" - CORE_RPC_PASSWORD: "{{ bitcoin_rpc_password }}" - # Electrum (Fulcrum via Tailnet MagicDNS) - ELECTRUM_HOST: "{{ fulcrum_host }}" - ELECTRUM_PORT: "{{ fulcrum_port }}" - ELECTRUM_TLS_ENABLED: "{{ fulcrum_tls }}" - # Mempool settings - MEMPOOL_NETWORK: "{{ mempool_network }}" - MEMPOOL_BACKEND: "electrum" - MEMPOOL_CLEAR_PROTECTION_MINUTES: "20" - MEMPOOL_INDEXING_BLOCKS_AMOUNT: "52560" - volumes: - - {{ mempool_data_dir }}:/backend/cache - depends_on: - mariadb: - condition: service_healthy - healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:8999/api/v1/backend-info"] - interval: 30s - timeout: 10s - retries: 3 - start_period: 60s - - mempool-frontend: - image: mempool/frontend:{{ mempool_version }} - container_name: mempool-frontend - restart: unless-stopped - network_mode: host - environment: - FRONTEND_HTTP_PORT: "{{ mempool_frontend_port }}" - BACKEND_MAINNET_HTTP_HOST: "127.0.0.1" - depends_on: - - mempool-backend - healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:{{ mempool_frontend_port }}"] - interval: 30s - timeout: 10s - retries: 3 - start_period: 30s diff --git a/ansible/roles/mempool/templates/healthcheck-backend.sh.j2 b/ansible/roles/mempool/templates/healthcheck-backend.sh.j2 deleted file mode 100644 index fe0f57c..0000000 --- a/ansible/roles/mempool/templates/healthcheck-backend.sh.j2 +++ /dev/null @@ -1,25 +0,0 @@ -#!/bin/bash -# Mempool backend health check — managed by Ansible (roles/mempool) -# The exit code is the answer; systemd keeps it. Reporting is optional. -PUSH_URL="${HEALTHCHECK_PUSH_URL:-}" -PUSH_TOKEN="${HEALTHCHECK_PUSH_TOKEN:-}" -BACKEND_PORT="{{ mempool_backend_port }}" - -check() { - curl -sf --max-time 5 "http://localhost:${BACKEND_PORT}/api/v1/backend-info" > /dev/null 2>&1 -} - -report() { - [ -n "$PUSH_URL" ] || return 0 - # Gatus external endpoint: a POST with a bearer token, NOT Uptime Kuma's - # GET with ?status=up. The callers still pass up/down, so the mapping is - # done here rather than at every call site. - local _ok=false - [ "$1" = "up" ] && _ok=true - curl -s --max-time 15 --retry 2 -o /dev/null -X POST \ - -H "Authorization: Bearer ${PUSH_TOKEN}" \ - "${PUSH_URL}?success=${_ok}&error=${2// /%20}" || true -} - -if check; then report up "OK"; exit 0 -else echo "Mempool backend not responding"; report down "Mempool backend not responding"; exit 1; fi diff --git a/ansible/roles/mempool/templates/healthcheck-frontend.sh.j2 b/ansible/roles/mempool/templates/healthcheck-frontend.sh.j2 deleted file mode 100644 index 8452202..0000000 --- a/ansible/roles/mempool/templates/healthcheck-frontend.sh.j2 +++ /dev/null @@ -1,25 +0,0 @@ -#!/bin/bash -# Mempool frontend health check — managed by Ansible (roles/mempool) -# The exit code is the answer; systemd keeps it. Reporting is optional. -PUSH_URL="${HEALTHCHECK_PUSH_URL:-}" -PUSH_TOKEN="${HEALTHCHECK_PUSH_TOKEN:-}" -FRONTEND_PORT="{{ mempool_frontend_port }}" - -check() { - curl -sf --max-time 5 "http://localhost:${FRONTEND_PORT}" > /dev/null 2>&1 -} - -report() { - [ -n "$PUSH_URL" ] || return 0 - # Gatus external endpoint: a POST with a bearer token, NOT Uptime Kuma's - # GET with ?status=up. The callers still pass up/down, so the mapping is - # done here rather than at every call site. - local _ok=false - [ "$1" = "up" ] && _ok=true - curl -s --max-time 15 --retry 2 -o /dev/null -X POST \ - -H "Authorization: Bearer ${PUSH_TOKEN}" \ - "${PUSH_URL}?success=${_ok}&error=${2// /%20}" || true -} - -if check; then report up "OK"; exit 0 -else echo "Mempool frontend not responding"; report down "Mempool frontend not responding"; exit 1; fi diff --git a/ansible/roles/mempool/templates/healthcheck-mariadb.sh.j2 b/ansible/roles/mempool/templates/healthcheck-mariadb.sh.j2 deleted file mode 100644 index 922adab..0000000 --- a/ansible/roles/mempool/templates/healthcheck-mariadb.sh.j2 +++ /dev/null @@ -1,28 +0,0 @@ -#!/bin/bash -# Mempool MariaDB health check — managed by Ansible (roles/mempool) -# The exit code is the answer; systemd keeps it. Reporting is optional. -PUSH_URL="${HEALTHCHECK_PUSH_URL:-}" -PUSH_TOKEN="${HEALTHCHECK_PUSH_TOKEN:-}" - -check() { -{% raw %} - [ "$(docker inspect --format='{{.State.Health.Status}}' mempool-db 2>/dev/null)" = "healthy" ] -{% endraw %} -} - -report() { - # No push URL is normal, not an error. The previous version logged - # "ERROR: UPTIME_KUMA_PUSH_URL not set" on every fire, once a minute. - [ -n "$PUSH_URL" ] || return 0 - # Gatus external endpoint: a POST with a bearer token, NOT Uptime Kuma's - # GET with ?status=up. The callers still pass up/down, so the mapping is - # done here rather than at every call site. - local _ok=false - [ "$1" = "up" ] && _ok=true - curl -s --max-time 15 --retry 2 -o /dev/null -X POST \ - -H "Authorization: Bearer ${PUSH_TOKEN}" \ - "${PUSH_URL}?success=${_ok}&error=${2// /%20}" || true -} - -if check; then report up "OK"; exit 0 -else echo "MariaDB container unhealthy"; report down "MariaDB container unhealthy"; exit 1; fi diff --git a/ansible/roles/mempool/templates/healthcheck.service.j2 b/ansible/roles/mempool/templates/healthcheck.service.j2 deleted file mode 100644 index 76fa4e9..0000000 --- a/ansible/roles/mempool/templates/healthcheck.service.j2 +++ /dev/null @@ -1,15 +0,0 @@ -[Unit] -Description=Mempool {{ hc.label }} Health Check -After=network.target docker.service - -[Service] -Type=oneshot -User=root -ExecStart=/usr/local/bin/mempool-{{ hc.name }}-healthcheck-push.sh -Environment=HEALTHCHECK_PUSH_URL={{ hc.push_url }} -Environment=HEALTHCHECK_PUSH_TOKEN={{ healthcheck_push_token }} -StandardOutput=journal -StandardError=journal - -[Install] -WantedBy=multi-user.target diff --git a/ansible/roles/mempool/templates/healthcheck.timer.j2 b/ansible/roles/mempool/templates/healthcheck.timer.j2 deleted file mode 100644 index 868cb7c..0000000 --- a/ansible/roles/mempool/templates/healthcheck.timer.j2 +++ /dev/null @@ -1,10 +0,0 @@ -[Unit] -Description=Mempool {{ hc.name }} Health Check Timer - -[Timer] -OnBootSec=2min -OnUnitActiveSec=1min -Persistent=true - -[Install] -WantedBy=timers.target diff --git a/ansible/roles/monero/README.md b/ansible/roles/monero/README.md deleted file mode 100644 index df9033d..0000000 --- a/ansible/roles/monero/README.md +++ /dev/null @@ -1,156 +0,0 @@ -# monero - -A Monero full node (`monerod`) on Debian 13, bare metal, systemd, no Docker. -Runs on `monerod-box` (Proxmox VMID 2400), reachable from Feather over the WAN. - -## Why the published binary and not a build from source - -The intent was to build from source and check the result against the published -binary checksums. That is not achievable for the current release, and it is -worth writing down why so nobody re-litigates it: - -- `contrib/guix`, the reproducible-build system that produces a byte-identical - tarball, **exists only on monero `master`**. At tag `v0.18.5.1` `contrib/` - contains `gitian`, not `guix`. -- Reproducing v0.18.5.1 therefore means **Gitian**, whose README requires an - Ubuntu 18.04 host (EOL 2023), LXC or Docker, and `python-cheetah` (Python 2). -- A plain `make release-static` is **not** reproducible. It would give us a - self-built binary whose hash matches nothing - not upstream's, not anyone - else's - so it satisfies neither half of "build it ourselves AND check the - checksum". - -So the role installs the published binary and proves its provenance instead. -That chain is real, and it is the strongest claim available for this release: - -1. binaryFate's public key is **vendored** at `files/binaryfate.asc`. A trust - root fetched over the network at the moment you need it is not a trust root. - Its fingerprint is asserted after import anyway - (`81AC591FE9C4B65C5806AFC3F0AF4D462A0BDF92`), so a swapped file is caught. -2. `hashes.txt` is downloaded and its clearsigned signature verified. The - assertion is on gpg's `VALIDSIG ` status line, **not** on the - string "Good signature" - `bitcoin_knots` greps for the latter, which passes - for *any* key in the keyring. -3. The verified plaintext is written out by `gpg --decrypt`, and the tarball's - SHA256 is looked up in *that*, not in the raw clearsigned file. -4. The lookup requires the hash **and** the filename on one line, so a hash - belonging to a different artefact cannot satisfy it. - -`monero_sha256` in `defaults/` is a second, in-git assertion passed to -`get_url`. It is not the security control - step 4 is - but it means a version -bump cannot happen without the checksum moving in the same diff. - -**Bumping the version: change `monero_version` and `monero_sha256` together.** - -### The forward path - -The published `monero-linux-x64-v0.18.5.1.tar.bz2` unpacks to -`monero-x86_64-linux-gnu-v0.18.5.1/` - that is the Guix `DISTNAME`, renamed at -publish time. So once v0.19 ships with Guix on the release branch, -`sha256(guix output) == hashes.txt entry` holds exactly, and a `guix` install -method can be added here without changing anything else. - -## The data disk - -The blockchain is on a **dedicated 500 GB disk** (`scsi1`), not on the 10 GB -root filesystem. `tasks/storage.yml` creates the filesystem and the fstab entry. - -This is done in code on purpose. `knots-box` mounts `/mnt/knots_data` (~875 GB -of chain) entirely by hand, and that mount appears nowhere in Ansible - a -rebuild of that host would come up with the chain silently missing. - -Three things there are load-bearing: - -- **The whole device, no partition table.** One disk, one job; a partition table - is just another layer that can disagree with itself. -- **The device name is read once, ever.** Each run first looks for a filesystem - labelled `monero-data`; if it exists, `monero_data_device` is not consulted - and the mount goes by that filesystem's UUID. Only a blank disk is found by - name, and that name is the Proxmox slot as published under `/dev/disk/by-id` - (`...drive-scsi1`), fixed before the VM boots and kept across `qm disk move`. - Never `/dev/sdX`: the kernel assigns those in probe order, and on 2026-10-04 a - reboot swapped `sda` and `sdb`, leaving the old `/dev/sdb` value naming the - root disk. The root-disk guard caught it - but a routine run should never - have been reading the name in the first place. -- **`RequiresMountsFor=` in the unit.** `nofail` in fstab means a missing disk - does not drop a headless VM into an emergency shell - but without - `RequiresMountsFor` monerod would then start, find an empty directory, and - begin writing a ~300 GB chain onto a 10 GB root disk. With it, the unit - refuses to start and `infra/401` reports it within five minutes. - -The mount-point task and the post-mount ownership task deliberately use the -**same** owner and mode. Writing the obvious `root:root 0755` on the first made -the two disagree, and every run flipped the directory back and forth. - -## Exposure: one HTTPS vhost, no new open ports - -- **P2P is outbound only.** Nothing forwards 18080 to this box. The node dials - out, syncs fully and serves us; it just does not relay for strangers. - `no-igd=1` follows - with nothing forwarded there is nothing for UPnP to do - except announce us to the router. -- **Two RPC servers, not one.** - - unrestricted on `127.0.0.1:18081` - administration and the health check. - - restricted on `:18089` - the only surface anything off-box touches. - - This is `rpc-restricted-bind-port`, deliberately **not** `restricted-rpc=1`. - That flag restricts the loopback server too, and then we could not ask our own - node anything useful. Verified live: the restricted port reports - `restricted=true` and blanks `version` and `start_time`. -- **`rpc-login` is HTTP Digest** (`contrib/epee/src/http_auth.cpp`), and - `core_rpc_server::init` hands the same login to *both* servers. Both ports - answer an unauthenticated request with 401. -- **Caddy on the edge host terminates TLS and nothing else.** No - `caddy_site_basic_auth`: Caddy's Basic challenge would collide with the Digest - challenge Feather expects. monerod's own auth is the authentication. - -### No UFW rule, and that is correct - -Traffic arriving over the tailnet **bypasses UFW entirely**. Verified on -`mempool-box`: UFW is active with default-deny-incoming and `22/tcp` as its only -allow rule, yet the edge host reaches its frontend port - because -`iptables -S INPUT` puts `-A INPUT -j ts-input` ahead of every `ufw-*` chain. -Adding a UFW allow here would be cargo cult. - -## Monitoring - -Four Gatus endpoints, all verified passing: - -| Endpoint | Source | Answers | -|---|---|---| -| `liveness/monerod_box_local` | `infra/400` | is the host up | -| `disk/monerod_box_local` | `infra/400` | is it running out of space | -| `services/monerod-box-local/monerod` | `infra/401` via `monitored_services` | is the unit active | -| `probe/monerod` | this role + `infra/403` | does monerod actually answer RPC | - -**The probe deliberately does not check `synchronized`.** A full initial sync -takes days. A check requiring `synchronized: true` would report DOWN for every -one of those days, which is not a monitor - it is an alarm with the batteries -taken out. Answering RPC is the health question; height and sync state go in the -message so they are visible without being alertable. - -The timer carries **no `Requires=monerod.service`**. Requires on a timer stops -the timer when the watched unit stops - "if the thing I am watching goes down, -stop watching it" - and leaves nothing to re-arm it. Fulcrum's timer carried it -and sat `active`/`enabled` for seven months without firing once. `bitcoin_knots` -still carries it; do not copy that. - -`Enable and restart the monerod health check timer` reports `changed` on every -run. That is intentional and matches `datum_gateway`: `state: restarted` is how -a timer gets re-armed, and `started` is a no-op on an already-active timer. - -## Backups: none, deliberately - -The blockchain is re-syncable public data and there is no wallet on this node, -so a backup job would move hundreds of GB around to protect nothing. This -follows the `phoenixd` precedent (Plan 5, Model C) of recording an exclusion as -a decision rather than leaving it to look like an oversight. - -## Known constraint: RAM - -The VM has **2 GB and no swap**. Mid-sync, monerod's RSS was already 1,333 MiB -of 1,973 MiB with ~100 MiB free. No OOM kill has happened yet, but the working -set grows with the database and there is no swap to absorb a spike. `out-peers` -and `in-peers` are held down (16/8) partly for this reason. - -**8 GB is the recommendation.** Bumping it means editing `memory_mb` in -`tofu/nodito/vms.auto.tfvars` and resizing the VM in the Proxmox UI - the vm -resource sets `ignore_changes = all`, so tofu will not do it for you. diff --git a/ansible/roles/monero/defaults/main.yml b/ansible/roles/monero/defaults/main.yml deleted file mode 100644 index 686b1c9..0000000 --- a/ansible/roles/monero/defaults/main.yml +++ /dev/null @@ -1,120 +0,0 @@ -# Monero node (monerod) configuration variables. -# https://docs.getmonero.org/interacting/monerod-reference/ - -# --- Version and artefact --------------------------------------------------- -# Bump monero_version and monero_sha256 TOGETHER. The sha256 here is a second, -# in-git assertion; the one that actually matters is the grep against the -# GPG-signed hashes.txt in install.yml. Having both means a version bump cannot -# be done silently - the checksum has to move in the same diff. -monero_version: "0.18.5.1" -monero_platform: "linux-x64" -monero_tarball: "monero-{{ monero_platform }}-v{{ monero_version }}.tar.bz2" -monero_url: "https://downloads.getmonero.org/cli/{{ monero_tarball }}" -monero_sha256: "22a7dda7b0cb699fdd6b7674c3b4a4465b337cc98a54983523b759e1e7cc9958" - -# The signed list of checksums for every published Monero artefact. -monero_hashes_url: "https://www.getmonero.org/downloads/hashes.txt" - -# binaryFate, the release signing key. The .asc is vendored in files/ rather -# than fetched at deploy time: a trust root downloaded over the network at the -# moment you need it is not a trust root. The fingerprint is asserted after -# import anyway, so a swapped file cannot go unnoticed. -monero_signing_fingerprint: "81AC591FE9C4B65C5806AFC3F0AF4D462A0BDF92" - -# --- Service user ----------------------------------------------------------- -monero_user: monero -monero_group: monero - -# --- Directories ------------------------------------------------------------ -monero_conf_dir: /etc/monero -monero_log_dir: /var/log/monero -monero_bin_dir: /usr/local/bin - -# The blockchain. This is the mount point of the dedicated 500 GB disk, NOT a -# directory on the 10 GB root filesystem - see storage.yml. -monero_data_dir: /var/lib/monero - -# --- The data disk ---------------------------------------------------------- -# The whole device, no partition table: this disk has one job and a partition -# table would only add a layer that can disagree with itself. -# -# The disk is known by two names, for two different moments: -# -# monero_data_label is its identity once it holds a filesystem. Every run looks -# for it first, and if it is there the device name is never consulted at all - -# a routine run cannot be broken by a device name, because it does not read one. -# -# monero_data_device is used ONLY on the first run, to find the blank disk and -# create that filesystem. It must be a name fixed before the VM boots: Proxmox -# passes the slot ("scsi1") in as the disk's serial, and udev publishes it under -# /dev/disk/by-id. That survives reboots and `qm disk move` alike. NEVER -# /dev/sdX: the kernel hands those out in probe order, and on 2026-10-04 a reboot -# after moving this disk to the SSD swapped sda and sdb - the value written here -# then named the ROOT disk. -monero_data_label: monero-data -monero_data_device: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1 -monero_data_fstype: ext4 -# -m 0: the 5% ext4 reserve exists to stop root-owned daemons deadlocking a full -# root filesystem. On a dedicated data disk it is 25 GB thrown away. -monero_data_mkfs_opts: "-m 0 -L {{ monero_data_label }}" - -# --- Network ---------------------------------------------------------------- -# P2P is OUTBOUND ONLY. No inbound port is published from the edge host, so the -# node dials out to peers and never receives. It syncs and serves us completely; -# it just does not relay for strangers. no-igd follows from that: with nothing -# forwarded there is nothing for UPnP to do but tell the router about us. -monero_p2p_port: 18080 - -# Unrestricted RPC, loopback only: administration and the health check. -monero_rpc_bind_ip: "127.0.0.1" -monero_rpc_port: 18081 - -# The RESTRICTED RPC is a second server on its own port, reachable over the -# tailnet and published by Caddy on the edge host. This is deliberately -# rpc-restricted-bind-port and NOT `restricted-rpc=1` - the latter would -# restrict the loopback server too, and then we could not ask our own node -# anything useful. -# -# The authoritative value for the live deployment is in -# host_vars/monerod_box_local/main.yml, which the edge play also reads; the -# value here is the upstream default so the role still works standalone. -monero_restricted_rpc_port: 18089 - -# Peer counts kept modest on purpose: this VM has 2 GB of RAM. See README. -monero_out_peers: 16 -monero_in_peers: 8 - -# --- RPC credentials -------------------------------------------------------- -# monerod's --rpc-login is HTTP Digest (contrib/epee/src/http_auth.cpp) and the -# same login guards BOTH the loopback and the restricted server. Feather sends -# these as the node username/password. -monero_rpc_user: "" -monero_rpc_password: "" - -# monerod.conf holds the RPC login, so --diff is suppressed by default. -# Pass -e monero_reveal_config=true to see what moved. -monero_reveal_config: false - -# --- Health check ----------------------------------------------------------- -# WHERE TO REPORT HEALTH — the one place to plug in monitoring. Empty means -# check, exit honestly, report nowhere. Any endpoint accepting an HTTP ping -# works; nothing here is specific to a monitoring product. -healthcheck_push_url: "" -# Bearer token for the Gatus external endpoint. Required whenever a push URL -# is set: Gatus rejects an unauthenticated push with 401. -healthcheck_push_token: "" - -# WHEN TO CHECK. Every 5 minutes, on the calendar rather than OnUnitActiveSec so -# the cadence does not drift with every restart. (It was daily while the node -# did its initial sync on a rotating disk; since 2026-10-03 the chain is on NVMe -# and fully synced, and get_info answers in well under a second.) -# -# Prefixed because the generic `healthcheck` role owns the unprefixed -# healthcheck_on_calendar; an unprefixed default here would shadow it for any -# play that includes both. -# -# Paired with the standard 11m Gatus heartbeat in -# infra/403_service_probe_registration.yml. Change one and you must change the -# other: the window has to stay above the interval or the check alerts on its -# own jitter. -monero_healthcheck_on_calendar: "*:0/5" diff --git a/ansible/roles/monero/files/binaryfate.asc b/ansible/roles/monero/files/binaryfate.asc deleted file mode 100644 index 55c5b16..0000000 --- a/ansible/roles/monero/files/binaryfate.asc +++ /dev/null @@ -1,87 +0,0 @@ ------BEGIN PGP PUBLIC KEY BLOCK----- - -mQINBF3yDrwBEAC1UgCSLILsbdrSk5kfcEYKMvj7lJpIIj9D6LeeyIvovgO7beM0 -63cFCT0v+RH3CVKV8bCCJr8teR3Zgk+IeI6C0CQk+ocqlu0qBAALdZyGyZonozbc -lHGOfQ0rWEy01V/TB36bGhrsE/cM8nhICJ72Pkv3rrukZkprxvEJ+IYCk26Umiue -K1+Pm0sUMrxAQUYlvg8r1swOgLOuo7r8c1gZYvGixdb0t7mBzUgkSdmFUeAa/X8W -WzBPFluWMyetGUKzrV66W1ISHHi/2AyXim235Lqc4MbK2ObKfkZJCjC7y2afs5MS -t+uejz8bchLMM/LvV2TxKIbenho7ZxtGd8blNRAPe6FTOA+yOM50atJah1W3BmLx -sxk4gvII2/zlzaW4RNEy1Ma/47DINPwYB9BA7FqF7BTVt8WxpJ5Y2+8aR58ZtM+q -ObbO5s2O8kuDj94qQKAT8btetbb/pMKhF3XXSARkNZPzNFFtSy9xSin+hLAWpM0b -cftEtJrE7HY7DHOa9J4P/xFqLSQnZGpClg5lyRw34kU1l8sAzovFngN2Zn+RYkiW -y14I+uhVOFzroH2ymIYTwQ0crQJ4OYgtqzv0Rc+U2mTrTE+MRG28COsBqoMvWao9 -K4bhsP6qUmlTtMVFob4AP3eFqF3wg987zqeeCqaidAc7n5fuJHTDGNWFxQARAQAB -tCViaW5hcnlGYXRlIDxiaW5hcnlmYXRlQGdldG1vbmVyby5vcmc+iQJOBBMBCgA4 -FiEEgaxZH+nEtlxYBq/D8K9NRioL35IFAl3yDrwCGy8FCwkIBwIGFQoJCAsCBBYC -AwECHgECF4AACgkQ8K9NRioL35IT2w/+JaVcSVo3vXTpC9TOSxnyjFP1QuPVxn+K -P4la6MzTQlaZOSaoGuXra+82ghWqJGbFhGYmsXWQZtUYVwTMq9N9IUJ55clfkiV3 -HQNIDh/6zz+ibBzxfNLAPTbz0OQ+26dWRmBKkjpIs8KWyVIR/Ma/ax1No20fPKoN -cgRvOct/BKNoX2+PlHh/0tTTIjc0NqNP79ptNyJUUTGL1uZOgfTGa7/FRxil1xiC -JZfbZ2mbdesqjTtF3CI/ahhPKUEKy5Tciy5pnErALPk4rsxdLNdl3NOVrb5tYqGv -GqoQT3KsHkYVHvLCx0Ji2peFa6lF0PyJpudUzlo7L3nn39b8v9ebKd/fk2gthnWr -eoFxEFa6dI0x+HA47FE00I7ze56oH0zFVUhx+3QOHFG3u2mHV+H+GfKaXq9Nd5nU -Wk8AGKJ/xYgX7G0TnUek9aOzSt3dRd+xtSjmgwA3T7vCi6F67SthRpq7F4zuecPl -+Ww1MQhXSdIi6P3ll09gTCOICaqOxuOkMeFFrs2w4XEU517QUzy1VTAcF3gSy9TL -Xpy5jqssjLjvDAHs/Jm+iryQtn/t9YHr7/DP7AAr4+e7aOY3659jP7UiMZ3RKqY5 -/15Ru/usXXFyC56Epsv1spBCthimiK/RdX8XW7mG+BIQJXN337WdicXGWrKVOixj -eJ7ghiSblIeJAjMEEAEKAB0WIQTsEs+GtgjVvRuRCZGpI7HrTHj9WAUCXfOqVwAK -CRCpI7HrTHj9WPSaD/wMa2PWbRj06DUHiKRMrrQF9gkzox2ZOd4om4YIa3opA+GJ -CRl6nMHO7MdGTiun0/jBoT4oU4rqvPYcQlKOXUFYaMz/3vPYyTotC8n1Rvf+DhQZ -CIubdrRW0B5pqa3MiyJ9S/oMf/5otto3eBZSnfegI0J/fNzyDAyS2WIj/iyoAXGQ -yy9lY+KQnjchIHItxFm6RWgZmbO6xbhRq+LUa90jEiSul1PoE+ldpH2wUlRvjP78 -lL4YqWC7U6puhgGPlQV2gvFG4zscXVf8XXV65KysqrB5RI8wx1JxC1n1ocOqwYbw -0P6WIUGSr98cBjzB1l9zKhHumV7AM9FNUf4GD+Mgl8I8i/vOSz8kYRaWCcoRZvWa -pgSEG4uA94MFZUP9b789quoMUv8kTboHU1XFDo/n7K83AVBThcj0BMt7ueZmwjtu -2YVc3vGX7GvsUr8XXYn82hqnHcBoVLI4awGMzzAiOI1KSPtksGr36mYgOuY1jYme -435YlVxH4A/NC1qvOZYLq/6u1pB/BouBtlVxCvyjhlM3/r0PgDB0KveiR72LzVXH -TB0CzmyizZnVYbNR1PvsxFvyPxRLIIfN1nzXFYxwILASlg0zhdkQgFWkZ+WJWA6g -6HrAmrrA8iV7x6ITR6dXrT2fSLdQ+nYTntaXunUw74gGWYaMPe/GTvgcguFoU4kC -MwQQAQoAHRYhBOyzccBByW6YhVJsYUZMOzgUXzoUBQJd8647AAoJEEZMOzgUXzoU -fvgP/jI1w6DxC4qRlEwsIpjvyKQbCWYr9w817gKRPOI0ZHbjoL2zBnBwAWvgEKXV -QTOkldwpWVHvt5qZweO89bJxQFShFdQklEbVXNlf9hMZqbWHt6gyS/65caEUT/nB -pugg6Ug3MIeszin7S2sKeKj3BT2ynJioMeNNUPntogUfWlOuhkVr3JezHV3TtxKf -yrs49AAmoAHK4DH34H7d8HuVjjr0v7gQc2Yu3MeWdR4drGmyU3uNY44fGqU9pP99 -HNW2Ec//HFIz+qgBZ63ps6qyWGOjXyrpn6k9L1lg8zilC1sJxRtYTwp1a/57Kpyu -NkR1265o2zu/cvb5oQxjCWmQPiZWKsoguHxiA93g14QdmnugHPM5L6XCqDl1fkQo -dnCy5uqlqs9Ucz6O4C1GV4Av3YsMepS2Bw5uZFyujqPUdw82oWWOwZfO2xw7kKzN -zd8OnQAKj3piMh1vh375HE+HwdN74txKDcD/HqoTMwuaqX145csUvORoABA/wU4w -yBJiOhJeqvUaWAsG0q6XQMIDYjGQ0rrtY6Ba9E/1UP3D2CpjH5fZChV1fxhCaQOi -nMSCwgA3HRiT4CnTYScsbnk7RcdOlc2Zff+cxyZfMvqcGM4R50NDYu8YnVjnyajn -No5WhvYYPQIoYaDM8b+RHGw8Zw4WpQqAqli0VoNJAux19os9uQINBF3yDrwBEADI -LMleZnQ87iFofqyMm6wd+146dPC0xINz3+ExkCcFJgwiHL4o1sF8LtXjBXVuoc33 -vK7mdU1Fm3N9D4W5tkxEW3NdHICc7r3IHqThFv6lOdckKg1t9HKWzEjWkB4/4Epr -sj7Xc1owuQkOVtkaWiXzv8e/pYM4j+21V71+8b6fuInvA1nNufawzN4m8RDDgbe9 -I9SdndJoO0vnhUrv55APVG6KJBRjT+SGrxkEa3cckhpGUsQkW8KEx1VH0Wc4NBdD -xSRzW+ZUDyMzoUD/b5kxNCqinaM3P95jmNJgqB9l/3ZEPZ0G/hqDg6EhQMe8HrYT -F90Q+QqsX35RbBrdXxjbod+GZPSTg0PExm/hBrt2tTE/t/yovn22PZeJm6Mg9trN -x08b/LLZ1L040s8WqQQUK0btG065PIVRpyIlsUEPt7W+Qdj+eJdZAjIDr04qBxUQ -bwHhnxCYH4PNQq75y/w684ZWoVW3UJE1P8AcXKx/0gp4jjrZCJTAkzfUBEc6vy6H -hzYLpAar52ALBmFGPpkV4PJDdlFX4uijsjVVf23Hi3AQnGWkUw34bftJuHYlluGg -aGXVzISn8MxqcaLUiapxMMg3pZ40m5gCHG7/Gm74xKZCQiNrxCLHn7/rgNV17U6m -aWCNGcaiCXfkbtqUPJrzK6ulDAhg3700Ok5RlVOtAwARAQABiQRsBBgBCgAgFiEE -gaxZH+nEtlxYBq/D8K9NRioL35IFAl3yDrwCGy4CQAkQ8K9NRioL35LBdCAEGQEK -AB0WIQStVkzajxZlrOeLXf0lk4OOq7H2VQUCXfIOvAAKCRAlk4OOq7H2VfMbD/oC -NxmlGhAJ2okXzcpFURNyIXTCt/Dlys+lwH2mwCjcniUcA5KsT1mV1Sb4SxsBsX18 -Fhol2wUUI/B3ELz15rOYTJq7u8TPpPnGsHtALgNwN1o90EYMRpMl7jiPJLKSWVA1 -spp7FBginaP/jkQwtcy+sHiJvmEiY2kmeB27aAA5bM8bafjzpqQ6Io/iguRVYex1 -ypH5D1Dq5bkrk8Fd+3bs2dcuB9jYy0lQlaYl8bJ4I4ZOgdO4nTKq0U0RTfYAR+wH -vpfzKrYQD7ZnX6mllXyvC+L4CujngND3GiavoOPX9JHJURrYC1WiMyKPb8nk1CrF -cIfLxFYqy535suen35T23G8B0ZY/i48ccRjI7ZrickIBSjJZmJLYevsejz+T1vqf -348WCAYisi44NL6vwAoBkvYLRw45YhzAZDZoKCv+ke7zjZAO3lnh+rR3j+QCe9gH -22gqJbhGpU54EpYYVx3GxIfaqQeHa5H02rSW8AUGyGjgdTo7x5NCAJjgtYyR08k1 -19aghaA8yi1wsbmiuz9viH2xJ1vPTgLLQE5qv0ed0uOGEeDtRZSfaWeYwk+qhier -bY3R3dzVOyan6W8uWaVgE8Ch9LttYAKa9qLoLDQE9tQLU6t1gklFfQiaLBzyPIgT -Ct6Aezwlcp2cDhxYazXjyxRTYtfshVPTXJMBF+0o+W78D/9c3f9Yv61Z1MktrHIt -5REIj5YGT3bAeDnUoGE+k+QzNx8856Hu5j4prQGav8F8+V69iqwVciK0sBoFUgI5 -JJizfVbEfmbZgl5c4S8OhApNv0Gbhj9U/XftM2Pz/m/QfsB5Az1jQ8jQGHULqEWB -d4Z+Cnt0LVmGxJkz0prlaUxrGp8NyO3A1RxWJbP6P9fPIJFFwDSer2xeq79dZGFQ -CLFxA3d8kjr9nioGZFaGK/LfbVcZBBM6VI0ziKRnWPeS3Zi50oy7+/gk7HgUvUmT -8uenHLE/kmfHg1lAycQ/TDY7/sP1Wtnbr72HEFOKDIvFIF8zaWwpaYSWrj6NRTJE -b9wa2Y5TervcvOcgzhleJP3PfDrahlfgUFtD/919cuQNG416tedumAaAlLpdBEUH -9F9FQ2Hzp8Pxm3N6LzbtgDoc6NXY2C08NPI+IwHysNIXzaH3jJ53TV8pCNvvd4Ok -Sk2fOVG/fDuZibwthAFjR3NIVJIP19xyxkalbeuLT+IUSzgN2ndojtK9eh7awMii -S/AkjI5G33OKYP0WZx+6o0i21rWXIepWgm74wMa9t9NCnNY50NIvesnG1AaXNKJ5 -D2o1uLXbwis6Fm2sMNutCkQFYsk/IFWC/Y8DsJzwkOty8gl3Q0NQi64IXpFGzUuG -lmE0kDaYQqoBKXNXcybElBID8Q== -=tkJZ ------END PGP PUBLIC KEY BLOCK----- diff --git a/ansible/roles/monero/handlers/main.yml b/ansible/roles/monero/handlers/main.yml deleted file mode 100644 index 15f7e63..0000000 --- a/ansible/roles/monero/handlers/main.yml +++ /dev/null @@ -1,17 +0,0 @@ ---- -# Ungated, and that is the whole point. Across this estate the restart handlers -# were written with `when: uptime_kuma_enabled | default(false)`, which made -# them inert: Fulcrum's config changes landed on disk and silently never took -# effect for months. Three tasks notify this handler - the config, the systemd -# unit and the binary itself - and all three need it to actually fire. -- name: Restart monerod - systemd: - name: monerod - state: restarted - daemon_reload: yes - -- name: Restart monerod health check timer - systemd: - name: monerod-healthcheck.timer - state: restarted - daemon_reload: yes diff --git a/ansible/roles/monero/tasks/configure.yml b/ansible/roles/monero/tasks/configure.yml deleted file mode 100644 index b8af394..0000000 --- a/ansible/roles/monero/tasks/configure.yml +++ /dev/null @@ -1,55 +0,0 @@ ---- -# Directory layout follows upstream's systemd guide: -# https://docs.getmonero.org/running-node/monerod-systemd/ -# The data directory is not here - it is the mount point, handled in storage.yml. - -- name: Create the Monero config directory - ansible.builtin.file: - path: "{{ monero_conf_dir }}" - state: directory - owner: root - group: "{{ monero_group }}" - mode: '0750' - -- name: Create the Monero log directory - ansible.builtin.file: - path: "{{ monero_log_dir }}" - state: directory - owner: "{{ monero_user }}" - group: "{{ monero_group }}" - mode: '0750' - -- name: Assert the RPC credentials are set - ansible.builtin.assert: - that: - - monero_rpc_user | length > 0 - - monero_rpc_password | length > 0 - quiet: true - fail_msg: >- - monero_rpc_user and monero_rpc_password must be set. They are the only - authentication in front of the restricted RPC that Caddy publishes to the - internet, and monerod started without them would serve it to anyone. - -- name: Create monerod.conf - ansible.builtin.template: - src: monerod.conf.j2 - dest: "{{ monero_conf_dir }}/monerod.conf" - owner: root - group: "{{ monero_group }}" - mode: '0640' - # monerod.conf carries the RPC login, which is the only authentication in - # front of the endpoint Caddy publishes. `--diff` prints rendered content, so - # running with --diff would put it on the terminal and into any log capturing - # it. Suppressed by default; pass -e monero_reveal_config=true when you - # genuinely need the diff. This is a task keyword, not a module parameter. - diff: "{{ monero_reveal_config | default(false) | bool }}" - notify: Restart monerod - -- name: Create the monerod systemd unit - ansible.builtin.template: - src: monerod.service.j2 - dest: /etc/systemd/system/monerod.service - owner: root - group: root - mode: '0644' - notify: Restart monerod diff --git a/ansible/roles/monero/tasks/healthcheck.yml b/ansible/roles/monero/tasks/healthcheck.yml deleted file mode 100644 index 794ccba..0000000 --- a/ansible/roles/monero/tasks/healthcheck.yml +++ /dev/null @@ -1,58 +0,0 @@ ---- -# "Is monerod actually answering", as opposed to infra/401's "is the unit -# active". A daemon can be perfectly `active` and wedged. - -- name: Install the health check's dependencies - apt: - name: - - curl - - jq - state: present - -- name: Create the monerod health check script - ansible.builtin.template: - src: healthcheck.sh.j2 - dest: /usr/local/bin/monerod-healthcheck-push.sh - owner: root - group: root - mode: '0755' - validate: "bash -n %s" - -# 0600: this unit carries both the Gatus bearer token and the RPC password. -# They live here rather than in the script so the script itself stays readable -# and credential-free. -- name: Create the monerod health check systemd service - ansible.builtin.template: - src: healthcheck.service.j2 - dest: /etc/systemd/system/monerod-healthcheck.service - owner: root - group: root - mode: '0600' - notify: Restart monerod health check timer - -- name: Create the monerod health check systemd timer - ansible.builtin.template: - src: healthcheck.timer.j2 - dest: /etc/systemd/system/monerod-healthcheck.timer - owner: root - group: root - mode: '0644' - notify: Restart monerod health check timer - -- name: Reload systemd daemon after health check units - systemd: - daemon_reload: yes - -# Ungated: enabling a timer is deployment, not monitoring. -- name: Enable and restart the monerod health check timer - systemd: - name: monerod-healthcheck.timer - enabled: yes - state: restarted - daemon_reload: yes - -# Arms the timer and smoke-tests the check. Restarting the timer alone is not -# enough with OnBootSec + OnUnitActiveSec - see roles/bitcoin_knots/README.md. -- name: Run the monerod health check once to arm the timer - ansible.builtin.command: systemctl start monerod-healthcheck.service - changed_when: false diff --git a/ansible/roles/monero/tasks/install.yml b/ansible/roles/monero/tasks/install.yml deleted file mode 100644 index 0175680..0000000 --- a/ansible/roles/monero/tasks/install.yml +++ /dev/null @@ -1,232 +0,0 @@ ---- -# Install monerod from the published release, and prove the bytes are the ones -# binaryFate signed before they are allowed anywhere near /usr/local/bin. -# -# These tasks are the security control of this role. Do not "simplify" them. -# -# Why the published binary and not a build from source: the reproducible-build -# system that yields a byte-identical tarball (contrib/guix) exists only on -# monero master. The v0.18.x release branch still uses Gitian, which wants an -# Ubuntu 18.04 host and Python 2. A plain `make release-static` is not -# reproducible, so it would give us a self-built binary whose hash matches -# nothing - neither upstream's nor anyone else's. The signature chain below is -# the strongest claim actually available for this release. See README.md. - -- name: Install the tools the verification chain needs - apt: - name: - - gnupg - - curl - - bzip2 - state: present - update_cache: yes - -- name: Check whether monerod is already installed - ansible.builtin.stat: - path: "{{ monero_bin_dir }}/monerod" - register: monero_binary - -- name: Read the installed monerod version - ansible.builtin.command: "{{ monero_bin_dir }}/monerod --version" - register: monero_installed_version - changed_when: false - failed_when: false - check_mode: false - when: monero_binary.stat.exists - -- name: Decide whether monerod needs installing - ansible.builtin.set_fact: - monero_needs_install: >- - {{ not monero_binary.stat.exists - or monero_version not in (monero_installed_version.stdout | default('')) }} - -# Everything below runs only for a fresh install or a version bump. - -# A throwaway keyring. The verification must not depend on, or add to, whatever -# happens to be in root's keyring already. -- name: Create a temporary GnuPG home for the verification - ansible.builtin.tempfile: - state: directory - suffix: monero-gpg - register: monero_gnupghome - when: monero_needs_install | bool - -- name: Copy binaryFate's signing key to the host - ansible.builtin.copy: - src: binaryfate.asc - dest: "{{ monero_gnupghome.path }}/binaryfate.asc" - mode: '0644' - when: monero_needs_install | bool - -- name: Import binaryFate's signing key - ansible.builtin.command: "gpg --batch --import {{ monero_gnupghome.path }}/binaryfate.asc" - environment: - GNUPGHOME: "{{ monero_gnupghome.path }}" - register: monero_key_import - changed_when: "'imported' in monero_key_import.stderr" - when: monero_needs_install | bool - -- name: Read the fingerprints of the imported key - ansible.builtin.shell: >- - set -o pipefail && - gpg --batch --with-colons --fingerprint | awk -F: '/^fpr:/{print $10}' - args: - executable: /bin/bash - environment: - GNUPGHOME: "{{ monero_gnupghome.path }}" - register: monero_key_fingerprints - changed_when: false - when: monero_needs_install | bool - -# The vendored file could be swapped in a bad pull request. The fingerprint is -# the thing being trusted, so it is checked, not assumed. -- name: Assert the imported key is binaryFate's - ansible.builtin.assert: - that: - - monero_signing_fingerprint in monero_key_fingerprints.stdout_lines - quiet: true - fail_msg: >- - files/binaryfate.asc does not contain the expected key - {{ monero_signing_fingerprint }}. Refusing to go on. - when: monero_needs_install | bool - -- name: Download the signed hashes.txt - ansible.builtin.get_url: - url: "{{ monero_hashes_url }}" - dest: "{{ monero_gnupghome.path }}/hashes.txt" - mode: '0644' - force: yes - when: monero_needs_install | bool - -# One command that both verifies the signature and writes out ONLY the verified -# plaintext. Grepping the clearsigned file directly would search text that has -# not been through the signature check. -- name: Verify hashes.txt and extract the verified plaintext - ansible.builtin.command: >- - gpg --batch --status-fd 2 - --output {{ monero_gnupghome.path }}/hashes-verified.txt - --decrypt {{ monero_gnupghome.path }}/hashes.txt - environment: - GNUPGHOME: "{{ monero_gnupghome.path }}" - register: monero_hashes_verify - changed_when: false - failed_when: false - when: monero_needs_install | bool - -# Greping for "Good signature" - which is what the bitcoin_knots role does - -# passes for ANY key in the keyring. VALIDSIG names the key, and the line -# carries both the signing subkey and the primary key fingerprint, so checking -# that our fingerprint appears on it survives a subkey rotation. -- name: Assert hashes.txt carries a valid signature from binaryFate - ansible.builtin.assert: - that: - - monero_hashes_verify.rc == 0 - - monero_hashes_verify.stderr is search('VALIDSIG[^\n]*' ~ monero_signing_fingerprint) - quiet: true - fail_msg: >- - hashes.txt did not verify against {{ monero_signing_fingerprint }}. - This is the whole security control of this role - aborting. - gpg said: {{ monero_hashes_verify.stderr | default('') }} - when: monero_needs_install | bool - -- name: Download the monerod release tarball - ansible.builtin.get_url: - url: "{{ monero_url }}" - dest: "/tmp/{{ monero_tarball }}" - mode: '0644' - checksum: "sha256:{{ monero_sha256 }}" - force: yes - when: monero_needs_install | bool - -- name: Compute the SHA256 of the downloaded tarball - ansible.builtin.stat: - path: "/tmp/{{ monero_tarball }}" - checksum_algorithm: sha256 - get_checksum: yes - register: monero_tarball_stat - when: monero_needs_install | bool - -# The check that actually ties these bytes to binaryFate's signature. The -# get_url checksum above only ties them to a string in our own repo; this ties -# them to the signed list. Both the hash AND the filename must be on one line, -# so a hash belonging to a different artefact cannot satisfy it. -- name: Look up the downloaded tarball's checksum in the signed hashes - ansible.builtin.command: - argv: - - grep - - -F - - -x - - "{{ monero_tarball_stat.stat.checksum }} {{ monero_tarball }}" - - "{{ monero_gnupghome.path }}/hashes-verified.txt" - register: monero_hash_match - changed_when: false - failed_when: false - when: monero_needs_install | bool - -- name: Assert the tarball is the one binaryFate signed - ansible.builtin.assert: - that: - - monero_hash_match.rc == 0 - quiet: true - fail_msg: >- - {{ monero_tarball }} hashes to {{ monero_tarball_stat.stat.checksum }}, - which is not the value signed for that filename in hashes.txt. - Do not install this file. - when: monero_needs_install | bool - -- name: Create a temporary extraction directory - ansible.builtin.tempfile: - state: directory - suffix: monero-extract - register: monero_extract_dir - when: monero_needs_install | bool - -- name: Extract the monerod tarball - ansible.builtin.unarchive: - src: "/tmp/{{ monero_tarball }}" - dest: "{{ monero_extract_dir.path }}" - remote_src: yes - when: monero_needs_install | bool - -# By find rather than by a hardcoded path: the tarball unpacks to -# monero-x86_64-linux-gnu-v/, which is not the name of the tarball. -- name: Locate the extracted monerod binary - ansible.builtin.find: - paths: "{{ monero_extract_dir.path }}" - patterns: monerod - recurse: yes - file_type: file - register: monero_extracted - when: monero_needs_install | bool - -- name: Assert the archive contained monerod - ansible.builtin.assert: - that: - - monero_extracted.files | length > 0 - quiet: true - fail_msg: "No monerod binary in {{ monero_tarball }}" - when: monero_needs_install | bool - -# Only monerod. The wallet and blockchain utilities are in the same tarball if -# they are ever wanted; a node does not need them and every binary installed is -# surface that has to be kept current. -- name: Install monerod - ansible.builtin.copy: - src: "{{ monero_extracted.files[0].path }}" - dest: "{{ monero_bin_dir }}/monerod" - remote_src: yes - owner: root - group: root - mode: '0755' - when: monero_needs_install | bool - notify: Restart monerod - -- name: Clean up the download and verification artefacts - ansible.builtin.file: - path: "{{ item }}" - state: absent - loop: - - "/tmp/{{ monero_tarball }}" - - "{{ monero_extract_dir.path | default('/nonexistent') }}" - - "{{ monero_gnupghome.path | default('/nonexistent') }}" - when: monero_needs_install | bool diff --git a/ansible/roles/monero/tasks/main.yml b/ansible/roles/monero/tasks/main.yml deleted file mode 100644 index 699cf3b..0000000 --- a/ansible/roles/monero/tasks/main.yml +++ /dev/null @@ -1,15 +0,0 @@ ---- -# import_tasks, not include_tasks: static imports stay visible to --list-tasks, -# which is the main verification tool in this repo. -# -# The order is load-bearing: -# user must exist before anything chowns to it -# storage the data directory IS the mount point of the 500 GB disk, so if the -# disk is not there the daemon must not start and quietly begin -# filling a 10 GB root filesystem with a 300 GB blockchain -- ansible.builtin.import_tasks: user.yml -- ansible.builtin.import_tasks: storage.yml -- ansible.builtin.import_tasks: install.yml -- ansible.builtin.import_tasks: configure.yml -- ansible.builtin.import_tasks: service.yml -- ansible.builtin.import_tasks: healthcheck.yml diff --git a/ansible/roles/monero/tasks/service.yml b/ansible/roles/monero/tasks/service.yml deleted file mode 100644 index ea11f62..0000000 --- a/ansible/roles/monero/tasks/service.yml +++ /dev/null @@ -1,35 +0,0 @@ ---- -- name: Reload systemd daemon - systemd: - daemon_reload: yes - -- name: Enable and start monerod - systemd: - name: monerod - enabled: yes - state: started - -- name: Flush handlers so monerod is running before we ask it anything - meta: flush_handlers - -# Generous, and non-fatal. On a first start monerod initialises LMDB on a fresh -# 500 GB disk before it opens the RPC port; on a later start it is quick. A -# timeout here is not a deployment failure, and the health check is the thing -# that decides whether the node is actually well. -- name: Wait for the monerod RPC port - ansible.builtin.wait_for: - port: "{{ monero_rpc_port }}" - host: "{{ monero_rpc_bind_ip }}" - delay: 5 - timeout: 180 - ignore_errors: yes - -- name: Check the monerod service status - systemd: - name: monerod - register: monero_service_status - changed_when: false - -- name: Display the monerod service status - ansible.builtin.debug: - msg: "monerod is {{ 'running' if monero_service_status.status.ActiveState == 'active' else 'NOT running' }}" diff --git a/ansible/roles/monero/tasks/storage.yml b/ansible/roles/monero/tasks/storage.yml deleted file mode 100644 index e9cb403..0000000 --- a/ansible/roles/monero/tasks/storage.yml +++ /dev/null @@ -1,172 +0,0 @@ ---- -# The 500 GB data disk. -# -# knots-box does this by hand: /mnt/knots_data holds ~875 GB of blockchain and -# appears nowhere in Ansible, so a rebuild of that host would come up with the -# chain silently missing. This role does it in code instead. - -# Does the disk already carry our filesystem? Its label is the identity from -# here on. When it is found, the device name below is never read: everything -# that follows works from the labelled filesystem, so a stale or reordered -# device name cannot break a routine run. blkid exits 2 for "no such label". -- name: Look for an existing Monero filesystem by its label - ansible.builtin.command: "blkid -L {{ monero_data_label }}" - register: monero_labelled_fs - changed_when: false - failed_when: monero_labelled_fs.rc not in [0, 2] - check_mode: false - -- name: Find the device backing the root filesystem - ansible.builtin.command: findmnt -no SOURCE / - register: monero_root_source - changed_when: false - check_mode: false - -# First run only: the disk is blank and has to be found by name. -- name: Prepare a blank data disk - when: monero_labelled_fs.rc != 0 - block: - # The configured name is a /dev/disk/by-id symlink. Resolve it before - # comparing it to anything: the root source is a kernel name like - # /dev/sdb1, and a symlink path is never a prefix of one, so comparing the - # unresolved name would make the root-disk guard below always pass. - - name: Resolve the configured data device to its kernel name - ansible.builtin.command: "readlink -e {{ monero_data_device }}" - register: monero_device_path - changed_when: false - failed_when: false - check_mode: false - - - name: Check the Monero data device - ansible.builtin.stat: - path: "{{ monero_device_path.stdout | default('', true) or monero_data_device }}" - register: monero_device - - - name: Find the disk the root filesystem lives on - ansible.builtin.command: "lsblk -no PKNAME {{ monero_root_source.stdout }}" - register: monero_root_disk - changed_when: false - check_mode: false - - # A guard, not ceremony. mkfs on the wrong device is unrecoverable. - - name: Assert the data device is a real block device and is not the root disk - ansible.builtin.assert: - that: - - monero_device_path.rc == 0 - - monero_device.stat.exists - - monero_device.stat.isblk | default(false) - - monero_device_path.stdout != '/dev/' ~ (monero_root_disk.stdout | trim) - - not monero_root_source.stdout.startswith(monero_device_path.stdout) - quiet: true - fail_msg: >- - {{ monero_data_device }} (-> {{ monero_device_path.stdout | default('nothing', true) }}) - is not a usable data device, or it is the disk the root filesystem - lives on ({{ monero_root_source.stdout }}). Refusing to touch it. - Check `ls -l /dev/disk/by-id` on the host. - - - name: Read any existing filesystem on the data device - ansible.builtin.command: "blkid -o value -s TYPE {{ monero_device_path.stdout }}" - register: monero_existing_fs - changed_when: false - failed_when: false - check_mode: false - - # No monero-data label was found, so a filesystem here is something else. - # community.general.filesystem would leave an ext4 of any label alone but - # happily format over a disk it does not recognise. Say so out loud instead. - - name: Refuse to reformat a disk that already carries a different filesystem - ansible.builtin.assert: - that: - - monero_existing_fs.stdout | trim in ['', monero_data_fstype] - quiet: true - fail_msg: >- - {{ monero_data_device }} already holds a '{{ monero_existing_fs.stdout | trim }}' - filesystem, not '{{ monero_data_fstype }}'. Refusing to overwrite it. - - - name: Create the filesystem on the data device - community.general.filesystem: - fstype: "{{ monero_data_fstype }}" - dev: "{{ monero_device_path.stdout }}" - opts: "{{ monero_data_mkfs_opts }}" - -# By label: on the first run the filesystem was just created with it, on every -# other run it was found by it. -- name: Read the filesystem UUID - ansible.builtin.command: "blkid -o value -s UUID -t LABEL={{ monero_data_label }}" - register: monero_fs_uuid - changed_when: false - failed_when: false - check_mode: false - -# Everything past this point mounts by UUID, so there is nothing sensible to do -# without one. On a --check run against a disk that has no filesystem yet this -# is expected and unavoidable: check mode did not create the filesystem, so -# there is no UUID to read. Say that, rather than leaving a bare blkid rc=2. -# Exactly one: two disks with this label (a cloned disk, say) would make the -# mount below ambiguous. -- name: Assert exactly one UUID was found - ansible.builtin.assert: - that: - - monero_fs_uuid.stdout | trim | length == 36 - quiet: true - fail_msg: >- - Expected exactly one filesystem labelled {{ monero_data_label }}, found: - '{{ monero_fs_uuid.stdout | trim }}'. - {% if ansible_check_mode %}No UUID is expected under --check when the disk - has not been formatted yet - check mode cannot create the filesystem, so - it cannot dry-run the mount either. Run without --check once, then - --check is meaningful from then on.{% else %}If none: mkfs appears to have - failed. If several: more than one disk carries the label. Check `lsblk -f` - on the host.{% endif %} - -# Same owner and mode as the "Own the mounted data directory" task below, and -# that is deliberate rather than duplication. Before the disk is mounted this -# creates the mount point; afterwards it targets the mounted filesystem and is a -# no-op. Giving it root:root 0755 here - the obvious thing to write - made the -# two tasks disagree, and each run flipped the directory back and forth. -- name: Create the Monero data directory as a mount point - ansible.builtin.file: - path: "{{ monero_data_dir }}" - state: directory - owner: "{{ monero_user }}" - group: "{{ monero_group }}" - mode: '0750' - -# By UUID, never by a device name. nofail so a missing disk does not drop a headless -# VM into an emergency shell - monerod is stopped from starting by -# RequiresMountsFor= in its unit instead, which the systemd monitor then reports. -# noatime because LMDB touches a lot of pages and none of it wants an atime write. -- name: Mount the data disk by UUID - ansible.posix.mount: - path: "{{ monero_data_dir }}" - src: "UUID={{ monero_fs_uuid.stdout | trim }}" - fstype: "{{ monero_data_fstype }}" - opts: defaults,noatime,nofail - state: mounted - -# AFTER the mount, never before: chowning the mount point first sets permissions -# on the directory the mount then hides, and the blockchain lands on a root-owned -# filesystem that monerod cannot write to. -- name: Own the mounted data directory - ansible.builtin.file: - path: "{{ monero_data_dir }}" - state: directory - owner: "{{ monero_user }}" - group: "{{ monero_group }}" - mode: '0750' - -- name: Confirm the data directory is a mount point and not the root filesystem - ansible.builtin.command: "findmnt -no SOURCE --target {{ monero_data_dir }}" - register: monero_mounted_source - changed_when: false - check_mode: false - -- name: Assert the blockchain will land on the data disk - ansible.builtin.assert: - that: - - monero_mounted_source.stdout | trim != monero_root_source.stdout | trim - quiet: true - fail_msg: >- - {{ monero_data_dir }} resolves to {{ monero_mounted_source.stdout | trim }}, - which is the root filesystem. The 500 GB disk is not mounted and a full - sync would fill the 10 GB root disk. diff --git a/ansible/roles/monero/tasks/user.yml b/ansible/roles/monero/tasks/user.yml deleted file mode 100644 index 269ed53..0000000 --- a/ansible/roles/monero/tasks/user.yml +++ /dev/null @@ -1,24 +0,0 @@ ---- -# The service account, created before anything that chowns to it. -# -# This is its own file because two later steps both depend on it and neither -# owns it: storage.yml chowns the mount point to this user, and install.yml -# installs a binary that runs as it. Creating it inside either one makes the -# other's ordering an accident. - -- name: Create the monero system group - group: - name: "{{ monero_group }}" - system: yes - state: present - -- name: Create the monero system user - user: - name: "{{ monero_user }}" - group: "{{ monero_group }}" - system: yes - shell: /usr/sbin/nologin - home: "{{ monero_data_dir }}" - create_home: no - comment: "Monero node daemon" - state: present diff --git a/ansible/roles/monero/templates/healthcheck.service.j2 b/ansible/roles/monero/templates/healthcheck.service.j2 deleted file mode 100644 index 7619581..0000000 --- a/ansible/roles/monero/templates/healthcheck.service.j2 +++ /dev/null @@ -1,19 +0,0 @@ -[Unit] -Description=Monero Node Health Check -After=network.target monerod.service - -[Service] -Type=oneshot -User=root -ExecStart=/usr/local/bin/monerod-healthcheck-push.sh -Environment=HEALTHCHECK_PUSH_URL={{ healthcheck_push_url }} -Environment=HEALTHCHECK_PUSH_TOKEN={{ healthcheck_push_token }} -Environment=MONERO_RPC_HOST={{ monero_rpc_bind_ip }} -Environment=MONERO_RPC_PORT={{ monero_rpc_port }} -Environment=MONERO_RPC_USER={{ monero_rpc_user }} -Environment=MONERO_RPC_PASSWORD={{ monero_rpc_password }} -StandardOutput=journal -StandardError=journal - -[Install] -WantedBy=multi-user.target diff --git a/ansible/roles/monero/templates/healthcheck.sh.j2 b/ansible/roles/monero/templates/healthcheck.sh.j2 deleted file mode 100644 index 7aebd80..0000000 --- a/ansible/roles/monero/templates/healthcheck.sh.j2 +++ /dev/null @@ -1,70 +0,0 @@ -#!/bin/bash -# Monero node health check — managed by Ansible (roles/monero) -# -# The exit code is the answer and systemd keeps it: -# systemctl is-failed monerod-healthcheck.service -# Reporting anywhere else is optional and generic. -# -# WHAT THIS DELIBERATELY DOES NOT CHECK: whether the node is synchronised. -# A full initial sync takes days. A check that required `synchronized: true` -# would report DOWN for every one of those days, which is not a monitor, it is -# an alarm with the batteries taken out. Answering RPC is the health question; -# sync progress is reported in the message so it is visible without being -# alertable. - -RPC_HOST="${MONERO_RPC_HOST:-127.0.0.1}" -RPC_PORT="${MONERO_RPC_PORT:-18081}" -RPC_USER="${MONERO_RPC_USER:-}" -RPC_PASSWORD="${MONERO_RPC_PASSWORD:-}" -PUSH_URL="${HEALTHCHECK_PUSH_URL:-}" -PUSH_TOKEN="${HEALTHCHECK_PUSH_TOKEN:-}" - -report() { - local status=$1 - local msg=$2 - - # No push URL is normal, not an error: the exit code below is still a - # complete answer for anything reading unit state. - [ -n "$PUSH_URL" ] || return 0 - - local encoded_msg="${msg// /%20}" - - # Gatus external endpoint: a POST with a bearer token. - local _ok=false - [ "${status}" = "up" ] && _ok=true - curl -s --max-time 15 --retry 2 -o /dev/null -X POST \ - -H "Authorization: Bearer ${PUSH_TOKEN}" \ - "${PUSH_URL}?success=${_ok}&error=${encoded_msg}" || true -} - -# --digest, not --user: monerod's --rpc-login is HTTP Digest auth, so curl has -# to answer the 401 challenge rather than send credentials up front. -response=$(curl -s --max-time 30 --digest -u "${RPC_USER}:${RPC_PASSWORD}" \ - -H 'Content-Type: application/json' \ - -d '{"jsonrpc":"2.0","id":"healthcheck","method":"get_info"}' \ - "http://${RPC_HOST}:${RPC_PORT}/json_rpc" 2>/dev/null) - -if [ -z "$response" ]; then - report "down" "monerod RPC not responding" - exit 1 -fi - -status=$(echo "$response" | jq -r '.result.status // empty' 2>/dev/null) - -if [ "$status" != "OK" ]; then - report "down" "monerod RPC status: ${status:-unparseable}" - exit 1 -fi - -height=$(echo "$response" | jq -r '.result.height // 0' 2>/dev/null) -target=$(echo "$response" | jq -r '.result.target_height // 0' 2>/dev/null) -synced=$(echo "$response" | jq -r '.result.synchronized // false' 2>/dev/null) - -# monerod reports target_height=0 once it has caught up, so show the height we -# actually reached rather than a confusing "of 0". -if [ "$target" = "0" ]; then - report "up" "OK height=${height} synced=${synced}" -else - report "up" "OK height=${height}/${target} synced=${synced}" -fi -exit 0 diff --git a/ansible/roles/monero/templates/healthcheck.timer.j2 b/ansible/roles/monero/templates/healthcheck.timer.j2 deleted file mode 100644 index 306a575..0000000 --- a/ansible/roles/monero/templates/healthcheck.timer.j2 +++ /dev/null @@ -1,21 +0,0 @@ -[Unit] -Description=Monero Node Health Check Timer -# Deliberately NO `Requires=monerod.service`. Requires on a timer stops the -# timer when the required unit stops - "if the thing I am watching goes down, -# stop watching it" - and leaves nothing to re-arm when the service returns. -# Fulcrum's timer carried it and sat active+enabled for seven months without -# firing once. bitcoin_knots' timer still carries it; do not copy that. - -[Timer] -OnBootSec=2min -# Every 5 minutes (see monero_healthcheck_on_calendar). The cadence and the -# Gatus heartbeat window are ONE decision - the window MUST exceed this -# interval, and it is the standard 11m in -# infra/403_service_probe_registration.yml. -OnCalendar={{ monero_healthcheck_on_calendar }} -# Small: a delay comparable to the interval would eat the heartbeat's slack. -RandomizedDelaySec=30 -Persistent=true - -[Install] -WantedBy=timers.target diff --git a/ansible/roles/monero/templates/monerod.conf.j2 b/ansible/roles/monero/templates/monerod.conf.j2 deleted file mode 100644 index 9043ca8..0000000 --- a/ansible/roles/monero/templates/monerod.conf.j2 +++ /dev/null @@ -1,46 +0,0 @@ -# monerod configuration - managed by Ansible (roles/monero) -# -# Config-file syntax is the command-line options with the leading -- removed; -# valueless flags take =1. https://docs.getmonero.org/interacting/monero-config-file/ - -data-dir={{ monero_data_dir }} -log-file={{ monero_log_dir }}/monerod.log -log-level=0 -max-log-file-size=104850000 - -# --- P2P: outbound only ----------------------------------------------------- -# Nothing forwards {{ monero_p2p_port }} to this box, so the node dials out to -# peers and never receives. It syncs and serves us completely; it simply does -# not relay for strangers. no-igd follows: with nothing forwarded there is -# nothing for UPnP to do except announce us to the router. -p2p-bind-ip=0.0.0.0 -p2p-bind-port={{ monero_p2p_port }} -no-igd=1 -out-peers={{ monero_out_peers }} -in-peers={{ monero_in_peers }} - -# --- Unrestricted RPC: loopback only ---------------------------------------- -# Administration and the health check. Never leaves the box. -rpc-bind-ip={{ monero_rpc_bind_ip }} -rpc-bind-port={{ monero_rpc_port }} - -# --- Restricted RPC: the only surface anything off-box touches -------------- -# A SECOND server, view-only, reached over the tailnet by Caddy on the edge -# host. This is rpc-restricted-bind-port and deliberately NOT `restricted-rpc=1`: -# that flag restricts the loopback server too, and then we could not ask our own -# node anything useful. -# -# Bound to 0.0.0.0 and reachable only over the tailnet - Tailscale's ts-input -# chain sits ahead of UFW, so nothing on the LAN or the internet reaches this -# port without going through Caddy. confirm-external-bind is monerod's -# "yes, I meant to do that" for a non-loopback bind. -rpc-restricted-bind-ip=0.0.0.0 -rpc-restricted-bind-port={{ monero_restricted_rpc_port }} -confirm-external-bind=1 - -# HTTP Digest, and it guards BOTH servers above - core_rpc_server::init passes -# the same login to each. This is what Feather sends as the node credentials. -rpc-login={{ monero_rpc_user }}:{{ monero_rpc_password }} - -# Nothing subscribes to ZMQ here, so it is one less listener. -no-zmq=1 diff --git a/ansible/roles/monero/templates/monerod.service.j2 b/ansible/roles/monero/templates/monerod.service.j2 deleted file mode 100644 index 929d2ca..0000000 --- a/ansible/roles/monero/templates/monerod.service.j2 +++ /dev/null @@ -1,38 +0,0 @@ -[Unit] -Description=Monero Daemon -Documentation=https://docs.getmonero.org/running-node/monerod-systemd/ -After=network-online.target -Wants=network-online.target -# The blockchain lives on a separate 500 GB disk mounted at {{ monero_data_dir }}. -# Without this line a failed mount would not stop monerod - it would start, find -# an empty directory, and begin writing a ~300 GB chain onto the 10 GB root -# filesystem. This turns that silent disaster into a unit that refuses to start, -# which the systemd monitor reports within five minutes. -RequiresMountsFor={{ monero_data_dir }} - -[Service] -Type=simple -User={{ monero_user }} -Group={{ monero_group }} -ExecStart={{ monero_bin_dir }}/monerod --config-file {{ monero_conf_dir }}/monerod.conf --non-interactive - -Restart=always -RestartSec=30 -# monerod flushes LMDB on shutdown and that is not instant on a full chain. -# Killing it mid-flush is how the database gets corrupted and resynced from zero. -TimeoutStopSec=300 - -StandardOutput=journal -StandardError=journal - -# Hardening: the daemon only ever writes to its own data and log directories. -NoNewPrivileges=true -PrivateTmp=true -ProtectSystem=strict -ProtectHome=read-only -ReadWritePaths={{ monero_data_dir }} {{ monero_log_dir }} - -LimitNOFILE=65535 - -[Install] -WantedBy=multi-user.target diff --git a/ansible/roles/navidrome/README.md b/ansible/roles/navidrome/README.md deleted file mode 100644 index 0fd5199..0000000 --- a/ansible/roles/navidrome/README.md +++ /dev/null @@ -1,179 +0,0 @@ -# navidrome - -Navidrome — a music server — plus [beets](https://beets.io), the CLI that fixes -tags before Navidrome ever indexes them. Both live on `music-box`. - -Navidrome is a single Go binary with a SQLite database and no external -dependencies beyond `ffmpeg` for transcoding. It speaks two protocols on one -port: its own web UI, and the **Subsonic API** (`/rest/*`), which is what every -mobile client in the ecosystem — Symfonium, Amperfy, Tempo, substreamer — -actually talks to. - -| | | -|---|---| -| Host | `music-box` (nodito VM 8200, 1 core / 1 GB / 50 GB) | -| Port | 4533 | -| Public at | `music.contrapeso.xyz`, proxied by Caddy on `vipy` | -| Library | `/srv/music` — beets-managed, Navidrome reads only | -| Staging | `/srv/incoming` — unsorted, Navidrome never looks here | -| State | `/var/lib/navidrome` — SQLite, cache, transcode scratch | -| Backups | none, deliberately — see below | - -## Exposure - -Public, authenticated by **Navidrome's own login**. There is no `basic_auth` in -front of it, and that is a decision rather than an omission. - -The reason is the Subsonic API. A Subsonic client authenticates with -`?u=&t=&s=` in the query string and has nowhere -to put a second set of credentials. HTTP Basic Auth in front of the whole site -is invisible to most of them, so the choice is between a proxy credential and a -working phone. This estate has made the same call before, for the same reason: -`monerod`'s restricted RPC is proxied bare so Feather can reach it, and its own -Digest auth does the authenticating. - -The cost is stated plainly: Navidrome's login page and the whole `/rest/*` -surface are reachable by anyone who finds the hostname. Navidrome has **no rate -limiting and no 2FA**. The fail2ban jail below covers part of that gap. - -### The fail2ban jail on `vipy`, and what it does not cover - -Installed by the edge play in `services/navidrome/deploy_navidrome_playbook.yml`, -reading a per-vhost Caddy access log at `/var/log/caddy/music.log`. - -It catches **web UI brute force**: a bad login on `/auth/login` returns HTTP -401, which lands in the access log and trips the filter. - -It very likely does **not** catch Subsonic API brute force, because the -Subsonic protocol answers a bad password with **HTTP 200** and puts the failure -in the body (``). -Caddy never sees response bodies, so a status-code filter is blind to it. - -The obvious fix does not work either. Navidrome's own journal logs those -failures, but the request arrives from `vipy` over the tailnet, so the only -address fail2ban could ban on this box is `vipy`'s — which would take down every -service behind the edge. Banning has to happen at the edge; the edge cannot see -the failure. - -So: `/rest/*` is protected by password strength alone. That is the reason the -Navidrome password must be long, random, and used nowhere else. - -To re-check whether this is still true after a Navidrome upgrade: - -```bash -curl -s -o /dev/null -w '%{http_code}\n' \ - 'https://music.contrapeso.xyz/rest/ping.view?u=x&p=wrong&v=1.16.1&c=test' -``` - -`200` means the caveat holds. `401` means it no longer does — widen the jail -and delete this section. - -## Passwords are stored reversibly - -Navidrome **cannot** hash user passwords one-way. The Subsonic API requires the -server to compute `md5(password + salt)` on every request, so it must be able to -recover the plaintext. It encrypts passwords at rest with -`PasswordEncryptionKey` instead (`navidrome_password_encryption_key`, from -`group_vars/all/vault.yml`). - -Two consequences, both real: - -- Root on this box means every Navidrome password in plaintext. **Use a password - unique to this service.** -- **Changing the key invalidates every stored password.** Set it before the first - account exists and leave it alone. - -## Backups: none, deliberately - -Recorded as a decision so it does not read as an oversight — the same treatment -`phoenixd`, `monero` and `xmrblocks` get. - -The music files are large and would swamp the age-encrypted nightly path that -`backup_source` uses. `navidrome.db` is mostly a rebuildable index: delete it and -a rescan reconstructs the library. - -What is **not** rebuildable is the curation on top — playlists, play counts, -ratings, starred items, user accounts. Today there is none of it, which is why -this is a fair trade. **If that changes, this is the decision to revisit**, and -the fix is small: `navidrome.db` is a single SQLite file and would fit the -`backup_source` pattern in about fifteen lines. - -The beets database (`/var/lib/beets/library.db`) is in the same position — it -can be rebuilt from the tags beets itself wrote into the files. - -## beets - -Installed and configured by this role; **not** a service, and deliberately so. -`beet import` is interactive and it *moves and renames files*. A timer that -reorganises a music library unattended is a way to lose one. - -Config at `~counterweight/.config/beets/config.yaml`, library DB in -`/var/lib/beets`. - -```bash -# the usual loop -rsync -avP ~/Music/newstuff/ music-box:/srv/incoming/ -ssh music-box -beet import /srv/incoming -``` - -Navidrome picks up the result on its next scan. - -The single most important line in that config is -`paths.default: $albumartist/...`. Organising by `$artist` instead scatters -every compilation and every "feat." track into a phantom artist, which is how a -library ends up unusable and Navidrome gets blamed for it. - -ReplayGain and lyrics are set `auto: no` — on one core they turn a two-minute -import into a twenty-minute one. Run `beet replaygain` and `beet lyrics` -afterwards, separately. - -### Ownership, which is the fiddly part - -`/srv/music` is `counterweight:navidrome` mode **2750**. The setgid bit is -load-bearing: without it, files beets creates land in the curator's own group, -Navidrome cannot read them, and the symptom is an **empty library rather than a -permission error**. - -## Monitoring - -| Layer | What it answers | Where | -|---|---|---| -| `infra/400` | is the host alive, is the disk filling | automatic, needs `gatus_push_tokens.music_box_local` in the vault | -| `infra/401` | is the `navidrome` unit active | `monitored_services` in `host_vars/music_box_local/main.yml` | -| `infra/403` + this role | does Navidrome actually serve its own UI (`/app/`) | `probe_navidrome` | -| `infra/402` | does `music.contrapeso.xyz` answer from outside, is the cert valid | `public_sites` | - -The role's own check fetches `/app/` and greps for `navidrome`. A unit can be -perfectly `active` while serving nothing useful. - -`/app/` was chosen after testing all three candidates against 0.64.0: -`/ping` returns a bare `.` (proves only that something listens); -`/rest/ping.view` identifies the server but logs two warnings per call, -~576 a day; `/app/` returns `Navidrome` and logs nothing. - -## Upgrading - -```bash -curl -sL https://github.com/navidrome/navidrome/releases/download/v/navidrome_checksums.txt \ - | grep linux_amd64.tar.gz -``` - -Bump `navidrome_version` and `navidrome_sha256` **together** in -`defaults/main.yml`, then re-run the playbook. The install is version-gated, so -an unchanged version is a no-op. - -## Sizing - -1 core / 1 GB / 50 GB. Two things to know: - -- **50 GB is the library ceiling**, not just the OS — `/srv/music` is on the root - disk. Roughly 300–450 albums in FLAC. Growing it means a Proxmox resize plus - `growpart`/`resize2fs` (see `03_vm_disk_enlargement.md`), because - `ignore_changes = all` in the tofu config means tofu will not do it. -- **1 GB is fine for serving, tight for scanning.** The initial scan of a large - library is the memory peak. If it gets OOM-killed, do what `monerod-box` - needed: raise it in the Proxmox UI and record it back in `vms.auto.tfvars`. - -Transcoding is the CPU cost, and it is avoidable: clients that direct-play -(Symfonium, Amperfy) make the server a file shipper. Roku and browsers transcode. diff --git a/ansible/roles/navidrome/defaults/main.yml b/ansible/roles/navidrome/defaults/main.yml deleted file mode 100644 index e328804..0000000 --- a/ansible/roles/navidrome/defaults/main.yml +++ /dev/null @@ -1,85 +0,0 @@ -# Navidrome Configuration Variables -# https://www.navidrome.org/docs/usage/configuration-options/ - -# --- Version ---------------------------------------------------------------- -# Bump navidrome_version and navidrome_sha256 TOGETHER. The hash is the one -# for the linux_amd64 tarball in that release's navidrome_checksums.txt: -# curl -sL https://github.com/navidrome/navidrome/releases/download/v/navidrome_checksums.txt \ -# | grep linux_amd64.tar.gz -navidrome_version: "0.64.0" -navidrome_arch: "linux_amd64" -navidrome_sha256: "efd94d11253234035b5d3cf284da917111736e72daeeda4f45a58251c60553ac" -navidrome_url: "https://github.com/navidrome/navidrome/releases/download/v{{ navidrome_version }}/navidrome_{{ navidrome_version }}_{{ navidrome_arch }}.tar.gz" - -# --- Service user ----------------------------------------------------------- -navidrome_user: navidrome -navidrome_group: navidrome - -# --- Paths ------------------------------------------------------------------ -navidrome_bin_path: /usr/local/bin/navidrome -navidrome_config_dir: /etc/navidrome -navidrome_config_file: "{{ navidrome_config_dir }}/navidrome.toml" -# SQLite database, cache, transcoding scratch. The only thing Navidrome writes. -navidrome_data_dir: /var/lib/navidrome - -# The canonical library. Navidrome READS this and never writes it, which is why -# it is absent from ReadWritePaths in the unit. beets is what writes here. -navidrome_music_dir: /srv/music -# Staging area. Whatever lands here is unsorted and has NOT been through beets; -# Navidrome never looks at it. -navidrome_incoming_dir: /srv/incoming - -# The human who curates the library with beets. Owns both directories above; -# navidrome only needs group read. -navidrome_curator_user: "{{ new_user }}" - -# --- HTTP ------------------------------------------------------------------- -# 0.0.0.0, not loopback: the edge host reaches this across the tailnet. UFW -# denies the port from the LAN and the internet, and tailnet traffic bypasses -# UFW entirely (-A INPUT -j ts-input sits ahead of every ufw-* chain), so -# binding wide here does not widen exposure. Confirmed on mempool-box. -navidrome_bind_address: "0.0.0.0" -navidrome_port: 4533 -navidrome_log_level: info - -# Navidrome's telemetry. Off. -navidrome_enable_insights: false - -# --- Secrets ---------------------------------------------------------------- -# Navidrome CANNOT store user passwords as a one-way hash: the Subsonic API -# authenticates with md5(password + salt), so the server has to be able to -# recover the plaintext. It encrypts them at rest with this key instead. -# -# Consequences, both real: -# - root on this box means every Navidrome password in plaintext. Use a -# password unique to this service and nowhere else. -# - CHANGING THIS KEY INVALIDATES EVERY STORED PASSWORD. Set it before the -# first account exists and then leave it alone. -# Injected by the playbook from group_vars/all/vault.yml; empty here so the -# role stays self-contained and asserts loudly rather than starting insecure. -navidrome_password_encryption_key: "" - -# Set true to see the rendered config in --diff. It carries the key above, so -# the template is diff-suppressed by default. -navidrome_reveal_config: false - -# --- beets ------------------------------------------------------------------ -# beets is NOT a service. It is an interactive CLI the curator runs over SSH to -# fix tags before Navidrome ever indexes them; there is no unit and no timer. -navidrome_beets_enabled: true -navidrome_beets_dir: /var/lib/beets -navidrome_beets_config: "{{ '/home/' ~ navidrome_curator_user ~ '/.config/beets/config.yaml' }}" - -# --- Health check ----------------------------------------------------------- -# "Does Navidrome actually answer", as opposed to infra/401's "is the unit -# active". The exit code is the answer and systemd keeps it: -# systemctl is-failed navidrome-healthcheck.service -navidrome_healthcheck_script_path: /usr/local/bin/navidrome-healthcheck-push.sh -navidrome_healthcheck_service_name: navidrome-healthcheck - -# WHERE TO REPORT HEALTH — the one place to plug in monitoring. -# Empty means check, log, exit honestly, report nowhere. -healthcheck_push_url: "" -# Bearer token for the Gatus external endpoint. Required whenever a push URL -# is set: Gatus rejects an unauthenticated push with 401. -healthcheck_push_token: "" diff --git a/ansible/roles/navidrome/handlers/main.yml b/ansible/roles/navidrome/handlers/main.yml deleted file mode 100644 index acaaf7e..0000000 --- a/ansible/roles/navidrome/handlers/main.yml +++ /dev/null @@ -1,16 +0,0 @@ ---- -# NO `when:` on either of these, ever. The uptime_kuma_enabled banner was once -# applied to contiguous blocks and took Fulcrum's entire restart handler with -# it: config changes landed on disk and silently never took effect for months. -# A handler that can be gated off is a handler that will be. -- name: Restart navidrome - systemd: - name: navidrome - state: restarted - daemon_reload: yes - -- name: Restart navidrome health check timer - systemd: - name: navidrome-healthcheck.timer - state: restarted - daemon_reload: yes diff --git a/ansible/roles/navidrome/tasks/beets.yml b/ansible/roles/navidrome/tasks/beets.yml deleted file mode 100644 index ab9d395..0000000 --- a/ansible/roles/navidrome/tasks/beets.yml +++ /dev/null @@ -1,70 +0,0 @@ ---- -# beets: the tag fixer that runs BEFORE Navidrome sees anything. -# -# Navidrome has no metadata fetcher and no tag editor by design - it trusts the -# tags on disk completely. That is the right call, but it means a library with -# bad tags renders as a bad library and there is nothing in Navidrome to fix it -# with. beets is the fix, and it belongs on this box because that is where the -# files are. -# -# There is no unit and no timer here on purpose: `beet import` is interactive, -# it MOVES and RENAMES files, and a cron job that reorganises a music library -# unattended is a way to lose one. - -- name: Install beets and its fingerprinting dependencies - apt: - name: - - beets - # fpcalc. Without it the chroma plugin cannot acoustically identify a - # file whose tags are too mangled to match, which is exactly the case - # beets is most useful for. - - libchromaprint-tools - - python3-acoustid - # lastgenre imports pylast, and Debian's beets package does NOT depend on - # it. Without this the plugin fails to load and every single `beet` - # invocation prints a ModuleNotFoundError traceback while genres are - # silently never fetched - configured, listed in `plugins:`, and inert. - - python3-pylast - state: present - update_cache: true - when: navidrome_beets_enabled | bool - -- name: Create the beets state directory - ansible.builtin.file: - path: "{{ navidrome_beets_dir }}" - state: directory - owner: "{{ navidrome_curator_user }}" - group: "{{ navidrome_curator_user }}" - mode: '0750' - when: navidrome_beets_enabled | bool - -# Both levels explicitly, parent first. The file module creates missing parents -# like `mkdir -p`, but it creates them owned by ROOT - which would leave -# ~/.config root-owned on a fresh cloud-init box and break every other -# user-level tool that wants to write there later. -- name: Create the curator's XDG config directory - ansible.builtin.file: - path: "{{ navidrome_beets_config | dirname | dirname }}" - state: directory - owner: "{{ navidrome_curator_user }}" - group: "{{ navidrome_curator_user }}" - mode: '0755' - when: navidrome_beets_enabled | bool - -- name: Create the beets config directory - ansible.builtin.file: - path: "{{ navidrome_beets_config | dirname }}" - state: directory - owner: "{{ navidrome_curator_user }}" - group: "{{ navidrome_curator_user }}" - mode: '0755' - when: navidrome_beets_enabled | bool - -- name: Write the beets configuration - ansible.builtin.template: - src: beets-config.yaml.j2 - dest: "{{ navidrome_beets_config }}" - owner: "{{ navidrome_curator_user }}" - group: "{{ navidrome_curator_user }}" - mode: '0644' - when: navidrome_beets_enabled | bool diff --git a/ansible/roles/navidrome/tasks/healthcheck.yml b/ansible/roles/navidrome/tasks/healthcheck.yml deleted file mode 100644 index c263c43..0000000 --- a/ansible/roles/navidrome/tasks/healthcheck.yml +++ /dev/null @@ -1,48 +0,0 @@ ---- -# "Does Navidrome actually answer", as opposed to infra/401's "is the unit -# active". Navidrome can be perfectly `active` and serving 500s - a music -# folder it cannot read does exactly that, and presents as an empty library. - -- name: Create the navidrome health check script - ansible.builtin.template: - src: healthcheck.sh.j2 - dest: "{{ navidrome_healthcheck_script_path }}" - owner: root - group: root - mode: '0755' - validate: "bash -n %s" - -# 0600: carries the Gatus bearer token. -- name: Create the navidrome health check systemd service - ansible.builtin.template: - src: healthcheck.service.j2 - dest: "/etc/systemd/system/{{ navidrome_healthcheck_service_name }}.service" - owner: root - group: root - mode: '0600' - notify: Restart navidrome health check timer - -- name: Create the navidrome health check systemd timer - ansible.builtin.template: - src: healthcheck.timer.j2 - dest: "/etc/systemd/system/{{ navidrome_healthcheck_service_name }}.timer" - owner: root - group: root - mode: '0644' - notify: Restart navidrome health check timer - -- name: Reload systemd daemon after health check units - systemd: - daemon_reload: yes - -# Ungated: enabling a timer is deployment, not monitoring. -- name: Enable and restart the navidrome health check timer - systemd: - name: "{{ navidrome_healthcheck_service_name }}.timer" - enabled: yes - state: restarted - daemon_reload: yes - -- name: Run the navidrome health check once to arm the timer - ansible.builtin.command: "systemctl start {{ navidrome_healthcheck_service_name }}.service" - changed_when: false diff --git a/ansible/roles/navidrome/tasks/install.yml b/ansible/roles/navidrome/tasks/install.yml deleted file mode 100644 index 72adfeb..0000000 --- a/ansible/roles/navidrome/tasks/install.yml +++ /dev/null @@ -1,134 +0,0 @@ ---- -# The user, the directories, and the binary. - -- name: Install Navidrome's runtime dependencies - apt: - name: - # Navidrome shells out to ffmpeg for on-the-fly transcoding. Without it - # the server starts fine and every format the client cannot play natively - # simply fails, which looks like a client bug. - - ffmpeg - - curl - state: present - update_cache: true - -- name: Create the navidrome system group - group: - name: "{{ navidrome_group }}" - system: yes - state: present - -- name: Create the navidrome system user - user: - name: "{{ navidrome_user }}" - group: "{{ navidrome_group }}" - system: yes - shell: /usr/sbin/nologin - home: "{{ navidrome_data_dir }}" - create_home: no - comment: "Navidrome Music Server" - state: present - -- name: Create the navidrome data directory - ansible.builtin.file: - path: "{{ navidrome_data_dir }}" - state: directory - owner: "{{ navidrome_user }}" - group: "{{ navidrome_group }}" - mode: '0750' - -- name: Create the navidrome config directory - ansible.builtin.file: - path: "{{ navidrome_config_dir }}" - state: directory - owner: root - group: "{{ navidrome_group }}" - mode: '0750' - -# Two principals share this tree and neither is root: the curator writes it -# through beets, Navidrome only reads it. setgid (the 2) is load-bearing - -# without it, files beets creates land in the curator's own group and Navidrome -# cannot read them, which presents as an empty library rather than an error. -- name: Create the music library directory - ansible.builtin.file: - path: "{{ navidrome_music_dir }}" - state: directory - owner: "{{ navidrome_curator_user }}" - group: "{{ navidrome_group }}" - mode: '2750' - -- name: Create the incoming staging directory - ansible.builtin.file: - path: "{{ navidrome_incoming_dir }}" - state: directory - owner: "{{ navidrome_curator_user }}" - group: "{{ navidrome_curator_user }}" - mode: '0755' - -# check_mode: no on the next two - this repo verifies with `--check --diff`, so -# the role has to survive a dry run. A read-only version probe and a temp -# directory are both harmless to perform for real, and without them every task -# below evaluates against an undefined navidrome_tmp.path and the check aborts. -- name: Check the installed navidrome version - ansible.builtin.command: "{{ navidrome_bin_path }} --version" - register: navidrome_installed_version - changed_when: false - failed_when: false - check_mode: no - -- name: Decide whether navidrome needs installing - ansible.builtin.set_fact: - navidrome_needs_install: "{{ navidrome_version not in (navidrome_installed_version.stdout | default('')) }}" - -- name: Create a temporary directory for the navidrome download - ansible.builtin.tempfile: - state: directory - suffix: navidrome - register: navidrome_tmp - when: navidrome_needs_install | bool - check_mode: no - -# checksum: is the verification. Upstream publishes navidrome_checksums.txt per -# release but signs nothing, so there is no signature to check the way -# roles/monero checks binaryfate's - this pins the exact artefact and no more. -- name: Download the navidrome release tarball - ansible.builtin.get_url: - url: "{{ navidrome_url }}" - dest: "{{ navidrome_tmp.path }}/navidrome.tar.gz" - checksum: "sha256:{{ navidrome_sha256 }}" - mode: '0644' - when: navidrome_needs_install | bool and not ansible_check_mode - -- name: Unpack the navidrome tarball - ansible.builtin.unarchive: - src: "{{ navidrome_tmp.path }}/navidrome.tar.gz" - dest: "{{ navidrome_tmp.path }}" - remote_src: yes - when: navidrome_needs_install | bool and not ansible_check_mode - -- name: Install the navidrome binary - ansible.builtin.copy: - src: "{{ navidrome_tmp.path }}/navidrome" - dest: "{{ navidrome_bin_path }}" - remote_src: yes - owner: root - group: root - mode: '0755' - when: navidrome_needs_install | bool and not ansible_check_mode - notify: Restart navidrome - -- name: Remove the navidrome download directory - ansible.builtin.file: - path: "{{ navidrome_tmp.path }}" - state: absent - when: navidrome_needs_install | bool - check_mode: no - -# Check mode skips the three tasks above, so without this a dry run is silent -# about the single most important thing it would do. -- name: Report what a real run would install - ansible.builtin.debug: - msg: >- - Would install navidrome {{ navidrome_version }} to {{ navidrome_bin_path }} - (currently: {{ navidrome_installed_version.stdout | default('not installed') | trim }}) - when: navidrome_needs_install | bool and ansible_check_mode diff --git a/ansible/roles/navidrome/tasks/main.yml b/ansible/roles/navidrome/tasks/main.yml deleted file mode 100644 index 70494a9..0000000 --- a/ansible/roles/navidrome/tasks/main.yml +++ /dev/null @@ -1,7 +0,0 @@ ---- -# import_tasks, not include_tasks: static imports stay visible to --list-tasks, -# which is the main verification tool in this repo. -- ansible.builtin.import_tasks: install.yml -- ansible.builtin.import_tasks: beets.yml -- ansible.builtin.import_tasks: service.yml -- ansible.builtin.import_tasks: healthcheck.yml diff --git a/ansible/roles/navidrome/tasks/service.yml b/ansible/roles/navidrome/tasks/service.yml deleted file mode 100644 index 92b0aef..0000000 --- a/ansible/roles/navidrome/tasks/service.yml +++ /dev/null @@ -1,67 +0,0 @@ ---- -# Config, unit, start, prove it is up. - -# The key encrypts every stored password. Starting without it would come up -# fine and quietly use a built-in default, so this fails the run instead. -- name: Assert the password encryption key arrived from the vault - ansible.builtin.assert: - that: - - navidrome_password_encryption_key | length > 0 - fail_msg: >- - navidrome_password_encryption_key is empty. It is injected by - services/navidrome/deploy_navidrome_playbook.yml from - group_vars/all/vault.yml. Navidrome stores user passwords REVERSIBLY - (the Subsonic API needs the plaintext to compute md5(password+salt)), - and this key is what encrypts them at rest. Set it before the first - account exists - changing it later invalidates every stored password. - quiet: yes - -- name: Write the navidrome configuration - ansible.builtin.template: - src: navidrome.toml.j2 - dest: "{{ navidrome_config_file }}" - owner: root - group: "{{ navidrome_group }}" - mode: '0640' - diff: "{{ navidrome_reveal_config | default(false) | bool }}" - notify: Restart navidrome - -- name: Create the navidrome systemd unit - ansible.builtin.template: - src: navidrome.service.j2 - dest: /etc/systemd/system/navidrome.service - owner: root - group: root - mode: '0644' - notify: Restart navidrome - -- name: Reload systemd daemon - systemd: - daemon_reload: yes - -- name: Enable and start navidrome - systemd: - name: navidrome - enabled: yes - state: started - -- name: Flush handlers so navidrome is running before we probe it - meta: flush_handlers - -- name: Wait for the navidrome HTTP port - ansible.builtin.wait_for: - port: "{{ navidrome_port }}" - host: 127.0.0.1 - delay: 5 - timeout: 300 - ignore_errors: yes - -- name: Check the navidrome service status - systemd: - name: navidrome - register: navidrome_status - changed_when: false - -- name: Display the navidrome service status - ansible.builtin.debug: - msg: "navidrome is {{ 'running' if navidrome_status.status.ActiveState == 'active' else 'NOT running' }}" diff --git a/ansible/roles/navidrome/templates/beets-config.yaml.j2 b/ansible/roles/navidrome/templates/beets-config.yaml.j2 deleted file mode 100644 index 9620a7d..0000000 --- a/ansible/roles/navidrome/templates/beets-config.yaml.j2 +++ /dev/null @@ -1,81 +0,0 @@ -# beets configuration — managed by Ansible (roles/navidrome) -# https://beets.readthedocs.io/ -# -# Run it by hand, over SSH, when material has accumulated in {{ navidrome_incoming_dir }}: -# beet import {{ navidrome_incoming_dir }} -# -# It is interactive on purpose. `beet import -q` auto-applies only strong -# matches and leaves the rest where they are, which makes a usable review queue -# if the batches ever get boring. - -directory: {{ navidrome_music_dir }} -library: {{ navidrome_beets_dir }}/library.db - -# $albumartist, NOT $artist. This is the single most important line in the -# file: organising by $artist scatters every compilation and every "feat." -# track into its own phantom artist, which is how a library ends up with four -# hundred one-album artists and Navidrome gets blamed for it. -paths: - default: $albumartist/$album%aunique{}/$track $title - singleton: Non-Album/$artist/$title - comp: Compilations/$album%aunique{}/$track $title - albumtype:soundtrack: Soundtracks/$album/$track $title - -import: - # MOVES files out of {{ navidrome_incoming_dir }} into the library. Set - # `copy: yes` and `move: no` instead if you would rather keep originals - # until you trust it. - move: yes - write: yes - languages: en es - log: {{ navidrome_beets_dir }}/import.log - -plugins: chroma fetchart embedart scrub duplicates mbsync missing info lastgenre replaygain lyrics permissions - -# THE trap this library is exposed to, and the fix for it. -# -# beets MOVES files into place, and rename(2) preserves the original group and -# mode - so the setgid bit on {{ navidrome_music_dir }} does NOT apply to them -# (it only affects files CREATED there, such as the album directories beets -# makes, which do come out group-{{ navidrome_group }}). A track that arrives -# mode 0600 therefore lands mode 0600 owned by the curator's own group, and -# Navidrome cannot read it. It does not error: the track is simply ABSENT from -# the library, which is indistinguishable from never having imported it. -# -# Measured on this box: a 0600 file was unreadable by the navidrome user; the -# same file at 0644 was fine. So normalise every imported file. -# NO leading zeros. The plugin does int(str(value), 8), and YAML would parse -# `0644` as octal -> decimal 420, which the plugin then re-reads as octal 0o420. -# Verified on the box: `0644` silently left a 0600 file at 0600; `644` fixes it. -permissions: - file: 644 - dir: 755 - -# Acoustic fingerprinting: identifies a file from the audio itself when the -# tags are too mangled to match. Needs fpcalc, installed by the role. -chroma: - auto: yes - -fetchart: - auto: yes -embedart: - auto: yes -# Strip junk and stale tags before writing clean ones. -scrub: - auto: yes - -lastgenre: - auto: yes - source: album - -# auto: no on both of these deliberately. This box has one core; ReplayGain -# analysis and lyrics fetching during an import turn a two-minute job into a -# twenty-minute one. Run them separately, after the import, when nothing is -# waiting on you: -# beet replaygain -# beet lyrics -replaygain: - auto: no - backend: ffmpeg -lyrics: - auto: no diff --git a/ansible/roles/navidrome/templates/healthcheck.service.j2 b/ansible/roles/navidrome/templates/healthcheck.service.j2 deleted file mode 100644 index 7adcbac..0000000 --- a/ansible/roles/navidrome/templates/healthcheck.service.j2 +++ /dev/null @@ -1,18 +0,0 @@ -[Unit] -Description=Navidrome Health Check -After=network.target navidrome.service - -[Service] -Type=oneshot -User=root -ExecStart={{ navidrome_healthcheck_script_path }} -# Comfortably longer than the script's 30s curl plus its push retries. -TimeoutStartSec=120 -Environment=HEALTHCHECK_PUSH_URL={{ healthcheck_push_url }} -Environment=HEALTHCHECK_PUSH_TOKEN={{ healthcheck_push_token }} -Environment=NAVIDROME_URL=http://127.0.0.1:{{ navidrome_port }} -StandardOutput=journal -StandardError=journal - -[Install] -WantedBy=multi-user.target diff --git a/ansible/roles/navidrome/templates/healthcheck.sh.j2 b/ansible/roles/navidrome/templates/healthcheck.sh.j2 deleted file mode 100644 index 1c63e6d..0000000 --- a/ansible/roles/navidrome/templates/healthcheck.sh.j2 +++ /dev/null @@ -1,65 +0,0 @@ -#!/bin/bash -# Navidrome health check — managed by Ansible (roles/navidrome) -# -# The exit code is the answer and systemd keeps it: -# systemctl is-failed {{ navidrome_healthcheck_service_name }}.service -# Reporting anywhere else is optional and generic. -# -# /app/ rather than /ping, established by testing against 0.64.0 on the box: -# -# /ping 200, Content-Type text/plain, body is a single "." - it -# proves something is listening and nothing more. -# /rest/ping.view identifies itself ("type":"navidrome") but logs TWO -# warning lines per call. At one call per 5 minutes that is -# ~576 spurious warnings a day, which is how you train -# yourself to ignore the warning level. -# /app/ 200, 2.8 KB, Navidrome, and SILENT. -# -# So /app/ is the only one that proves it is really Navidrome serving its real -# UI without polluting the journal. Same reasoning as the xmrblocks check, -# which fetches a real page because a missing templates dir serves 200s of -# nothing. - -URL="${NAVIDROME_URL:-http://127.0.0.1:{{ navidrome_port }}}" -PUSH_URL="${HEALTHCHECK_PUSH_URL:-}" -PUSH_TOKEN="${HEALTHCHECK_PUSH_TOKEN:-}" - -report() { - local status=$1 - local msg=$2 - - # No push URL is normal, not an error: the exit code below is still a - # complete answer for anything reading unit state. - [ -n "$PUSH_URL" ] || return 0 - - local encoded_msg="${msg// /%20}" - local _ok=false - [ "${status}" = "up" ] && _ok=true - curl -s --max-time 15 --retry 2 -o /dev/null -X POST \ - -H "Authorization: Bearer ${PUSH_TOKEN}" \ - "${PUSH_URL}?success=${_ok}&error=${encoded_msg}" || true -} - -# 30s: Navidrome answers /ping in milliseconds normally, but this box has one -# core and a library scan will happily eat it. A tight timeout here would -# manufacture false DOWNs every time a scan runs. -body=$(curl -s --max-time 30 -w '\n%{http_code}' "${URL}/app/" 2>/dev/null) -code=$(printf '%s' "$body" | tail -1) - -if [ -z "$code" ] || [ "$code" = "000" ]; then - report "down" "navidrome not responding" - exit 1 -fi - -if [ "$code" != "200" ]; then - report "down" "navidrome HTTP ${code}" - exit 1 -fi - -if ! printf '%s' "$body" | grep -qi "navidrome"; then - report "down" "navidrome served 200 but not its own UI (something else on the port?)" - exit 1 -fi - -report "up" "OK" -exit 0 diff --git a/ansible/roles/navidrome/templates/healthcheck.timer.j2 b/ansible/roles/navidrome/templates/healthcheck.timer.j2 deleted file mode 100644 index bacb7cd..0000000 --- a/ansible/roles/navidrome/templates/healthcheck.timer.j2 +++ /dev/null @@ -1,14 +0,0 @@ -[Unit] -Description=Navidrome Health Check Timer -# Deliberately NO `Requires=navidrome.service`. Requires on a timer stops the -# timer when the watched unit stops - "if the thing I am watching goes down, -# stop watching it" - leaving nothing to re-arm when it returns. Fulcrum's -# timer carried it and sat active+enabled for seven months without firing. - -[Timer] -OnBootSec=2min -OnUnitActiveSec=5min -Persistent=true - -[Install] -WantedBy=timers.target diff --git a/ansible/roles/navidrome/templates/navidrome.service.j2 b/ansible/roles/navidrome/templates/navidrome.service.j2 deleted file mode 100644 index 17410da..0000000 --- a/ansible/roles/navidrome/templates/navidrome.service.j2 +++ /dev/null @@ -1,33 +0,0 @@ -[Unit] -Description=Navidrome Music Server -Documentation=https://www.navidrome.org/docs/ -After=network-online.target -Wants=network-online.target - -[Service] -Type=simple -User={{ navidrome_user }} -Group={{ navidrome_group }} -WorkingDirectory={{ navidrome_data_dir }} -ExecStart={{ navidrome_bin_path }} --configfile {{ navidrome_config_file }} -Restart=always -RestartSec=20 -TimeoutStopSec=60 -StandardOutput=journal -StandardError=journal - -# Hardening: Navidrome writes its database and cache, and nothing else. -NoNewPrivileges=true -PrivateTmp=true -ProtectSystem=strict -ProtectHome=read-only -# {{ navidrome_music_dir }} is deliberately absent: ProtectSystem=strict leaves -# it read-only, which is exactly right. Navidrome indexes the library; beets -# writes it. If Navidrome ever appears to need write access here, something is -# wrong with the request, not with this line. -ReadWritePaths={{ navidrome_data_dir }} - -LimitNOFILE=65535 - -[Install] -WantedBy=multi-user.target diff --git a/ansible/roles/navidrome/templates/navidrome.toml.j2 b/ansible/roles/navidrome/templates/navidrome.toml.j2 deleted file mode 100644 index 2db0057..0000000 --- a/ansible/roles/navidrome/templates/navidrome.toml.j2 +++ /dev/null @@ -1,24 +0,0 @@ -# Navidrome configuration — managed by Ansible (roles/navidrome) -# https://www.navidrome.org/docs/usage/configuration-options/ -# -# Mode 0640 root:{{ navidrome_group }}: PasswordEncryptionKey is below. - -MusicFolder = "{{ navidrome_music_dir }}" -DataFolder = "{{ navidrome_data_dir }}" - -Address = "{{ navidrome_bind_address }}" -Port = {{ navidrome_port }} - -LogLevel = "{{ navidrome_log_level }}" - -# See the long note in defaults/main.yml. Changing this invalidates every -# stored password. -PasswordEncryptionKey = "{{ navidrome_password_encryption_key }}" - -EnableInsightsCollector = {{ navidrome_enable_insights | bool | lower }} - -# Scan schedule is left at Navidrome's own default deliberately. The option has -# moved between releases (top-level ScanSchedule vs a [Scanner] section), and -# pinning the wrong spelling here is silently ignored rather than rejected - -# you would get the default anyway and believe you had configured something. -# Check `navidrome config` on the box before adding it. diff --git a/ansible/roles/phoenixd/README.md b/ansible/roles/phoenixd/README.md deleted file mode 100644 index 3c147e3..0000000 --- a/ansible/roles/phoenixd/README.md +++ /dev/null @@ -1,56 +0,0 @@ -# `phoenixd` - -Deploys and runs [phoenixd](https://phoenix.acinq.co/server), an ACINQ Lightning -node, on the edge host. LNBits uses it as a wallet backend. The HTTP API stays on -loopback — phoenixd is never published through Caddy. - -Converted from `deploy_phoenixd_playbook.yml` (552 lines) under Plan 6. The -playbook is now 18 lines. - -## Phases - -| | | -|---|---| -| `install.yml` | packages, system user, directories, versioned download and install | -| `service.yml` | systemd unit, start, then first-boot checks (config written, seed created) | -| `healthcheck.yml` | check script, unit, timer | - -## The seed - -`{{ phoenixd_data_dir }}/seed.dat` **is** the funds. phoenixd is deliberately -excluded from the automated backups (Plan 5, Model C): the seed is twelve fixed -words that never change, so an automated job would only manufacture more copies -of a static secret on more machines. Write them down offline, once. - -Note the live file is mode `0644`. That is phoenixd's own doing, not this role's, -and it is worth tightening. - -## Monitoring: one variable, no product knowledge - -The check asks the node itself — the service must be active **and** -`phoenix-cli getinfo` must return a `nodeId` — and records the answer in its exit -code, which systemd keeps: - -```bash -systemctl is-failed phoenixd-healthcheck.service -``` - -That is a complete answer with no monitoring system involved. To report -elsewhere, set `healthcheck_push_url` to anything accepting an HTTP ping. Gone -from this role: the embedded Python that created monitors over the Uptime Kuma -API, the `/tmp` credentials file, the push-URL file written and parsed back, and -the systemd `Environment=` rewrite. - -### Two things the conversion fixed - -**The check used to log an error once a minute.** Its `Environment=` push URL had -been empty since the decommissioning, and the script printed -`ERROR: UPTIME_KUMA_PUSH_URL not set` on every fire — roughly 1,400 times a day. -The exit code was still correct, so nothing was broken; it was pure noise, and -noise that trains you to ignore the log. An unset push URL is now normal and -silent. - -**`Enable and start phoenixd health check timer` was guarded by -`uptime_kuma_enabled`** and so had not run since the decommissioning — while the -timer itself was still live on the host from before. Ansible had quietly stopped -managing something that was still running. Ungated. diff --git a/ansible/roles/phoenixd/defaults/main.yml b/ansible/roles/phoenixd/defaults/main.yml deleted file mode 100644 index 2fcf1ea..0000000 --- a/ansible/roles/phoenixd/defaults/main.yml +++ /dev/null @@ -1,56 +0,0 @@ -# phoenixd Configuration Variables -# https://phoenix.acinq.co/server - -# Version - pin to a specific release tag -phoenixd_version: "0.9.0" -phoenixd_arch: "linux-x64" -phoenixd_url: "https://github.com/ACINQ/phoenixd/releases/download/v{{ phoenixd_version }}/phoenixd-{{ phoenixd_version }}-{{ phoenixd_arch }}.zip" - -# Service user -phoenixd_user: phoenix -phoenixd_group: phoenix - -# Directories -phoenixd_home: /opt/phoenixd -# phoenixd resolves its data dir from PHOENIX_DATADIR; seed.dat, phoenix.conf -# and the channel db all live here. -phoenixd_data_dir: "{{ phoenixd_home }}/.phoenix" -phoenixd_bin_dir: /usr/local/bin - -# Lightning settings -phoenixd_chain: mainnet -# Amount automatically requested when inbound liquidity is needed: off, 2m, 5m, 10m -phoenixd_auto_liquidity: "2m" -# Max mining fee for on-chain operations, in sats. Empty means phoenixd's default. -phoenixd_max_mining_fee: "" -# Required for non-interactive deployment -phoenixd_agree_tos: true - -# HTTP API -# Stays on loopback: phoenixd is never exposed through Caddy, only local -# consumers (e.g. LNBits on the same host) talk to it. -phoenixd_http_bind_ip: "127.0.0.1" -phoenixd_http_bind_port: 9740 - -# Optional webhook for payment events. Leave empty to disable. -phoenixd_webhook_url: "" - - - - -# --- Health check ----------------------------------------------------------- -# The check asks phoenixd itself whether it is healthy (service active AND the -# node answers getinfo with a nodeId) and records the answer in its exit code, -# which systemd keeps: -# systemctl is-failed phoenixd-healthcheck.service -# That is a complete answer with no monitoring system involved. -phoenixd_healthcheck_script_path: /usr/local/bin/phoenixd-healthcheck-push.sh -phoenixd_healthcheck_service_name: phoenixd-healthcheck - -# WHERE TO REPORT HEALTH — the one place to plug in monitoring. -# Empty means check, log, exit honestly, report nowhere. Any endpoint that -# accepts an HTTP ping works; nothing here is specific to a monitoring product. -healthcheck_push_url: "" -# Bearer token for the Gatus external endpoint. Required whenever a push URL -# is set: Gatus rejects an unauthenticated push with 401. -healthcheck_push_token: "" diff --git a/ansible/roles/phoenixd/handlers/main.yml b/ansible/roles/phoenixd/handlers/main.yml deleted file mode 100644 index 7779684..0000000 --- a/ansible/roles/phoenixd/handlers/main.yml +++ /dev/null @@ -1,12 +0,0 @@ ---- -- name: Restart phoenixd - systemd: - name: phoenixd - state: restarted - daemon_reload: yes - -- name: Restart phoenixd health check timer - systemd: - name: "{{ phoenixd_healthcheck_service_name }}.timer" - state: restarted - daemon_reload: yes diff --git a/ansible/roles/phoenixd/tasks/healthcheck.yml b/ansible/roles/phoenixd/tasks/healthcheck.yml deleted file mode 100644 index d439047..0000000 --- a/ansible/roles/phoenixd/tasks/healthcheck.yml +++ /dev/null @@ -1,68 +0,0 @@ ---- -# Everything here answers "is phoenixd healthy" and records the answer. The -# Uptime Kuma specifics that used to follow it — an embedded Python script -# creating monitors over the API, a /tmp credentials file, a push-URL file read -# back and parsed, and a systemd Environment= rewrite — are gone. What reports -# where is now one variable, healthcheck_push_url. See the role README. -- name: Create phoenixd health check script - ansible.builtin.template: - src: healthcheck.sh.j2 - dest: "{{ phoenixd_healthcheck_script_path }}" - owner: root - group: root - mode: "0755" - validate: "bash -n %s" - -- name: Create phoenixd health check systemd service - ansible.builtin.template: - src: healthcheck.service.j2 - dest: "/etc/systemd/system/{{ phoenixd_healthcheck_service_name }}.service" - owner: root - group: root - mode: "0600" - notify: Restart phoenixd health check timer - -- name: Create phoenixd health check systemd timer - ansible.builtin.template: - src: healthcheck.timer.j2 - dest: "/etc/systemd/system/{{ phoenixd_healthcheck_service_name }}.timer" - owner: root - group: root - mode: "0644" - notify: Restart phoenixd health check timer - -- name: Reload systemd daemon after health check units - systemd: - daemon_reload: yes - -# Ungated on purpose. This was guarded by `uptime_kuma_enabled`, but enabling a -# timer is deployment, not monitoring — the deprecation banner swept it up with -# the push plumbing. The timer is in fact running on the host, from before the -# decommissioning, so the guard meant Ansible had stopped managing something -# that was still live. -- name: Enable and start phoenixd health check timer - systemd: - name: "{{ phoenixd_healthcheck_service_name }}.timer" - enabled: yes - state: started - -- name: Display post-install information - debug: - msg: | - ✓ phoenixd {{ phoenixd_version }} deployed - - Status: systemctl status phoenixd - Logs: journalctl -u phoenixd -f - CLI: sudo PHOENIX_DATADIR={{ phoenixd_data_dir }} phoenix-cli --http-bind-port {{ phoenixd_http_bind_port }} getinfo - HTTP API: http://{{ phoenixd_http_bind_ip }}:{{ phoenixd_http_bind_port }} (loopback only) - Data dir: {{ phoenixd_data_dir }} - Health: systemctl is-failed {{ phoenixd_healthcheck_service_name }}.service - - API password (needed to wire LNBits up to this node): - sudo grep '^http-password=' {{ phoenixd_data_dir }}/phoenix.conf - - ⚠️ BACK UP THE SEED: {{ phoenixd_data_dir }}/seed.dat - Losing it means losing the funds. phoenixd is deliberately excluded - from the automated backups (Plan 5, Model C) because the seed is 12 - fixed words — write them down offline, once: - sudo cat {{ phoenixd_data_dir }}/seed.dat diff --git a/ansible/roles/phoenixd/tasks/install.yml b/ansible/roles/phoenixd/tasks/install.yml deleted file mode 100644 index 89a4fa0..0000000 --- a/ansible/roles/phoenixd/tasks/install.yml +++ /dev/null @@ -1,125 +0,0 @@ ---- -- name: Install phoenixd runtime dependencies - apt: - name: - - unzip - - curl - state: present - update_cache: yes - -# System User and Directories -- name: Create phoenixd system group - group: - name: "{{ phoenixd_group }}" - system: yes - -- name: Create phoenixd system user - user: - name: "{{ phoenixd_user }}" - group: "{{ phoenixd_group }}" - system: yes - shell: /usr/sbin/nologin - home: "{{ phoenixd_home }}" - create_home: yes - comment: "phoenixd Lightning node" - -- name: Create phoenixd home directory - file: - path: "{{ phoenixd_home }}" - state: directory - owner: "{{ phoenixd_user }}" - group: "{{ phoenixd_group }}" - mode: "0750" - -- name: Create phoenixd data directory - file: - path: "{{ phoenixd_data_dir }}" - state: directory - owner: "{{ phoenixd_user }}" - group: "{{ phoenixd_group }}" - mode: "0700" - -# Download and Install -- name: Check if phoenixd is already installed - stat: - path: "{{ phoenixd_bin_dir }}/phoenixd" - register: phoenixd_binary - -- name: Check installed phoenixd version - command: "{{ phoenixd_bin_dir }}/phoenixd --version" - register: phoenixd_installed_version - changed_when: false - failed_when: false - when: phoenixd_binary.stat.exists - -- name: Decide whether phoenixd needs installing - set_fact: - phoenixd_needs_install: >- - {{ not phoenixd_binary.stat.exists - or phoenixd_version not in (phoenixd_installed_version.stdout | default('')) }} - -- name: Download phoenixd {{ phoenixd_version }} - get_url: - url: "{{ phoenixd_url }}" - dest: "/tmp/phoenixd-{{ phoenixd_version }}.zip" - mode: "0644" - when: phoenixd_needs_install | bool - -- name: Create temporary extraction directory - file: - path: /tmp/phoenixd-extract - state: directory - mode: "0755" - when: phoenixd_needs_install | bool - -- name: Extract phoenixd archive - unarchive: - src: "/tmp/phoenixd-{{ phoenixd_version }}.zip" - dest: /tmp/phoenixd-extract - remote_src: yes - when: phoenixd_needs_install | bool - -- name: Locate extracted binaries - find: - paths: /tmp/phoenixd-extract - patterns: "{{ item }}" - recurse: yes - file_type: file - register: phoenixd_extracted - loop: - - phoenixd - - phoenix-cli - when: phoenixd_needs_install | bool - -- name: Fail if the archive did not contain the expected binaries - assert: - that: - - item.files | length > 0 - fail_msg: "Could not find '{{ item.item }}' in the phoenixd {{ phoenixd_version }} archive" - loop: "{{ phoenixd_extracted.results }}" - loop_control: - label: "{{ item.item }}" - when: phoenixd_needs_install | bool - -- name: Install phoenixd and phoenix-cli binaries - copy: - src: "{{ item.files[0].path }}" - dest: "{{ phoenixd_bin_dir }}/{{ item.item }}" - remote_src: yes - owner: root - group: root - mode: "0755" - loop: "{{ phoenixd_extracted.results }}" - loop_control: - label: "{{ item.item }}" - when: phoenixd_needs_install | bool - notify: Restart phoenixd - -- name: Clean up phoenixd download artifacts - file: - path: "{{ item }}" - state: absent - loop: - - "/tmp/phoenixd-{{ phoenixd_version }}.zip" - - /tmp/phoenixd-extract - diff --git a/ansible/roles/phoenixd/tasks/main.yml b/ansible/roles/phoenixd/tasks/main.yml deleted file mode 100644 index bc7ff05..0000000 --- a/ansible/roles/phoenixd/tasks/main.yml +++ /dev/null @@ -1,6 +0,0 @@ ---- -# import_tasks, not include_tasks: static imports stay visible to --list-tasks, -# which is how this conversion was verified against the playbook it replaced. -- ansible.builtin.import_tasks: install.yml -- ansible.builtin.import_tasks: service.yml -- ansible.builtin.import_tasks: healthcheck.yml diff --git a/ansible/roles/phoenixd/tasks/service.yml b/ansible/roles/phoenixd/tasks/service.yml deleted file mode 100644 index 79bf6f5..0000000 --- a/ansible/roles/phoenixd/tasks/service.yml +++ /dev/null @@ -1,52 +0,0 @@ ---- -- name: Build phoenixd command line arguments - set_fact: - phoenixd_args: >- - {{ (['--agree-to-terms-of-service'] if phoenixd_agree_tos else []) - + ['--chain', phoenixd_chain] - + ['--auto-liquidity', phoenixd_auto_liquidity] - + ['--http-bind-ip', phoenixd_http_bind_ip] - + ['--http-bind-port', phoenixd_http_bind_port | string] - + (['--max-mining-fee', phoenixd_max_mining_fee | string] if phoenixd_max_mining_fee else []) - + (['--webhook', phoenixd_webhook_url] if phoenixd_webhook_url else []) - + ['--silent'] }} - -- name: Create phoenixd systemd service - ansible.builtin.template: - src: phoenixd.service.j2 - dest: /etc/systemd/system/phoenixd.service - owner: root - group: root - mode: "0644" - notify: Restart phoenixd - -- name: Reload systemd daemon - systemd: - daemon_reload: yes - -- name: Enable and start phoenixd - systemd: - name: phoenixd - enabled: yes - state: started - -- name: Flush handlers so phoenixd is running before we inspect its data dir - meta: flush_handlers - -# --- First boot checks --- -- name: Wait for phoenixd to write its config file - wait_for: - path: "{{ phoenixd_data_dir }}/phoenix.conf" - state: present - timeout: 120 - -- name: Check that the seed file exists - stat: - path: "{{ phoenixd_data_dir }}/seed.dat" - register: phoenixd_seed_file - -- name: Fail if phoenixd did not create a seed - assert: - that: - - phoenixd_seed_file.stat.exists - fail_msg: "phoenixd started but {{ phoenixd_data_dir }}/seed.dat is missing - check 'journalctl -u phoenixd'" diff --git a/ansible/roles/phoenixd/templates/healthcheck.service.j2 b/ansible/roles/phoenixd/templates/healthcheck.service.j2 deleted file mode 100644 index 6ff21ab..0000000 --- a/ansible/roles/phoenixd/templates/healthcheck.service.j2 +++ /dev/null @@ -1,15 +0,0 @@ -[Unit] -Description=phoenixd Health Check -After=network.target phoenixd.service - -[Service] -Type=oneshot -User=root -ExecStart={{ phoenixd_healthcheck_script_path }} -Environment=HEALTHCHECK_PUSH_URL={{ healthcheck_push_url }} -Environment=HEALTHCHECK_PUSH_TOKEN={{ healthcheck_push_token }} -StandardOutput=journal -StandardError=journal - -[Install] -WantedBy=multi-user.target diff --git a/ansible/roles/phoenixd/templates/healthcheck.sh.j2 b/ansible/roles/phoenixd/templates/healthcheck.sh.j2 deleted file mode 100644 index a1717c5..0000000 --- a/ansible/roles/phoenixd/templates/healthcheck.sh.j2 +++ /dev/null @@ -1,46 +0,0 @@ -#!/bin/bash -# phoenixd health check — managed by Ansible (roles/phoenixd) -# -# Asks the node whether it is healthy and records the answer in the exit code, -# which systemd keeps: -# systemctl is-failed {{ phoenixd_healthcheck_service_name }}.service -# That is a complete answer on its own. Reporting anywhere else is optional. -PUSH_URL="${HEALTHCHECK_PUSH_URL:-}" -PUSH_TOKEN="${HEALTHCHECK_PUSH_TOKEN:-}" -export PHOENIX_DATADIR="{{ phoenixd_data_dir }}" - -check_phoenixd() { - # Service must be active and the node must answer getinfo. - # phoenix-cli reads the api password from $PHOENIX_DATADIR/phoenix.conf, - # but not the bind address, so pass it explicitly. - systemctl is-active --quiet phoenixd && \ - {{ phoenixd_bin_dir }}/phoenix-cli \ - --http-bind-ip {{ phoenixd_http_bind_ip }} \ - --http-bind-port {{ phoenixd_http_bind_port }} \ - getinfo 2>/dev/null | grep -q '"nodeId"' -} - -report() { - local status=$1 msg=$2 - # No push URL configured is NORMAL, not an error: the exit code below still - # answers the question. The previous version logged ERROR here on every - # single fire, once a minute, which is noise that trains you to ignore it. - [ -n "$PUSH_URL" ] || return 0 - # Gatus external endpoint: a POST with a bearer token, NOT Uptime Kuma's - # GET with ?status=up. The callers still pass up/down, so the mapping is - # done here rather than at every call site. - local _ok=false - [ "${status}" = "up" ] && _ok=true - curl -s --max-time 15 --retry 2 -o /dev/null -X POST \ - -H "Authorization: Bearer ${PUSH_TOKEN}" \ - "${PUSH_URL}?success=${_ok}&error=${msg// /%20}" || true -} - -if check_phoenixd; then - report "up" "OK" - exit 0 -else - echo "phoenixd is not responding" - report "down" "phoenixd not responding" - exit 1 -fi diff --git a/ansible/roles/phoenixd/templates/healthcheck.timer.j2 b/ansible/roles/phoenixd/templates/healthcheck.timer.j2 deleted file mode 100644 index ee07257..0000000 --- a/ansible/roles/phoenixd/templates/healthcheck.timer.j2 +++ /dev/null @@ -1,10 +0,0 @@ -[Unit] -Description=phoenixd Health Check Timer - -[Timer] -OnBootSec=2min -OnUnitActiveSec=1min -Persistent=true - -[Install] -WantedBy=timers.target diff --git a/ansible/roles/phoenixd/templates/phoenixd.service.j2 b/ansible/roles/phoenixd/templates/phoenixd.service.j2 deleted file mode 100644 index 44fdb94..0000000 --- a/ansible/roles/phoenixd/templates/phoenixd.service.j2 +++ /dev/null @@ -1,31 +0,0 @@ -[Unit] -Description=phoenixd - Lightning Network Node -Documentation=https://phoenix.acinq.co/server -After=network-online.target -Wants=network-online.target - -[Service] -Type=simple -User={{ phoenixd_user }} -Group={{ phoenixd_group }} -WorkingDirectory={{ phoenixd_home }} -Environment=PHOENIX_DATADIR={{ phoenixd_data_dir }} -ExecStart={{ phoenixd_bin_dir }}/phoenixd {{ phoenixd_args | join(' ') }} -Restart=always -RestartSec=30 -TimeoutStartSec=120 -TimeoutStopSec=120 -StandardOutput=journal -StandardError=journal - -# Hardening: the node only ever writes to its own data directory -NoNewPrivileges=true -PrivateTmp=true -ProtectSystem=strict -ProtectHome=read-only -ReadWritePaths={{ phoenixd_data_dir }} - -LimitNOFILE=65535 - -[Install] -WantedBy=multi-user.target diff --git a/ansible/roles/signal_api/README.md b/ansible/roles/signal_api/README.md deleted file mode 100644 index eadcfb7..0000000 --- a/ansible/roles/signal_api/README.md +++ /dev/null @@ -1,133 +0,0 @@ -# signal_api - -Runs [signal-cli-rest-api](https://github.com/bbernhard/signal-cli-rest-api) on -the `observability` host. Gatus uses it to deliver alerts over Signal. - -Gatus does not speak Signal — it POSTs JSON to this service, which holds the -Signal identity and does the protocol work. - -## It is never published, and that is not optional - -**This API has no authentication of any kind.** No key, no token, no basic auth. -Anything that can reach the port can send messages as your identity and read -your Signal. So the compose file publishes **no ports at all** and there is no -Caddy vhost. - -Gatus reaches it over a shared docker network (`monitoring`) by service name: -`http://signal-api:8080`. That is also *why* a shared network is needed rather -than a published port — Gatus runs in a container, so `127.0.0.1` for Gatus is -the Gatus container, not the host. - -The network is created by an explicit Ansible task in both this role and -`gatus`, so neither stack has to be deployed before the other. - -## MODE, and why `native` - -Upstream offers `normal`, `native`, `json-rpc` and `json-rpc-native`. The -json-rpc modes keep a resident JVM daemon and upstream describes them as -"increased memory". - -**This VPS has 464 MB of RAM**, already running Gatus and Caddy. A resident JVM -is not affordable. `native` runs a precompiled GraalVM binary per request — no -daemon, no resident cost — and alerts are rare enough that paying startup cost -per alert is the right trade. - -## Linking the device — a one-time manual step - -Ansible cannot scan a QR code, so this is manual. **Do not use -`/v1/qrcodelink`** — it is broken in `native` mode. - -### The trap - -`GET /v1/qrcodelink?device_name=...` returns: - -```json -{"error":"Couldn't create QR code: no data to encode"} -``` - -The linking itself is fine: running the binary directly inside the container -emits a perfectly good provisioning URI. - -``` -$ docker exec signal-api signal-cli-native link -n gatus -sgnl://linkdevice?uuid=...&pub_key=... -``` - -It is the REST wrapper that fails to capture that output in `native` mode. - -**Do not "fix" this by switching MODE to `normal` or `json-rpc`.** That puts a -JVM in the path of *every alert* on a 464 MB host, permanently degrading the -running system to work around a step performed once. Generate the QR yourself -instead. - -### The procedure - -**`docker exec` runs as root, but the service runs as uid 1000.** Without -`--config`, signal-cli writes the linked account to `/root/.local/share/signal-cli` -— the container's ephemeral layer, NOT the mounted volume. It looks like it -worked (`Associated with: +34…`), `/v1/accounts` keeps returning `[]`, and the -account is destroyed on the next `docker compose up`. Always pass `--config`. - -1. Start the link and capture the URI. It must keep running while you scan: - - docker exec signal-api sh -c "rm -f /tmp/link.uri; \ - nohup signal-cli-native --config /home/.local/share/signal-cli \ - link -n gatus > /tmp/link.uri 2>/tmp/link.log & echo started" - sleep 10 - docker exec signal-api cat /tmp/link.uri - - Do **not** add `setsid`, and do **not** background `docker exec` itself from - the host — the first stops the URI appearing, the second is killed when the - Ansible task returns. The output is block-buffered because stdout is a file, - so the URI appears only after several seconds; `stdbuf` does not help, as the - buffering is GraalVM's, not libc's. - -2. Render the QR on your own machine and scan it: - - qrencode -o /tmp/qr.png -s 12 -m 4 "sgnl://linkdevice?uuid=...&pub_key=..." - -3. Phone: Signal → Settings → Linked devices → **+** → scan. Provisioning links - expire in a couple of minutes, so generate and scan in one sitting. - -4. Confirm — this must list the number, not `[]`: - - docker exec signal-api curl -s http://localhost:8080/v1/accounts - -5. Send a test message: - - docker exec signal-api curl -s -X POST -H "Content-Type: application/json" \ - -d '{"message":"test","number":"+34…","recipients":["+34…"]}' \ - http://localhost:8080/v2/send - -Alerts are sent **from your own number**, so sending to yourself lands in Note -to Self. If the device is ever unlinked from the phone, alerts stop silently — -which is why this service is itself monitored. - -### If the phone says "network error" - -The phone is not the problem. `chat.signal.org` resolves to AWS Global -Accelerator **dualstack** addresses with the AAAA records first, this container -has no IPv6 address at all, and this host's IPv6 path is broken — the same edge -that returned a bogus 404 for the Go tarball. signal-cli reaches for an -unreachable IPv6 address and dies with `Link request error: Connection closed!`, -while the phone can only report a failed handshake. - -That is what `gai.conf` (mounted at `/etc/gai.conf`) fixes. If linking starts -failing again, check it is still mounted and that `getent ahosts chat.signal.org` -returns an IPv4 address first. - -## Backups - -Deliberately **not** backed up. The data directory holds Signal private keys, -and the recovery path is to link again from the phone — which takes a minute and -does not depend on any stored artefact. Backing it up would copy a credential -off the host to buy nothing. - -## Verifying - -```bash -docker ps --filter name=signal-api -docker exec signal-api curl -fsS http://localhost:8080/v1/health -docker exec signal-api curl -fsS http://localhost:8080/v1/accounts -docker logs signal-api --tail 50 -``` diff --git a/ansible/roles/signal_api/defaults/main.yml b/ansible/roles/signal_api/defaults/main.yml deleted file mode 100644 index 5b1b79f..0000000 --- a/ansible/roles/signal_api/defaults/main.yml +++ /dev/null @@ -1,46 +0,0 @@ ---- -# signal-cli-rest-api: the transport Gatus uses to send Signal messages. -# -# Gatus does not speak Signal. It POSTs JSON to this service, which holds the -# actual Signal identity and does the protocol work. - -# Pinned by DIGEST, with the release tag alongside for humans - same as Gatus. -# A tag is mutable; a digest is a content address, so `docker compose pull` -# either fetches exactly this image or fails. Upstream publishes versioned -# release tags (0.97 ... 0.101); the two values below must move together. -# -# THIS PIN EXPIRES. Signal's servers stop accepting signal-cli releases that -# fall a few months behind. 0.100 (signal-cli 0.14.5, 2026-06-11) was refused -# on 2026-10-18 - the phone reported "network error" and signal-cli logged -# "Link request error: Connection closed!", with the network itself fine. A -# stale pin fails exactly the same way for ALERTS: silently. Bump this at least -# every couple of months: https://github.com/bbernhard/signal-cli-rest-api/releases -signal_api_version: "0.101" # bundles signal-cli 0.14.8 -signal_api_image_digest: "sha256:2cf09d66a86fdd210e42ee91662216364acc27da1163a8c44167a77ea3a6559a" -signal_api_image: "bbernhard/signal-cli-rest-api@{{ signal_api_image_digest }}" - -signal_api_dir: /opt/signal-api -signal_api_data_dir: "{{ signal_api_dir }}/data" - -# MODE matters on this host. Upstream offers normal / native / json-rpc / -# json-rpc-native. json-rpc keeps a resident JVM daemon and upstream describes it -# as "increased memory" - this VPS has 464MB total and already runs Gatus and -# Caddy, so a resident JVM is not affordable. `native` runs a precompiled -# GraalVM binary per request: no daemon, no resident cost, and alerts are rare -# enough that paying startup per alert is the right trade. -signal_api_mode: native - -# Port INSIDE the shared docker network. Never published to the host: this API -# has NO AUTHENTICATION of any kind. Anyone who can reach it can send messages -# as you and read your Signal. -signal_api_port: 8080 - -# Both this and Gatus join this network so Gatus can reach the API by service -# name. Gatus runs in a container, so the host's loopback is NOT reachable from -# it - this is why a shared network is required rather than a published port. -signal_api_network: monitoring -signal_api_service_name: signal-api - -# The uid the upstream image drops to (`setpriv --reuid=1000`). The data -# directory must be owned by it or signal-cli cannot write the account. -signal_api_uid: 1000 diff --git a/ansible/roles/signal_api/tasks/main.yml b/ansible/roles/signal_api/tasks/main.yml deleted file mode 100644 index 1aa92b4..0000000 --- a/ansible/roles/signal_api/tasks/main.yml +++ /dev/null @@ -1,93 +0,0 @@ ---- -- name: Assert Docker is available - ansible.builtin.command: docker --version - register: signal_docker_check - changed_when: false - -# Created explicitly rather than by either compose file, so neither stack has to -# be deployed before the other and neither owns it. -- name: Ensure the shared monitoring network exists - ansible.builtin.command: "docker network create {{ signal_api_network }}" - register: signal_net - changed_when: "'already exists' not in signal_net.stderr" - failed_when: - - signal_net.rc != 0 - - "'already exists' not in signal_net.stderr" - -- name: Create the signal-api directory - ansible.builtin.file: - path: "{{ signal_api_dir }}" - state: directory - owner: root - group: root - mode: "0755" - -# Owned by the container's uid, NOT root. -# -# The image drops to uid 1000 (`setpriv --reuid=1000`), and a root-owned 0700 -# directory cannot be traversed by uid 1000 - signal-cli then fails to write the -# account and linking silently never completes, leaving a 39-byte accounts.json -# with no accounts and the API returning "Failed to read local accounts list". -# -# 0700 on uid 1000 is still private: only that uid and root can read the Signal -# private keys, which is the property actually wanted. -- name: Create the signal-api data directory owned by the container user - ansible.builtin.file: - path: "{{ signal_api_data_dir }}" - state: directory - owner: "{{ signal_api_uid }}" - group: "{{ signal_api_uid }}" - mode: "0700" - -- name: Write the IPv4-preference resolver config - ansible.builtin.template: - src: gai.conf.j2 - dest: "{{ signal_api_dir }}/gai.conf" - owner: root - group: root - mode: "0644" - -- name: Write the docker compose file - ansible.builtin.template: - src: docker-compose.yml.j2 - dest: "{{ signal_api_dir }}/docker-compose.yml" - owner: root - group: root - mode: "0644" - -- name: Pull the pinned signal-api image - ansible.builtin.command: - cmd: docker compose pull - chdir: "{{ signal_api_dir }}" - register: signal_pull - changed_when: "'Downloaded newer image' in signal_pull.stderr or 'Pull complete' in signal_pull.stderr" - -- name: Start signal-api - ansible.builtin.command: - cmd: docker compose up -d --remove-orphans - chdir: "{{ signal_api_dir }}" - register: signal_up - changed_when: "'Started' in signal_up.stderr or 'Created' in signal_up.stderr or 'Recreated' in signal_up.stderr" - -- name: Wait for the API to answer - ansible.builtin.command: - cmd: "docker exec {{ signal_api_service_name }} curl -fsS http://localhost:{{ signal_api_port }}/v1/health" - register: signal_health - until: signal_health.rc == 0 - retries: 12 - delay: 5 - changed_when: false - -- name: Report whether an account is linked yet - ansible.builtin.command: - cmd: "docker exec {{ signal_api_service_name }} curl -fsS http://localhost:{{ signal_api_port }}/v1/accounts" - register: signal_accounts - changed_when: false - failed_when: false - -- name: Show the linking status - ansible.builtin.debug: - msg: >- - {{ 'Linked account(s): ' ~ signal_accounts.stdout - if (signal_accounts.stdout | default('[]') | trim) not in ['[]', '', 'null'] - else 'NO ACCOUNT LINKED YET - this is a one-time manual step, see the role README.' }} diff --git a/ansible/roles/signal_api/templates/docker-compose.yml.j2 b/ansible/roles/signal_api/templates/docker-compose.yml.j2 deleted file mode 100644 index 9825299..0000000 --- a/ansible/roles/signal_api/templates/docker-compose.yml.j2 +++ /dev/null @@ -1,47 +0,0 @@ -# Managed by Ansible (roles/signal_api) -services: - {{ signal_api_service_name }}: - image: {{ signal_api_image }} - container_name: {{ signal_api_service_name }} - restart: unless-stopped - - environment: - MODE: "{{ signal_api_mode }}" - - volumes: - # Prefer IPv4. See gai.conf.j2 - without this, signal-cli reaches for - # chat.signal.org's IPv6 address, which is unreachable from here, and - # linking fails with an opaque "network error" on the phone. - - {{ signal_api_dir }}/gai.conf:/etc/gai.conf:ro - # Holds the Signal identity: the linked-device keys and registration - # state. Lose this and the device must be linked again by scanning a new - # QR code from the phone. It is also the most sensitive thing on this - # host - anyone with these keys can send and read Signal as you. - - {{ signal_api_data_dir }}:/home/.local/share/signal-cli - - networks: - - {{ signal_api_network }} - - # NO PORTS. Deliberately. - # - # This API has no authentication whatsoever - no key, no token, nothing. - # Publishing it, even on 127.0.0.1, would expose "send a Signal message as - # this identity" to anything that can reach the host. Gatus talks to it over - # the shared docker network by service name instead, which is why no port is - # published and why there is no Caddy vhost. - - healthcheck: - test: ["CMD", "curl", "-fsS", "http://localhost:8080/v1/health"] - interval: 60s - timeout: 5s - retries: 3 - - logging: - driver: json-file - options: - max-size: "10m" - max-file: "3" - -networks: - {{ signal_api_network }}: - external: true diff --git a/ansible/roles/signal_api/templates/gai.conf.j2 b/ansible/roles/signal_api/templates/gai.conf.j2 deleted file mode 100644 index 9b2cca4..0000000 --- a/ansible/roles/signal_api/templates/gai.conf.j2 +++ /dev/null @@ -1,17 +0,0 @@ -# Managed by Ansible (roles/signal_api) -# -# Prefer IPv4 over IPv6 in getaddrinfo. -# -# chat.signal.org resolves to AWS Global Accelerator dualstack addresses, and -# DNS returns the AAAA records first. This container has NO IPv6 address at all, -# and this host's IPv6 path is unreliable anyway - the same edge that made -# Google's IPv6 endpoint return a confident 404 for the Go tarball. -# -# signal-cli would connect to the AAAA address, fail, and report -# Link request error: Connection closed! -# while the phone showed a bare "network error" - a failure with no obvious -# cause on either end. -# -# This line flips the precedence so IPv4-mapped addresses sort first, which is -# the standard glibc fix. It does NOT disable IPv6; it only changes the order. -precedence ::ffff:0:0/96 100 diff --git a/ansible/roles/socket_proxy/README.md b/ansible/roles/socket_proxy/README.md deleted file mode 100644 index cebb95f..0000000 --- a/ansible/roles/socket_proxy/README.md +++ /dev/null @@ -1,62 +0,0 @@ -# `socket_proxy` - -Exposes a service running on a private Tailscale host through a public TCP port -on an edge machine, using `systemd-socket-proxyd`. Writes a `.socket` and a -`.service` unit, enables the socket, opens the UFW port, and checks the upstream -is reachable. - -## Usage - -```yaml -- ansible.builtin.include_role: - name: socket_proxy - vars: - socket_proxy_name: fulcrum-ssl # -> fulcrum-ssl-proxy.{socket,service} - socket_proxy_description: "Fulcrum SSL" # -> "Fulcrum SSL Proxy Socket" - socket_proxy_listen_port: "{{ fulcrum_ssl_port }}" - socket_proxy_upstream_host: "{{ fulcrum_tailscale_hostname }}" -``` - -`socket_proxy_upstream_port` defaults to `socket_proxy_listen_port`, which is -what all three current callers want. - -## Optional unit settings - -These exist because the **live** `bitcoin-p2p-proxy` units on vipy carried -settings the playbook never wrote. Somebody added them by hand, so running -`deploy_bitcoin_knots_playbook.yml` would have silently removed them: - -| Variable | Emits | Why it matters | -|---|---|---| -| `socket_proxy_free_bind` | `FreeBind=true` in `[Socket]` | Lets the socket bind to an address that is not up yet. Without it the socket can fail to start on boot. | -| `socket_proxy_documentation` | `Documentation=` in both units | Cosmetic. | -| `socket_proxy_timeout_stop_sec` | `TimeoutStopSec=` | Bounds how long a stop can hang. | -| `socket_proxy_log_to_journal` | `StandardOutput=journal` + `StandardError=journal` | Cosmetic on modern systemd, which defaults to the journal anyway. | - -Only `bitcoin-p2p` passes any of them. - -## `socket_proxy_ufw_comment` - -Defaults to `" public access"`, which reproduces the live rule -comment for bitcoin-p2p and fulcrum-ssl. **datum-stratum must pass it -explicitly** — its live comment is `DATUM Gateway Stratum public access` while -the derived default would be `DATUM Stratum public access`, and UFW treats the -comment as part of the rule, so the mismatch rewrites the rule on every run. - -## The upstream check never fails the play - -`wait_for` on the upstream carries `failed_when: false`. The proxy is correctly -configured whether or not the backend happens to be up, and this is the one task -that depends on another machine. The original plays used `ignore_errors: yes`, -which prints a red "ignoring" line; `failed_when: false` is the quieter -equivalent. - -## Restarts - -The handler restarts the `.socket`, not the `.service` — that is what picks up a -changed unit; the service is started by the socket on the next connection. - -**Restarting a socket drops connections that are currently open through it.** -For bitcoin-p2p that means peers reconnect; for datum-stratum it means a mining -client has to reconnect and may lose in-flight shares. The handler only fires -when a unit file actually changes. diff --git a/ansible/roles/socket_proxy/defaults/main.yml b/ansible/roles/socket_proxy/defaults/main.yml deleted file mode 100644 index 6a512ea..0000000 --- a/ansible/roles/socket_proxy/defaults/main.yml +++ /dev/null @@ -1,17 +0,0 @@ ---- -# Required -socket_proxy_name: "" # "bitcoin-p2p" -> bitcoin-p2p-proxy.{socket,service} -socket_proxy_description: "" # "Bitcoin P2P" -> "Bitcoin P2P Proxy Socket" -socket_proxy_listen_port: 0 # public port on the edge host -socket_proxy_upstream_host: "" # Tailscale hostname, e.g. "knots-box" - -# Optional -socket_proxy_upstream_port: "" # defaults to socket_proxy_listen_port -socket_proxy_documentation: "" # Documentation= in both units -socket_proxy_free_bind: false # FreeBind=true: bind before the address is up -socket_proxy_timeout_stop_sec: "" # TimeoutStopSec= -socket_proxy_log_to_journal: false # StandardOutput/StandardError=journal - -# Firewall -socket_proxy_ufw_proto: tcp -socket_proxy_ufw_comment: "" # defaults to " public access" diff --git a/ansible/roles/socket_proxy/handlers/main.yml b/ansible/roles/socket_proxy/handlers/main.yml deleted file mode 100644 index 7fdf421..0000000 --- a/ansible/roles/socket_proxy/handlers/main.yml +++ /dev/null @@ -1,8 +0,0 @@ ---- -# Restarting the .socket is what picks up a changed unit; the .service is -# started by the socket on the next connection. -- name: Restart socket proxy - ansible.builtin.systemd: - name: "{{ socket_proxy_name }}-proxy.socket" - state: restarted - daemon_reload: yes diff --git a/ansible/roles/socket_proxy/tasks/main.yml b/ansible/roles/socket_proxy/tasks/main.yml deleted file mode 100644 index 4315555..0000000 --- a/ansible/roles/socket_proxy/tasks/main.yml +++ /dev/null @@ -1,54 +0,0 @@ ---- -- name: Assert socket_proxy parameters are sane - ansible.builtin.assert: - that: - - socket_proxy_name | length > 0 - - socket_proxy_description | length > 0 - - socket_proxy_listen_port | int > 0 - - socket_proxy_upstream_host | length > 0 - fail_msg: >- - socket_proxy: '{{ socket_proxy_name | default("") }}' needs a name, - a description, a listen port and an upstream host. - quiet: true - -- name: "Create the {{ socket_proxy_name }}-proxy socket unit" - ansible.builtin.template: - src: proxy.socket.j2 - dest: "/etc/systemd/system/{{ socket_proxy_name }}-proxy.socket" - owner: root - group: root - mode: '0644' - notify: Restart socket proxy - -- name: "Create the {{ socket_proxy_name }}-proxy service unit" - ansible.builtin.template: - src: proxy.service.j2 - dest: "/etc/systemd/system/{{ socket_proxy_name }}-proxy.service" - owner: root - group: root - mode: '0644' - notify: Restart socket proxy - -- name: "Enable and start the {{ socket_proxy_name }}-proxy socket" - ansible.builtin.systemd: - name: "{{ socket_proxy_name }}-proxy.socket" - enabled: yes - state: started - daemon_reload: yes - -- name: "Allow the {{ socket_proxy_name }} port through UFW" - community.general.ufw: - rule: allow - port: "{{ socket_proxy_listen_port | string }}" - proto: "{{ socket_proxy_ufw_proto }}" - comment: "{{ socket_proxy_ufw_comment | default(socket_proxy_description ~ ' public access', true) }}" - -# Reachability of the upstream over Tailscale. Deliberately non-fatal: the proxy -# is still correctly configured if the backend happens to be down, and this is -# the one check that depends on another machine being up. -- name: "Verify {{ socket_proxy_upstream_host }} is reachable over Tailscale" - ansible.builtin.wait_for: - host: "{{ socket_proxy_upstream_host }}" - port: "{{ socket_proxy_upstream_port | default(socket_proxy_listen_port, true) }}" - timeout: 10 - failed_when: false diff --git a/ansible/roles/socket_proxy/templates/proxy.service.j2 b/ansible/roles/socket_proxy/templates/proxy.service.j2 deleted file mode 100644 index e59d251..0000000 --- a/ansible/roles/socket_proxy/templates/proxy.service.j2 +++ /dev/null @@ -1,18 +0,0 @@ -[Unit] -Description={{ socket_proxy_description }} Proxy to {{ socket_proxy_upstream_host }} -{% if socket_proxy_documentation %} -Documentation={{ socket_proxy_documentation }} -{% endif %} -Requires={{ socket_proxy_name }}-proxy.socket -After=network.target - -[Service] -Type=notify -ExecStart=/lib/systemd/systemd-socket-proxyd {{ socket_proxy_upstream_host }}:{{ socket_proxy_upstream_port | default(socket_proxy_listen_port, true) }} -{% if socket_proxy_timeout_stop_sec %} -TimeoutStopSec={{ socket_proxy_timeout_stop_sec }} -{% endif %} -{% if socket_proxy_log_to_journal %} -StandardOutput=journal -StandardError=journal -{% endif %} diff --git a/ansible/roles/socket_proxy/templates/proxy.socket.j2 b/ansible/roles/socket_proxy/templates/proxy.socket.j2 deleted file mode 100644 index 0dc1721..0000000 --- a/ansible/roles/socket_proxy/templates/proxy.socket.j2 +++ /dev/null @@ -1,14 +0,0 @@ -[Unit] -Description={{ socket_proxy_description }} Proxy Socket -{% if socket_proxy_documentation %} -Documentation={{ socket_proxy_documentation }} -{% endif %} - -[Socket] -ListenStream={{ socket_proxy_listen_port }} -{% if socket_proxy_free_bind %} -FreeBind=true -{% endif %} - -[Install] -WantedBy=sockets.target diff --git a/ansible/roles/xmrblocks/README.md b/ansible/roles/xmrblocks/README.md deleted file mode 100644 index a7207dd..0000000 --- a/ansible/roles/xmrblocks/README.md +++ /dev/null @@ -1,176 +0,0 @@ -# xmrblocks - -The [Onion Monero Blockchain Explorer](https://github.com/moneroexamples/onion-monero-blockchain-explorer), -on `monerod-box` beside the node, published at `exmrplore.contrapeso.xyz`. - -## Why this role installs a binary instead of building one - -Every other compiled service in this estate builds on its own host — -`bitcoin_knots` and `datum_gateway` both do. This one cannot, and the reason is -worth recording so nobody "fixes" it later. - -`xmrblocks` is not a client of monerod. Its `cmake/FindMonero.cmake` imports 21 -**static libraries** out of a compiled monero source tree and its headers, because -it opens the LMDB directly rather than going through RPC. Building it therefore -means compiling monero first — and the release tarball we install for the daemon -contains only executables, no `.a` files and no headers, so there is nothing on -the box to link against. - -Measured on Debian 13, building only the libraries `xmrblocks` needs: - -| | | -|---|---| -| peak RSS, one `cc1plus` | **~3.2 GB** | -| peak RSS, `-j4` concurrent | **~5.5 GB** | -| wall time, `-j4` | 167 s | -| build tree | 125 MB (source 70 MB) | -| toolchain | 1.6 GB | - -`monerod-box` had **2 GB** when this was measured (it has since been raised to -6 GB, for monerod's own sake — see below). It was tried at 768 MiB and 1800 MiB, -at `-j1` and `-j2`, and every combination was OOM-killed — a single translation unit -exceeding 3 GB is not something lower parallelism can work around. Disk was -never the constraint; the often-quoted "~10 GB" for a monero build describes -upstream's full `make`, which also produces monerod, both wallets and the test -suite, none of which are linked here. - -So the binary is built elsewhere by **`scripts/build-xmrblocks.sh`**, which runs -the whole thing in a `debian:13` container. That matters: an artifact only one -machine can produce is an artifact nobody can audit. Any host with Docker and -~6 GB of RAM reproduces it, and the script verifies the monero source against -binaryFate's GPG signature exactly as `roles/monero` does for the daemon. - -The role refuses to ship anything it has not checked: `xmrblocks_sha256` is -asserted **on the control node, before the copy**, against the value the build -script prints. - -## Three upstream problems, and the fixes - -None of these are documented anywhere. All three were found by building. - -1. **`libasio-dev` is missing from upstream's dependency list.** The Dockerfile - is Ubuntu-based and installs `libboost-all-dev`; Crow wants standalone - `asio.hpp`, not `boost/asio.hpp`. On Debian 13 the build dies with - `asio.hpp: No such file or directory`. - -2. **The explorer does not compile against monero v0.18.5.1.** v0.18.5.0 changed - - ```cpp - crypto::hash get_pruned_transaction_hash(const transaction&, const crypto::hash&); - bool get_pruned_transaction_hash(const transaction&, const crypto::hash&, crypto::hash& res); - ``` - - and `src/page.h` still calls the old form. Upstream's Dockerfile pins - `MONERO_BRANCH=v0.18.4.0`, which is why nobody has hit it. The build script - applies a one-line patch, **anchored on the exact source line** so that if - upstream ever fixes it the build stops loudly instead of silently doing - something else. - - We patch rather than build against v0.18.4.0 deliberately: the current signed - `hashes.txt` only covers the current release, so using the older source would - mean giving up the provenance chain. The LMDB format is `VERSION 5` in both, - so either version reads the other's database safely. - -3. **`FindMonero.cmake` looks for miniupnpc inside the monero build tree** with - `NO_DEFAULT_PATH`, but monero stopped vendoring miniupnp as a submodule. Fixed - with a `-D Xmr_MINIUPNPC_LIBRARY=` cache override pointing at the system - library, rather than patching their cmake. - -`USE_DEVICE_TREZOR=OFF` is also passed: an explorer has no use for hardware -wallet support, and it drops the generated protobuf units, which are among the -heaviest to compile. `xmrblocks` was verified to link against the resulting -libraries. - -## Two things the unit gets right that look wrong - -**`WorkingDirectory` is load-bearing.** `src/page.h` hardcodes -`#define TMPL_DIR "./templates"` — the HTML is resolved relative to the process -working directory, at request time. Get it wrong and the service starts happily, -listens on 8081, and returns 500 for every page. That is also why the health -check fetches a real page and greps it rather than just probing the port. - -**`ReadWritePaths` must include the blockchain directory**, even though the -database is opened read-only. `MicroCore::init` sets `DBF_RDONLY`, and monero's -`db_lmdb.cpp` turns that into `mdb_flags = MDB_RDONLY` **without `MDB_NOLOCK`** — -so LMDB still registers this process in the reader table inside `lock.mdb`. With -`ProtectSystem=strict` and nothing writable, the open fails outright. The -intuitive "it only reads, grant it nothing" is wrong. `data.mdb` itself remains -read-only to the process, and writing to monerod's lock table is what any second -reader does. - -## Exposure - -Published by `caddy_site` on the edge host at `exmrplore.contrapeso.xyz`, with -TLS terminated there and the upstream reached over the tailnet via MagicDNS. - -The explorer listens on `0.0.0.0:8081`. Nothing on the LAN or the internet -reaches that directly: Tailscale's `ts-input` chain sits ahead of UFW, so only -tailnet peers can, and the Headscale ACL grants `tag:prd-vipy` exactly -`tag:monerod-box:8081`. A node with no ACL rule is *invisible* rather than -merely unreachable — it never appears in `tailscale status`, MagicDNS does not -resolve it, and the symptom is a Caddy 502 saying "no such host". - -**There is no authentication, by choice.** A block explorer serves public chain -data. But note what that includes: `--enable-pusher` is on (upstream's default), -so any visitor can submit a raw transaction and have *our* node relay it. That is -a deliberate decision, not an oversight — drop `xmrblocks_enable_pusher` to -`false` to close it. - -`--enable-json-api` and `--enable-autorefresh-option` are likewise upstream's -defaults. - -## Monitoring - -| Endpoint | Source | Answers | -|---|---|---| -| `services/monerod-box-local/xmrblocks` | `infra/401` via `monitored_services` | is the unit active | -| `probe/xmrblocks` | this role + `infra/403` | does it serve a real page | -| `dns/exmrplore.contrapeso.xyz`, `public/exmrplore` | `infra/402` | can the internet reach it | - -`public/exmrplore` asserts **200**, unlike the Monero RPC endpoint next door -which asserts 401 — this one genuinely has no auth in front of it. - -The timer carries no `Requires=xmrblocks.service`; see the comment in -`healthcheck.timer.j2` for why that would be backwards. - -## Backups: none, deliberately - -Everything here is derived or reproducible: the binary comes from -`scripts/build-xmrblocks.sh`, the templates ship with it, and the data it serves -is monerod's blockchain, which is itself excluded for the same reason. There is -no state on this service worth a backup job. Recorded as a decision so it does -not read as an oversight — the same treatment `phoenixd` and `monero` get. - -## Upgrading - -1. Edit the pins at the top of `scripts/build-xmrblocks.sh` (`EXPLORER_COMMIT`, - and `MONERO_VERSION` + `MONERO_SRC_SHA256` together). -2. Run it. It prints the new artifact sha256. -3. Put that in `xmrblocks_sha256` and `xmrblocks_explorer_commit` here. -4. Converge. The role asserts the checksum before copying anything. - -If the patch anchor no longer matches, the script stops — read the diff before -forcing it. - -## Runtime cost, and the RAM history - -On **2026-09-20 the VM was raised from 2 GB to 6 GB**, and that was not for the -explorer. monerod alone had outgrown 2 GB once the chain passed ~39%: it was -OOM-killed twice in one morning at 1.74 GB anon-rss, restarting every hour. With -6 GB the box sits at ~1.2 GB used of 5.9 GB with both services running -(monerod ~1.0 GB, xmrblocks ~96 MB). - -**The disk is the real performance limit, not memory.** `proxmox-tank-1` is a -spinning-disk mirror, and xmrblocks says so on startup: *"The blockchain is on a -rotating drive: this will be very slow, use an SSD if possible"*. Measured -consequences, both of which the role accounts for: - -- first start takes ~98 seconds to open the LMDB before it binds, so - `wait_for` allows 300s -- a COLD index page took **68 seconds**, while the two requests after it took - 0.87s and 0.012s — so the health check allows 90s and its unit 180s. A short - timeout there does not detect a slow explorer, it manufactures false DOWNs. - -Footprint note: xmrblocks mmaps the LMDB, so most of its apparent size is -reclaimable page cache rather than anonymous memory, and `--concurrency` is held -at 2 for the same reason. diff --git a/ansible/roles/xmrblocks/defaults/main.yml b/ansible/roles/xmrblocks/defaults/main.yml deleted file mode 100644 index 8661676..0000000 --- a/ansible/roles/xmrblocks/defaults/main.yml +++ /dev/null @@ -1,82 +0,0 @@ -# xmrblocks - the Onion Monero Blockchain Explorer. -# https://github.com/moneroexamples/onion-monero-blockchain-explorer -# -# This role INSTALLS a prebuilt binary; it does not compile one. xmrblocks links -# 21 static libraries out of a compiled monero source tree, and building that -# needs ~3.2 GB of RAM for a single compiler (~5.5 GB at -j4). monerod-box has -# 2 GB, so it cannot build this at any -j. See README.md. -# -# The artifact is produced by scripts/build-xmrblocks.sh, which runs the whole -# build in a debian:13 container - so it is reproducible on any machine with -# Docker, not just the one that happened to build it. - -# --- The artifact ------------------------------------------------------------ -# A directory on the CONTROL NODE holding `xmrblocks`, `templates/` and MANIFEST. -xmrblocks_artifact_dir: "{{ lookup('env', 'HOME') }}/personal_infra_artifacts/xmrblocks" - -# Pinned checksum of the binary we intend to ship. Asserted on the control node -# BEFORE anything is copied: shipping an unverified binary to a host is the one -# thing this role must never do. Update it together with the build pins in -# scripts/build-xmrblocks.sh. -xmrblocks_sha256: "a40c495e73a44fbb3b666fcd8a01466a2aef12ed1906558f3d5b7e458c256af6" - -# Recorded so the deployed host can be matched back to a build. -xmrblocks_explorer_commit: "29593b709edeb23daba035009ccd134a5862543a" -xmrblocks_monero_version: "0.18.5.1" - -# --- Install layout ---------------------------------------------------------- -# The binary hardcodes TMPL_DIR as "./templates", relative to the working -# directory - so templates MUST sit beside the binary and the unit MUST set -# WorkingDirectory. Get that wrong and it starts fine, then fails per-request. -xmrblocks_dir: /opt/xmrblocks - -# Runs as the monero user: the explorer opens monerod's LMDB directly, and -# /var/lib/monero is 0750 monero:monero. It opens it READ-ONLY (DBF_RDONLY in -# MicroCore::init), so it is safe alongside a running monerod. -xmrblocks_user: monero -xmrblocks_group: monero - -# --- Runtime ----------------------------------------------------------------- -# The LMDB directory, not the data directory: monerod's data-dir plus /lmdb. -xmrblocks_bc_path: /var/lib/monero/lmdb - -# 0.0.0.0 so the edge host can reach it over the tailnet. Nothing on the LAN or -# the internet can: Tailscale's ts-input chain sits ahead of UFW, and the only -# public route is the Caddy vhost on vipy. -xmrblocks_bind: "0.0.0.0" -xmrblocks_port: 8081 - -# The unrestricted RPC on loopback. Used for the mempool view and tx pushing. -xmrblocks_daemon_url: "127.0.0.1:18081" -# monerod's --rpc-login guards BOTH its RPC servers, so the explorer needs it. -xmrblocks_rpc_user: "{{ monero_rpc_user | default('') }}" -xmrblocks_rpc_password: "{{ monero_rpc_password | default('') }}" - -# Upstream's own defaults (the Dockerfile CMD). -# NOTE: enable-pusher lets a visitor submit a raw transaction through YOUR node. -# It is a deliberate choice here, not an oversight. -xmrblocks_enable_json_api: true -xmrblocks_enable_autorefresh: true -xmrblocks_enable_pusher: true - -# Crow's thread count. Held low on purpose: this box has 2 GB and monerod -# already uses ~1.3 GB of it. -xmrblocks_concurrency: 2 - -# The unit file carries the RPC login, so --diff is suppressed by default. -xmrblocks_reveal_config: false - -# --- Health check ------------------------------------------------------------ -# WHERE TO REPORT HEALTH — the one place to plug in monitoring. Empty means -# check, exit honestly, report nowhere. -healthcheck_push_url: "" -# Bearer token for the Gatus external endpoint. Required whenever a push URL -# is set: Gatus rejects an unauthenticated push with 401. -healthcheck_push_token: "" - -# WHEN TO CHECK. Every 5 minutes, offset 2 minutes from monerod's own check so -# the two do not land together. On NVMe a cold page render measured 29s and a -# warm one 0.2s (2026-10-17); a 5-minute poll is what keeps it warm. See -# roles/monero/defaults for why the name is prefixed and why this is paired -# with the 11m heartbeat in infra/403_service_probe_registration.yml. -xmrblocks_healthcheck_on_calendar: "*:2/5" diff --git a/ansible/roles/xmrblocks/handlers/main.yml b/ansible/roles/xmrblocks/handlers/main.yml deleted file mode 100644 index 7be239a..0000000 --- a/ansible/roles/xmrblocks/handlers/main.yml +++ /dev/null @@ -1,12 +0,0 @@ ---- -- name: Restart xmrblocks - systemd: - name: xmrblocks - state: restarted - daemon_reload: yes - -- name: Restart xmrblocks health check timer - systemd: - name: xmrblocks-healthcheck.timer - state: restarted - daemon_reload: yes diff --git a/ansible/roles/xmrblocks/tasks/healthcheck.yml b/ansible/roles/xmrblocks/tasks/healthcheck.yml deleted file mode 100644 index 7a3a3a1..0000000 --- a/ansible/roles/xmrblocks/tasks/healthcheck.yml +++ /dev/null @@ -1,54 +0,0 @@ ---- -# "Does the explorer actually serve a page", as opposed to infra/401's "is the -# unit active". xmrblocks can be perfectly `active` and still fail every -# request - a missing templates directory does exactly that. - -- name: Install the health check's dependencies - apt: - name: - - curl - state: present - -- name: Create the xmrblocks health check script - ansible.builtin.template: - src: healthcheck.sh.j2 - dest: /usr/local/bin/xmrblocks-healthcheck-push.sh - owner: root - group: root - mode: '0755' - validate: "bash -n %s" - -# 0600: carries the Gatus bearer token. -- name: Create the xmrblocks health check systemd service - ansible.builtin.template: - src: healthcheck.service.j2 - dest: /etc/systemd/system/xmrblocks-healthcheck.service - owner: root - group: root - mode: '0600' - notify: Restart xmrblocks health check timer - -- name: Create the xmrblocks health check systemd timer - ansible.builtin.template: - src: healthcheck.timer.j2 - dest: /etc/systemd/system/xmrblocks-healthcheck.timer - owner: root - group: root - mode: '0644' - notify: Restart xmrblocks health check timer - -- name: Reload systemd daemon after health check units - systemd: - daemon_reload: yes - -# Ungated: enabling a timer is deployment, not monitoring. -- name: Enable and restart the xmrblocks health check timer - systemd: - name: xmrblocks-healthcheck.timer - enabled: yes - state: restarted - daemon_reload: yes - -- name: Run the xmrblocks health check once to arm the timer - ansible.builtin.command: systemctl start xmrblocks-healthcheck.service - changed_when: false diff --git a/ansible/roles/xmrblocks/tasks/install.yml b/ansible/roles/xmrblocks/tasks/install.yml deleted file mode 100644 index 688e5bc..0000000 --- a/ansible/roles/xmrblocks/tasks/install.yml +++ /dev/null @@ -1,129 +0,0 @@ ---- -# Install the prebuilt xmrblocks artifact produced by scripts/build-xmrblocks.sh. -# -# The verification happens on the CONTROL NODE, before a single byte is copied. -# Shipping a binary to a host and checking it afterwards is not verification, -# it is an apology. - -- name: Assert the artifact checksum is pinned - ansible.builtin.assert: - that: - - xmrblocks_sha256 | length == 64 - quiet: true - fail_msg: >- - xmrblocks_sha256 is not set. Build the artifact with - scripts/build-xmrblocks.sh and pin the sha it prints, so this role cannot - ship a binary nobody has checked. - -- name: Check the artifact on the control node - ansible.builtin.stat: - path: "{{ xmrblocks_artifact_dir }}/xmrblocks" - checksum_algorithm: sha256 - get_checksum: yes - delegate_to: localhost - become: no - register: xmrblocks_local - changed_when: false - -- name: Assert the artifact exists and matches the pinned checksum - ansible.builtin.assert: - that: - - xmrblocks_local.stat.exists - - xmrblocks_local.stat.checksum == xmrblocks_sha256 - quiet: true - fail_msg: >- - {{ xmrblocks_artifact_dir }}/xmrblocks is - {{ 'missing - run scripts/build-xmrblocks.sh' - if not xmrblocks_local.stat.exists - else 'sha256 ' ~ xmrblocks_local.stat.checksum ~ ', expected ' ~ xmrblocks_sha256 }} - -- name: Check the templates directory on the control node - ansible.builtin.stat: - path: "{{ xmrblocks_artifact_dir }}/templates/index.html" - delegate_to: localhost - become: no - register: xmrblocks_local_templates - changed_when: false - -# The binary resolves TMPL_DIR as "./templates" at request time, so a missing -# templates directory is not a startup error - it is a 500 on the first page -# view. Catch it here instead. -- name: Assert the templates came with the artifact - ansible.builtin.assert: - that: - - xmrblocks_local_templates.stat.exists - quiet: true - fail_msg: "{{ xmrblocks_artifact_dir }}/templates/index.html is missing - the artifact is incomplete" - -- name: Install the runtime shared libraries xmrblocks needs - apt: - name: - # From `objdump -p xmrblocks | grep NEEDED`. Boost is version-locked to - # the build image's Debian release; monerod-box is Debian 13 and the - # artifact is built in debian:13, so 1.83 matches on both sides. - - libboost-chrono1.83.0 - - libboost-filesystem1.83.0 - - libboost-program-options1.83.0 - - libboost-serialization1.83.0 - - libboost-thread1.83.0 - - libhidapi-libusb0 - - libunbound8 - - libunwind8 - state: present - update_cache: yes - -- name: Create the xmrblocks directory - ansible.builtin.file: - path: "{{ xmrblocks_dir }}" - state: directory - owner: root - group: root - mode: '0755' - -- name: Install the xmrblocks binary - ansible.builtin.copy: - src: "{{ xmrblocks_artifact_dir }}/xmrblocks" - dest: "{{ xmrblocks_dir }}/xmrblocks" - owner: root - group: root - mode: '0755' - notify: Restart xmrblocks - -- name: Install the xmrblocks templates - ansible.builtin.copy: - src: "{{ xmrblocks_artifact_dir }}/templates/" - dest: "{{ xmrblocks_dir }}/templates/" - owner: root - group: root - mode: '0644' - directory_mode: '0755' - notify: Restart xmrblocks - -- name: Record which build is deployed - ansible.builtin.copy: - src: "{{ xmrblocks_artifact_dir }}/MANIFEST" - dest: "{{ xmrblocks_dir }}/MANIFEST" - owner: root - group: root - mode: '0644' - -- name: Assert the blockchain database is where we think it is - ansible.builtin.stat: - path: "{{ xmrblocks_bc_path }}/data.mdb" - # data.mdb is the whole blockchain - tens of GB and growing. stat computes a - # sha1 by DEFAULT, so without these it hashes the entire chain just to answer - # "does this file exist", and the play hangs for many minutes. - get_checksum: no - get_mime: no - get_attributes: no - register: xmrblocks_lmdb - -- name: Fail if the LMDB is missing - ansible.builtin.assert: - that: - - xmrblocks_lmdb.stat.exists - quiet: true - fail_msg: >- - No LMDB at {{ xmrblocks_bc_path }}/data.mdb. xmrblocks reads monerod's - database directly, so it has nothing to serve. Check that monerod is - deployed and its data disk is mounted. diff --git a/ansible/roles/xmrblocks/tasks/main.yml b/ansible/roles/xmrblocks/tasks/main.yml deleted file mode 100644 index 8a024a2..0000000 --- a/ansible/roles/xmrblocks/tasks/main.yml +++ /dev/null @@ -1,6 +0,0 @@ ---- -# import_tasks, not include_tasks: static imports stay visible to --list-tasks, -# which is the main verification tool in this repo. -- ansible.builtin.import_tasks: install.yml -- ansible.builtin.import_tasks: service.yml -- ansible.builtin.import_tasks: healthcheck.yml diff --git a/ansible/roles/xmrblocks/tasks/service.yml b/ansible/roles/xmrblocks/tasks/service.yml deleted file mode 100644 index 38fecd3..0000000 --- a/ansible/roles/xmrblocks/tasks/service.yml +++ /dev/null @@ -1,57 +0,0 @@ ---- -- name: Assert the daemon RPC credentials are available - ansible.builtin.assert: - that: - - xmrblocks_rpc_user | length > 0 - - xmrblocks_rpc_password | length > 0 - quiet: true - fail_msg: >- - xmrblocks needs monerod's RPC login: --rpc-login guards both of monerod's - RPC servers, so without it the mempool view and the pusher get 401. - -- name: Create the xmrblocks systemd unit - ansible.builtin.template: - src: xmrblocks.service.j2 - dest: /etc/systemd/system/xmrblocks.service - owner: root - group: root - mode: '0644' - # The unit embeds --daemon-login in ExecStart, so --diff would print the - # RPC password. -e xmrblocks_reveal_config=true when you need to see it. - diff: "{{ xmrblocks_reveal_config | default(false) | bool }}" - notify: Restart xmrblocks - -- name: Reload systemd daemon - systemd: - daemon_reload: yes - -- name: Enable and start xmrblocks - systemd: - name: xmrblocks - enabled: yes - state: started - -- name: Flush handlers so xmrblocks is running before we probe it - meta: flush_handlers - -# Generous on purpose. xmrblocks opens the whole LMDB before it binds, and this -# pool is a spinning-disk mirror - it logs "The blockchain is on a rotating -# drive: this will be very slow" and took 98 SECONDS to come up on first start. -# 90s was not enough and produced a scary failure on a service that was fine. -- name: Wait for the xmrblocks HTTP port - ansible.builtin.wait_for: - port: "{{ xmrblocks_port }}" - host: 127.0.0.1 - delay: 5 - timeout: 300 - ignore_errors: yes - -- name: Check the xmrblocks service status - systemd: - name: xmrblocks - register: xmrblocks_status - changed_when: false - -- name: Display the xmrblocks service status - ansible.builtin.debug: - msg: "xmrblocks is {{ 'running' if xmrblocks_status.status.ActiveState == 'active' else 'NOT running' }}" diff --git a/ansible/roles/xmrblocks/templates/healthcheck.service.j2 b/ansible/roles/xmrblocks/templates/healthcheck.service.j2 deleted file mode 100644 index ffae4da..0000000 --- a/ansible/roles/xmrblocks/templates/healthcheck.service.j2 +++ /dev/null @@ -1,20 +0,0 @@ -[Unit] -Description=Onion Monero Blockchain Explorer Health Check -After=network.target xmrblocks.service - -[Service] -Type=oneshot -User=root -ExecStart=/usr/local/bin/xmrblocks-healthcheck-push.sh -# Longer than the script's 90s curl plus its push retries. A oneshot defaults to -# TimeoutStartSec=90, which would kill the check mid-request on a cold cache and -# report a failure that says nothing about the service. -TimeoutStartSec=180 -Environment=HEALTHCHECK_PUSH_URL={{ healthcheck_push_url }} -Environment=HEALTHCHECK_PUSH_TOKEN={{ healthcheck_push_token }} -Environment=XMRBLOCKS_URL=http://127.0.0.1:{{ xmrblocks_port }} -StandardOutput=journal -StandardError=journal - -[Install] -WantedBy=multi-user.target diff --git a/ansible/roles/xmrblocks/templates/healthcheck.sh.j2 b/ansible/roles/xmrblocks/templates/healthcheck.sh.j2 deleted file mode 100644 index 2effe2a..0000000 --- a/ansible/roles/xmrblocks/templates/healthcheck.sh.j2 +++ /dev/null @@ -1,59 +0,0 @@ -#!/bin/bash -# xmrblocks health check — managed by Ansible (roles/xmrblocks) -# -# The exit code is the answer and systemd keeps it: -# systemctl is-failed xmrblocks-healthcheck.service -# Reporting anywhere else is optional and generic. -# -# This fetches a real page rather than just opening the port. The binary -# resolves its HTML as "./templates" relative to the working directory, so a -# deploy that got WorkingDirectory wrong listens happily on 8081 and fails -# every request. A TCP check would call that healthy. - -URL="${XMRBLOCKS_URL:-http://127.0.0.1:8081}" -PUSH_URL="${HEALTHCHECK_PUSH_URL:-}" -PUSH_TOKEN="${HEALTHCHECK_PUSH_TOKEN:-}" - -report() { - local status=$1 - local msg=$2 - - # No push URL is normal, not an error: the exit code below is still a - # complete answer for anything reading unit state. - [ -n "$PUSH_URL" ] || return 0 - - local encoded_msg="${msg// /%20}" - local _ok=false - [ "${status}" = "up" ] && _ok=true - curl -s --max-time 15 --retry 2 -o /dev/null -X POST \ - -H "Authorization: Bearer ${PUSH_TOKEN}" \ - "${PUSH_URL}?success=${_ok}&error=${encoded_msg}" || true -} - -# 90s, not the obvious 10 or 25. The index page renders ~46 KB from the LMDB, -# and this pool is a spinning-disk mirror shared with a syncing monerod: a COLD -# request measured 68 SECONDS, while the two immediately after it took 0.87s and -# 0.012s. A short timeout here does not detect a slow explorer, it manufactures -# a false DOWN every time the page cache has gone cold between runs. -body=$(curl -s --max-time 90 -w '\n%{http_code}' "${URL}/" 2>/dev/null) -code=$(printf '%s' "$body" | tail -1) - -if [ -z "$code" ] || [ "$code" = "000" ]; then - report "down" "xmrblocks not responding" - exit 1 -fi - -if [ "$code" != "200" ]; then - report "down" "xmrblocks HTTP ${code}" - exit 1 -fi - -# A 200 that is not the explorer is still a failure. The index page always -# carries the project name in its title. -if ! printf '%s' "$body" | grep -qi "monero"; then - report "down" "xmrblocks served 200 but not an explorer page (templates missing?)" - exit 1 -fi - -report "up" "OK" -exit 0 diff --git a/ansible/roles/xmrblocks/templates/healthcheck.timer.j2 b/ansible/roles/xmrblocks/templates/healthcheck.timer.j2 deleted file mode 100644 index b6c9ba8..0000000 --- a/ansible/roles/xmrblocks/templates/healthcheck.timer.j2 +++ /dev/null @@ -1,20 +0,0 @@ -[Unit] -Description=Onion Monero Blockchain Explorer Health Check Timer -# Deliberately NO `Requires=xmrblocks.service`. Requires on a timer stops the -# timer when the watched unit stops - "if the thing I am watching goes down, -# stop watching it" - leaving nothing to re-arm when it returns. Fulcrum's -# timer carried it and sat active+enabled for seven months without firing. - -[Timer] -OnBootSec=2min -# Every 5 minutes (see xmrblocks_healthcheck_on_calendar). The check renders a -# full page out of the LMDB: 29s cold, 0.2s warm on NVMe - frequent polling is -# what keeps it warm. The Gatus heartbeat window is the standard 11m, set in -# infra/403_service_probe_registration.yml, and MUST stay above this interval. -OnCalendar={{ xmrblocks_healthcheck_on_calendar }} -# Small: a delay comparable to the interval would eat the heartbeat's slack. -RandomizedDelaySec=30 -Persistent=true - -[Install] -WantedBy=timers.target diff --git a/ansible/roles/xmrblocks/templates/xmrblocks.service.j2 b/ansible/roles/xmrblocks/templates/xmrblocks.service.j2 deleted file mode 100644 index 64c58af..0000000 --- a/ansible/roles/xmrblocks/templates/xmrblocks.service.j2 +++ /dev/null @@ -1,58 +0,0 @@ -[Unit] -Description=Onion Monero Blockchain Explorer (xmrblocks) -Documentation=https://github.com/moneroexamples/onion-monero-blockchain-explorer -After=network-online.target monerod.service -Wants=network-online.target -# The explorer reads monerod's LMDB directly. If the blockchain disk is not -# mounted there is nothing to serve, and without this the unit would start and -# fail on every request instead of refusing to start at all. -RequiresMountsFor={{ xmrblocks_bc_path }} - -[Service] -Type=simple -User={{ xmrblocks_user }} -Group={{ xmrblocks_group }} - -# NOT decoration: the binary hardcodes TMPL_DIR as "./templates", so it resolves -# its HTML relative to the working directory. Without this it starts happily and -# then 500s on the first page view. -WorkingDirectory={{ xmrblocks_dir }} - -ExecStart={{ xmrblocks_dir }}/xmrblocks \ - --bc-path {{ xmrblocks_bc_path }} \ - --daemon-url {{ xmrblocks_daemon_url }} \ - --daemon-login {{ xmrblocks_rpc_user }}:{{ xmrblocks_rpc_password }} \ - --bindaddr {{ xmrblocks_bind }} \ - --port {{ xmrblocks_port }} \ - --concurrency {{ xmrblocks_concurrency }}{% if xmrblocks_enable_json_api %} \ - --enable-json-api{% endif %}{% if xmrblocks_enable_autorefresh %} \ - --enable-autorefresh-option{% endif %}{% if xmrblocks_enable_pusher %} \ - --enable-pusher{% endif %} - - -Restart=always -RestartSec=20 -TimeoutStopSec=30 - -StandardOutput=journal -StandardError=journal - -# Hardening. The explorer serves static templates and reads the chain. -NoNewPrivileges=true -PrivateTmp=true -ProtectSystem=strict -ProtectHome=read-only - -# The LMDB directory must be WRITABLE even though the database is opened -# read-only. monero sets `mdb_flags = MDB_RDONLY` without MDB_NOLOCK -# (db_lmdb.cpp), so LMDB still registers this process in the reader table -# inside lock.mdb. With ProtectSystem=strict and no ReadWritePaths the open -# fails outright - the obvious "it only reads, so grant nothing" is wrong here. -# Writing to monerod's own lock table is normal and is what any second reader -# does; data.mdb itself stays read-only to this process. -ReadWritePaths={{ xmrblocks_bc_path }} - -LimitNOFILE=16384 - -[Install] -WantedBy=multi-user.target diff --git a/ansible/services/arbret/deploy_arbret_edge_playbook.yml b/ansible/services/arbret/deploy_arbret_edge_playbook.yml deleted file mode 100644 index 3449e95..0000000 --- a/ansible/services/arbret/deploy_arbret_edge_playbook.yml +++ /dev/null @@ -1,59 +0,0 @@ ---- -# arbret's footprint on the edge host - and ONLY that. -# -# arbret itself (app, worker, Postgres, timers, its backup dump) is deployed by -# the arbret owner's own Ansible repo since prd-arbret was merged into vipy on -# 2026-10-17. This playbook owns the parts of the shared host that are ours: -# the public front door, the tailnet port, and the backup directory contract. -# See arbret_handoff.md at the repo root for the division of labour. -- name: Publish arbret on the edge host - hosts: edge - become: yes - vars: - # Local port the arbret app binds on 127.0.0.1, per the arbret owner. - arbret_port: 3100 - arbret_domain: arbret.com - # Admin port reachable from the tailnet only (ACL in headscale's acl.json.j2). - arbret_tailnet_port: 4117 - - tasks: - - name: Publish arbret through Caddy - ansible.builtin.include_role: - name: caddy_site - vars: - caddy_site_name: arbret - caddy_site_domain: "{{ arbret_domain }}" - caddy_site_upstream: "localhost:{{ arbret_port }}" - - - name: Redirect www to the apex - ansible.builtin.include_role: - name: caddy_site - vars: - caddy_site_name: arbret-www - caddy_site_domain: "www.{{ arbret_domain }}" - caddy_site_body: | - redir https://{{ arbret_domain }}{uri} permanent - - # tailscale0 only. UFW's default-deny applies to the tailnet interface too, - # so the headscale ACL alone does not open it - and this must never be - # reachable on the public interface. - - name: Allow the arbret admin port from the tailnet - community.general.ufw: - rule: allow - interface: tailscale0 - direction: in - port: "{{ arbret_tailnet_port | string }}" - proto: tcp - comment: "arbret admin, tailnet only" - - # The backup contract directory (arbret_handoff.md section 3.3). The arbret - # repo creates it identically; creating it here too means small-backups-box - # can pull - an empty dir - before arbret's first dump, so one missing - # source does not turn the whole pull job red. - - name: Ensure the arbret backup directory exists - ansible.builtin.file: - path: /opt/backups/arbret - state: directory - owner: root - group: backup-pull - mode: "0750" diff --git a/ansible/services/bitcoin-knots/bitcoin_knots_vars.yml b/ansible/services/bitcoin-knots/bitcoin_knots_vars.yml new file mode 100644 index 0000000..c9bd7ca --- /dev/null +++ b/ansible/services/bitcoin-knots/bitcoin_knots_vars.yml @@ -0,0 +1,38 @@ +# Bitcoin Knots Configuration Variables + +# Version - REQUIRED: Specify exact version/tag to build +bitcoin_knots_version: "v29.2.knots20251110" # Must specify exact version/tag +bitcoin_knots_version_short: "29.2.knots20251110" # Version without 'v' prefix (for tarball URLs) + +# Directories +bitcoin_knots_dir: /opt/bitcoin-knots +bitcoin_knots_source_dir: "{{ bitcoin_knots_dir }}/source" +bitcoin_data_dir: /var/lib/bitcoin # Standard location for config, logs, wallets +bitcoin_large_data_dir: /mnt/knots_data # Custom location for blockchain data (blocks, chainstate) +bitcoin_conf_dir: /etc/bitcoin + +# Network +bitcoin_rpc_port: 8332 +bitcoin_p2p_port: 8333 +bitcoin_rpc_bind: "0.0.0.0" + +# Build options +bitcoin_build_jobs: 4 # Parallel build jobs (-j flag), adjust based on CPU cores +bitcoin_build_prefix: /usr/local + +# Configuration options +bitcoin_enable_txindex: true # Set to true if transaction index needed (REQUIRED for Electrum servers like Electrs/ElectrumX) +bitcoin_max_connections: 125 +# dbcache will be calculated as 90% of host RAM automatically in playbook + +# ZMQ Configuration +bitcoin_zmq_enabled: true +bitcoin_zmq_bind: "tcp://0.0.0.0" +bitcoin_zmq_port_rawblock: 28332 +bitcoin_zmq_port_rawtx: 28333 +bitcoin_zmq_port_hashblock: 28334 +bitcoin_zmq_port_hashtx: 28335 + +# Service user +bitcoin_user: bitcoin +bitcoin_group: bitcoin diff --git a/ansible/services/bitcoin-knots/deploy_bitcoin_knots_playbook.yml b/ansible/services/bitcoin-knots/deploy_bitcoin_knots_playbook.yml index d67006b..d818945 100644 --- a/ansible/services/bitcoin-knots/deploy_bitcoin_knots_playbook.yml +++ b/ansible/services/bitcoin-knots/deploy_bitcoin_knots_playbook.yml @@ -1,35 +1,916 @@ ---- -# Bitcoin Knots: full node built from source, with PGP signature and SHA256 -# verification of the release tarball. The build is guarded by a binary-exists -# check, so a converged host skips the whole 30-60 minute compile. -# -# The chain lives in bitcoin_large_data_dir (/mnt/knots_data, ~875 GB). Nothing -# here touches it; the only `state: absent` tasks target the build tree under -# /opt/bitcoin-knots and run only when the binary is missing. - name: Build and Deploy Bitcoin Knots from Source - hosts: bitcoin + hosts: knots_box_local become: yes + vars_files: + - ../../infra_vars.yml + - ../../services_config.yml + - ../../infra_secrets.yml + - ./bitcoin_knots_vars.yml vars: - # Preserves the push URL this check has been reporting to. The role knows - # nothing about Uptime Kuma — this is just "a URL that accepts a ping". - healthcheck_push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/probe_bitcoin-knots/external" - healthcheck_push_token: "{{ gatus_push_tokens[inventory_hostname] }}" - roles: - - bitcoin_knots + bitcoin_repo_url: "https://github.com/bitcoinknots/bitcoin.git" + bitcoin_sigs_base_url: "https://raw.githubusercontent.com/bitcoinknots/guix.sigs/knots" + bitcoin_version_major: "{{ bitcoin_knots_version_short | regex_replace('^(\\d+)\\..*', '\\1') }}" + bitcoin_source_tarball_url: "https://bitcoinknots.org/files/{{ bitcoin_version_major }}.x/{{ bitcoin_knots_version_short }}/bitcoin-{{ bitcoin_knots_version_short }}.tar.gz" + uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}" -- name: Setup public Bitcoin P2P forwarding on the edge host - hosts: edge - become: yes tasks: - - name: Expose Bitcoin P2P through a socket proxy - ansible.builtin.include_role: - name: socket_proxy - vars: - socket_proxy_name: bitcoin-p2p - socket_proxy_description: "Bitcoin P2P" - socket_proxy_listen_port: "{{ hostvars['knots_box_local'].bitcoin_p2p_port }}" - socket_proxy_upstream_host: "{{ hostvars['knots_box_local'].ansible_host }}" - socket_proxy_documentation: "https://github.com/bitcoin/bitcoin" - socket_proxy_free_bind: true - socket_proxy_timeout_stop_sec: 5 - socket_proxy_log_to_journal: true + - name: Calculate 90% of system RAM for dbcache + set_fact: + bitcoin_dbcache_mb: "{{ (ansible_memtotal_mb | float * 0.9) | int }}" + changed_when: false + + - name: Display calculated dbcache value + debug: + msg: "Setting dbcache to {{ bitcoin_dbcache_mb }} MB (90% of {{ ansible_memtotal_mb }} MB total RAM)" + + + - name: Install build dependencies + apt: + name: + - build-essential + - libtool + - autotools-dev + - automake + - pkg-config + - bsdmainutils + - python3 + - python3-pip + - libevent-dev + - libboost-system-dev + - libboost-filesystem-dev + - libboost-test-dev + - libboost-thread-dev + - libboost-chrono-dev + - libboost-program-options-dev + - libboost-dev + - libssl-dev + - libdb-dev + - libminiupnpc-dev + - libzmq3-dev + - libnatpmp-dev + - libsqlite3-dev + - git + - curl + - wget + - cmake + state: present + update_cache: yes + + - name: Create bitcoin group + group: + name: "{{ bitcoin_group }}" + system: yes + state: present + + - name: Create bitcoin user + user: + name: "{{ bitcoin_user }}" + group: "{{ bitcoin_group }}" + system: yes + shell: /usr/sbin/nologin + home: "{{ bitcoin_data_dir }}" + create_home: yes + state: present + + - name: Create bitcoin-knots directory + file: + path: "{{ bitcoin_knots_dir }}" + state: directory + owner: root + group: root + mode: '0755' + + - name: Create bitcoin-knots source directory + file: + path: "{{ bitcoin_knots_source_dir }}" + state: directory + owner: root + group: root + mode: '0755' + + - name: Create bitcoin data directory (for config, logs, wallets) + file: + path: "{{ bitcoin_data_dir }}" + state: directory + owner: "{{ bitcoin_user }}" + group: "{{ bitcoin_group }}" + mode: '0750' + + - name: Create bitcoin large data directory (for blockchain) + file: + path: "{{ bitcoin_large_data_dir }}" + state: directory + owner: "{{ bitcoin_user }}" + group: "{{ bitcoin_group }}" + mode: '0750' + + - name: Create bitcoin config directory + file: + path: "{{ bitcoin_conf_dir }}" + state: directory + owner: root + group: root + mode: '0755' + + - name: Check if bitcoind binary already exists + stat: + path: "{{ bitcoin_build_prefix }}/bin/bitcoind" + register: bitcoind_binary_exists + changed_when: false + + - name: Install gnupg for signature verification + apt: + name: gnupg + state: present + when: not bitcoind_binary_exists.stat.exists + + - name: Import Luke Dashjr's Bitcoin Knots signing key + command: gpg --keyserver hkps://keyserver.ubuntu.com --recv-keys 90C8019E36C2E964 + register: key_import + changed_when: "'already in secret keyring' not in key_import.stdout and 'already in public keyring' not in key_import.stdout" + when: not bitcoind_binary_exists.stat.exists + failed_when: key_import.rc != 0 + + - name: Display imported key fingerprint + command: gpg --fingerprint 90C8019E36C2E964 + register: key_fingerprint + changed_when: false + when: not bitcoind_binary_exists.stat.exists + + - name: Download SHA256SUMS file + get_url: + url: "https://bitcoinknots.org/files/{{ bitcoin_version_major }}.x/{{ bitcoin_knots_version_short }}/SHA256SUMS" + dest: "/tmp/bitcoin-knots-{{ bitcoin_knots_version_short }}-SHA256SUMS" + mode: '0644' + when: not bitcoind_binary_exists.stat.exists + + - name: Download SHA256SUMS.asc signature file + get_url: + url: "https://bitcoinknots.org/files/{{ bitcoin_version_major }}.x/{{ bitcoin_knots_version_short }}/SHA256SUMS.asc" + dest: "/tmp/bitcoin-knots-{{ bitcoin_knots_version_short }}-SHA256SUMS.asc" + mode: '0644' + when: not bitcoind_binary_exists.stat.exists + + - name: Verify PGP signature on SHA256SUMS file + command: gpg --verify /tmp/bitcoin-knots-{{ bitcoin_knots_version_short }}-SHA256SUMS.asc /tmp/bitcoin-knots-{{ bitcoin_knots_version_short }}-SHA256SUMS + register: sha256sums_verification + changed_when: false + failed_when: false # Don't fail here - check for 'Good signature' in next task + when: not bitcoind_binary_exists.stat.exists + + + - name: Display SHA256SUMS verification result + debug: + msg: "{{ sha256sums_verification.stdout_lines + sha256sums_verification.stderr_lines }}" + when: not bitcoind_binary_exists.stat.exists + + - name: Fail if SHA256SUMS signature verification failed + fail: + msg: "SHA256SUMS signature verification failed. Aborting build." + when: not bitcoind_binary_exists.stat.exists and ('Good signature' not in sha256sums_verification.stdout and 'Good signature' not in sha256sums_verification.stderr) + + - name: Remove any existing tarball to force fresh download + file: + path: /tmp/bitcoin-{{ bitcoin_knots_version_short }}.tar.gz + state: absent + when: not bitcoind_binary_exists.stat.exists + + - name: Download Bitcoin Knots source tarball + get_url: + url: "{{ bitcoin_source_tarball_url }}" + dest: "/tmp/bitcoin-{{ bitcoin_knots_version_short }}.tar.gz" + mode: '0644' + validate_certs: yes + force: yes + when: not bitcoind_binary_exists.stat.exists + + - name: Calculate SHA256 checksum of downloaded tarball + command: sha256sum /tmp/bitcoin-{{ bitcoin_knots_version_short }}.tar.gz + register: tarball_checksum + changed_when: false + when: not bitcoind_binary_exists.stat.exists + + - name: Extract expected checksum from SHA256SUMS file + shell: grep "bitcoin-{{ bitcoin_knots_version_short }}.tar.gz" /tmp/bitcoin-knots-{{ bitcoin_knots_version_short }}-SHA256SUMS | awk '{print $1}' + register: expected_checksum + changed_when: false + when: not bitcoind_binary_exists.stat.exists + failed_when: expected_checksum.stdout == "" + + - name: Display checksum comparison + debug: + msg: + - "Expected: {{ expected_checksum.stdout | trim }}" + - "Actual: {{ tarball_checksum.stdout.split()[0] }}" + when: not bitcoind_binary_exists.stat.exists + + - name: Verify tarball checksum matches SHA256SUMS + fail: + msg: "Tarball checksum mismatch! Expected {{ expected_checksum.stdout | trim }}, got {{ tarball_checksum.stdout.split()[0] }}" + when: not bitcoind_binary_exists.stat.exists and expected_checksum.stdout | trim != tarball_checksum.stdout.split()[0] + + - name: Remove existing source directory if it exists (to force fresh extraction) + file: + path: "{{ bitcoin_knots_source_dir }}" + state: absent + when: not bitcoind_binary_exists.stat.exists + + - name: Remove extracted directory if it exists (from previous runs) + file: + path: "{{ bitcoin_knots_dir }}/bitcoin-{{ bitcoin_knots_version_short }}" + state: absent + when: not bitcoind_binary_exists.stat.exists + + - name: Extract verified source tarball + unarchive: + src: /tmp/bitcoin-{{ bitcoin_knots_version_short }}.tar.gz + dest: "{{ bitcoin_knots_dir }}" + remote_src: yes + when: not bitcoind_binary_exists.stat.exists + + - name: Check if extracted directory exists + stat: + path: "{{ bitcoin_knots_dir }}/bitcoin-{{ bitcoin_knots_version_short }}" + register: extracted_dir_stat + changed_when: false + when: not bitcoind_binary_exists.stat.exists + + - name: Rename extracted directory to expected name + command: mv "{{ bitcoin_knots_dir }}/bitcoin-{{ bitcoin_knots_version_short }}" "{{ bitcoin_knots_source_dir }}" + when: not bitcoind_binary_exists.stat.exists and extracted_dir_stat.stat.exists + + - name: Check if CMakeLists.txt exists + stat: + path: "{{ bitcoin_knots_source_dir }}/CMakeLists.txt" + register: cmake_exists + changed_when: false + when: not bitcoind_binary_exists.stat.exists + + - name: Create CMake build directory + file: + path: "{{ bitcoin_knots_source_dir }}/build" + state: directory + mode: '0755' + when: not bitcoind_binary_exists.stat.exists and cmake_exists.stat.exists | default(false) + + - name: Configure Bitcoin Knots build with CMake + command: > + cmake + -DCMAKE_INSTALL_PREFIX={{ bitcoin_build_prefix }} + -DBUILD_BITCOIN_WALLET=OFF + -DCMAKE_BUILD_TYPE=Release + -DWITH_ZMQ=ON + .. + args: + chdir: "{{ bitcoin_knots_source_dir }}/build" + when: not bitcoind_binary_exists.stat.exists and cmake_exists.stat.exists | default(false) + register: configure_result + changed_when: true + + - name: Verify CMake enabled ZMQ + shell: | + set -e + cd "{{ bitcoin_knots_source_dir }}/build" + cmake -LAH .. | grep -iE 'ZMQ|WITH_ZMQ|ENABLE_ZMQ|USE_ZMQ' + when: not bitcoind_binary_exists.stat.exists and cmake_exists.stat.exists | default(false) + register: zmq_check + changed_when: false + + - name: Fail if CMakeLists.txt not found + fail: + msg: "CMakeLists.txt not found in {{ bitcoin_knots_source_dir }}. Cannot build Bitcoin Knots." + when: not bitcoind_binary_exists.stat.exists and not (cmake_exists.stat.exists | default(false)) + + - name: Build Bitcoin Knots with CMake (this may take 30-60+ minutes) + command: cmake --build . -j{{ bitcoin_build_jobs }} + args: + chdir: "{{ bitcoin_knots_source_dir }}/build" + when: not bitcoind_binary_exists.stat.exists and cmake_exists.stat.exists | default(false) + async: 3600 + poll: 0 + register: build_result + changed_when: true + + - name: Check build status + async_status: + jid: "{{ build_result.ansible_job_id }}" + register: build_job_result + until: build_job_result.finished + retries: 120 + delay: 60 + when: not bitcoind_binary_exists.stat.exists and build_result.ansible_job_id is defined + + - name: Fail if build failed + fail: + msg: "Bitcoin Knots build failed: {{ build_job_result.msg }}" + when: not bitcoind_binary_exists.stat.exists and build_result.ansible_job_id is defined and build_job_result.failed | default(false) + + - name: Install Bitcoin Knots binaries + command: cmake --install . + args: + chdir: "{{ bitcoin_knots_source_dir }}/build" + when: not bitcoind_binary_exists.stat.exists and cmake_exists.stat.exists | default(false) + changed_when: true + + - name: Verify bitcoind binary exists + stat: + path: "{{ bitcoin_build_prefix }}/bin/bitcoind" + register: bitcoind_installed + changed_when: false + + - name: Verify bitcoin-cli binary exists + stat: + path: "{{ bitcoin_build_prefix }}/bin/bitcoin-cli" + register: bitcoin_cli_installed + changed_when: false + + - name: Fail if binaries not found + fail: + msg: "Bitcoin Knots binaries not found after installation" + when: not bitcoind_installed.stat.exists or not bitcoin_cli_installed.stat.exists + + - name: Create bitcoin.conf configuration file + copy: + dest: "{{ bitcoin_conf_dir }}/bitcoin.conf" + content: | + # Bitcoin Knots Configuration + # Generated by Ansible + + # Data directory (blockchain storage) + datadir={{ bitcoin_large_data_dir }} + + # RPC Configuration + server=1 + rpcuser={{ bitcoin_rpc_user }} + rpcpassword={{ bitcoin_rpc_password }} + rpcbind={{ bitcoin_rpc_bind }} + rpcport={{ bitcoin_rpc_port }} + rpcallowip=0.0.0.0/0 + + # Network Configuration + listen=1 + port={{ bitcoin_p2p_port }} + maxconnections={{ bitcoin_max_connections }} + + # Performance + dbcache={{ bitcoin_dbcache_mb }} + + # Transaction Index (optional) + {% if bitcoin_enable_txindex %} + txindex=1 + {% endif %} + + # Logging (to journald via systemd) + logtimestamps=1 + printtoconsole=1 + + # ZMQ Configuration + {% if bitcoin_zmq_enabled | default(false) %} + zmqpubrawblock={{ bitcoin_zmq_bind }}:{{ bitcoin_zmq_port_rawblock }} + zmqpubrawtx={{ bitcoin_zmq_bind }}:{{ bitcoin_zmq_port_rawtx }} + zmqpubhashblock={{ bitcoin_zmq_bind }}:{{ bitcoin_zmq_port_hashblock }} + zmqpubhashtx={{ bitcoin_zmq_bind }}:{{ bitcoin_zmq_port_hashtx }} + {% endif %} + + # Security + disablewallet=1 + owner: "{{ bitcoin_user }}" + group: "{{ bitcoin_group }}" + mode: '0640' + notify: Restart bitcoind + + - name: Create systemd service file for bitcoind + copy: + dest: /etc/systemd/system/bitcoind.service + content: | + [Unit] + Description=Bitcoin Knots daemon + After=network.target + + [Service] + Type=simple + User={{ bitcoin_user }} + Group={{ bitcoin_group }} + ExecStart={{ bitcoin_build_prefix }}/bin/bitcoind -conf={{ bitcoin_conf_dir }}/bitcoin.conf + Restart=always + RestartSec=10 + TimeoutStopSec=600 + StandardOutput=journal + StandardError=journal + + [Install] + WantedBy=multi-user.target + owner: root + group: root + mode: '0644' + notify: Restart bitcoind + + - name: Reload systemd daemon + systemd: + daemon_reload: yes + + - name: Enable and start bitcoind service + systemd: + name: bitcoind + enabled: yes + state: started + + - name: Wait for bitcoind RPC to be available + uri: + url: "http://{{ bitcoin_rpc_bind }}:{{ bitcoin_rpc_port }}" + method: POST + body_format: json + body: + jsonrpc: "1.0" + id: "healthcheck" + method: "getblockchaininfo" + params: [] + user: "{{ bitcoin_rpc_user }}" + password: "{{ bitcoin_rpc_password }}" + status_code: 200 + timeout: 10 + register: rpc_check + until: rpc_check.status == 200 + retries: 30 + delay: 5 + ignore_errors: yes + + - name: Display RPC connection status + debug: + msg: "Bitcoin Knots RPC is {{ 'available' if rpc_check.status == 200 else 'not yet available' }}" + + - name: Create Bitcoin Knots health check and push script + copy: + dest: /usr/local/bin/bitcoin-knots-healthcheck-push.sh + content: | + #!/bin/bash + # + # Bitcoin Knots Health Check and Push to Uptime Kuma + # Checks if bitcoind RPC is responding and pushes status to Uptime Kuma + # + + RPC_HOST="{{ bitcoin_rpc_bind }}" + RPC_PORT={{ bitcoin_rpc_port }} + RPC_USER="{{ bitcoin_rpc_user }}" + RPC_PASSWORD="{{ bitcoin_rpc_password }}" + UPTIME_KUMA_PUSH_URL="${UPTIME_KUMA_PUSH_URL}" + + # Check if bitcoind RPC is responding + check_bitcoind() { + local response + response=$(curl -s --max-time 30 \ + --user "${RPC_USER}:${RPC_PASSWORD}" \ + --data-binary '{"jsonrpc":"1.0","id":"healthcheck","method":"getblockchaininfo","params":[]}' \ + --header 'Content-Type: application/json' \ + "http://${RPC_HOST}:${RPC_PORT}" 2>&1) + + if [ $? -eq 0 ]; then + # Check if response contains a non-null error + # Successful responses have "error": null, failures have "error": {...} + if echo "$response" | grep -q '"error":null\|"error": null'; then + return 0 + else + return 1 + fi + else + return 1 + fi + } + + # Push status to Uptime Kuma + push_to_uptime_kuma() { + local status=$1 + local msg=$2 + + if [ -z "$UPTIME_KUMA_PUSH_URL" ]; then + echo "ERROR: UPTIME_KUMA_PUSH_URL not set" + return 1 + fi + + # URL encode spaces in message + local encoded_msg="${msg// /%20}" + + if ! curl -s --max-time 10 --retry 2 -o /dev/null \ + "${UPTIME_KUMA_PUSH_URL}?status=${status}&msg=${encoded_msg}&ping="; then + echo "ERROR: Failed to push to Uptime Kuma" + return 1 + fi + } + + # Main health check + if check_bitcoind; then + push_to_uptime_kuma "up" "OK" + exit 0 + else + push_to_uptime_kuma "down" "bitcoind RPC not responding" + exit 1 + fi + owner: root + group: root + mode: '0755' + + - name: Install curl for health check script + apt: + name: curl + state: present + + - name: Create systemd timer for Bitcoin Knots health check + copy: + dest: /etc/systemd/system/bitcoin-knots-healthcheck.timer + content: | + [Unit] + Description=Bitcoin Knots Health Check Timer + Requires=bitcoind.service + + [Timer] + OnBootSec=1min + OnUnitActiveSec=1min + Persistent=true + + [Install] + WantedBy=timers.target + owner: root + group: root + mode: '0644' + + - name: Create systemd service for Bitcoin Knots health check + copy: + dest: /etc/systemd/system/bitcoin-knots-healthcheck.service + content: | + [Unit] + Description=Bitcoin Knots Health Check and Push to Uptime Kuma + After=network.target bitcoind.service + + [Service] + Type=oneshot + User=root + ExecStart=/usr/local/bin/bitcoin-knots-healthcheck-push.sh + Environment=UPTIME_KUMA_PUSH_URL= + StandardOutput=journal + StandardError=journal + + [Install] + WantedBy=multi-user.target + owner: root + group: root + mode: '0644' + + - name: Reload systemd daemon for health check + systemd: + daemon_reload: yes + + - name: Enable and start Bitcoin Knots health check timer + systemd: + name: bitcoin-knots-healthcheck.timer + enabled: yes + state: started + + - name: Create Uptime Kuma push monitor setup script for Bitcoin Knots + delegate_to: localhost + become: no + copy: + dest: /tmp/setup_bitcoin_knots_monitor.py + content: | + #!/usr/bin/env python3 + import sys + import traceback + import yaml + from uptime_kuma_api import UptimeKumaApi, MonitorType + + try: + # Load configs + with open('/tmp/ansible_config.yml', 'r') as f: + config = yaml.safe_load(f) + + url = config['uptime_kuma_url'] + username = config['username'] + password = config['password'] + monitor_name = config['monitor_name'] + + # Connect to Uptime Kuma + api = UptimeKumaApi(url, timeout=30) + api.login(username, password) + + # Get all monitors + monitors = api.get_monitors() + + # Find or create "services" group + group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) + if not group: + group_result = api.add_monitor(type='group', name='services') + # Refresh to get the group with id + monitors = api.get_monitors() + group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) + + # Check if monitor already exists + existing_monitor = None + for monitor in monitors: + if monitor.get('name') == monitor_name: + existing_monitor = monitor + break + + # Get ntfy notification ID + notifications = api.get_notifications() + ntfy_notification_id = None + for notif in notifications: + if notif.get('type') == 'ntfy': + ntfy_notification_id = notif.get('id') + break + + if existing_monitor: + print(f"Monitor '{monitor_name}' already exists (ID: {existing_monitor['id']})") + push_token = existing_monitor.get('pushToken') or existing_monitor.get('push_token') + if not push_token: + raise ValueError("Could not find push token for monitor") + push_url = f"{url}/api/push/{push_token}" + print(f"Push URL: {push_url}") + else: + print(f"Creating push monitor '{monitor_name}'...") + api.add_monitor( + type=MonitorType.PUSH, + name=monitor_name, + parent=group['id'], + interval=60, + maxretries=3, + retryInterval=60, + notificationIDList={ntfy_notification_id: True} if ntfy_notification_id else {} + ) + monitors = api.get_monitors() + new_monitor = next((m for m in monitors if m.get('name') == monitor_name), None) + if new_monitor: + push_token = new_monitor.get('pushToken') or new_monitor.get('push_token') + if not push_token: + raise ValueError("Could not find push token for new monitor") + push_url = f"{url}/api/push/{push_token}" + print(f"Push URL: {push_url}") + + api.disconnect() + print("SUCCESS") + + except Exception as e: + error_msg = str(e) if str(e) else repr(e) + print(f"ERROR: {error_msg}", file=sys.stderr) + traceback.print_exc(file=sys.stderr) + sys.exit(1) + mode: '0755' + + - name: Create temporary config for monitor setup + delegate_to: localhost + become: no + copy: + dest: /tmp/ansible_config.yml + content: | + uptime_kuma_url: "{{ uptime_kuma_api_url }}" + username: "{{ uptime_kuma_username }}" + password: "{{ uptime_kuma_password }}" + monitor_name: "Bitcoin Knots" + mode: '0644' + + - name: Run Uptime Kuma push monitor setup + command: python3 /tmp/setup_bitcoin_knots_monitor.py + delegate_to: localhost + become: no + register: monitor_setup + changed_when: "'SUCCESS' in monitor_setup.stdout" + ignore_errors: yes + + - name: Extract push URL from monitor setup output + set_fact: + uptime_kuma_push_url: "{{ monitor_setup.stdout | regex_search('Push URL: (https?://[^\\s]+)', '\\1') | first | default('') }}" + delegate_to: localhost + become: no + when: monitor_setup.stdout is defined + + - name: Display extracted push URL + debug: + msg: "Uptime Kuma Push URL: {{ uptime_kuma_push_url }}" + when: uptime_kuma_push_url | default('') != '' + + - name: Set push URL in systemd service environment + lineinfile: + path: /etc/systemd/system/bitcoin-knots-healthcheck.service + regexp: '^Environment=UPTIME_KUMA_PUSH_URL=' + line: "Environment=UPTIME_KUMA_PUSH_URL={{ uptime_kuma_push_url }}" + state: present + insertafter: '^\[Service\]' + when: uptime_kuma_push_url | default('') != '' + + - name: Reload systemd daemon after push URL update + systemd: + daemon_reload: yes + when: uptime_kuma_push_url | default('') != '' + + - name: Restart health check timer to pick up new environment + systemd: + name: bitcoin-knots-healthcheck.timer + state: restarted + when: uptime_kuma_push_url | default('') != '' + + - name: Clean up temporary files + delegate_to: localhost + become: no + file: + path: "{{ item }}" + state: absent + loop: + - /tmp/setup_bitcoin_knots_monitor.py + - /tmp/ansible_config.yml + + handlers: + - name: Restart bitcoind + systemd: + name: bitcoind + state: restarted + + +- name: Setup public Bitcoin P2P forwarding on vipy via systemd-socket-proxyd + hosts: vipy + become: yes + vars_files: + - ../../infra_vars.yml + - ../../services_config.yml + - ../../infra_secrets.yml + - ./bitcoin_knots_vars.yml + vars: + bitcoin_tailscale_hostname: "knots-box" + uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}" + + tasks: + - name: Create Bitcoin P2P proxy socket unit + copy: + dest: /etc/systemd/system/bitcoin-p2p-proxy.socket + content: | + [Unit] + Description=Bitcoin P2P Proxy Socket + + [Socket] + ListenStream={{ bitcoin_p2p_port }} + + [Install] + WantedBy=sockets.target + owner: root + group: root + mode: '0644' + notify: Restart bitcoin-p2p-proxy socket + + - name: Create Bitcoin P2P proxy service unit + copy: + dest: /etc/systemd/system/bitcoin-p2p-proxy.service + content: | + [Unit] + Description=Bitcoin P2P Proxy to {{ bitcoin_tailscale_hostname }} + Requires=bitcoin-p2p-proxy.socket + After=network.target + + [Service] + Type=notify + ExecStart=/lib/systemd/systemd-socket-proxyd {{ bitcoin_tailscale_hostname }}:{{ bitcoin_p2p_port }} + owner: root + group: root + mode: '0644' + + - name: Reload systemd daemon + systemd: + daemon_reload: yes + + - name: Enable and start Bitcoin P2P proxy socket + systemd: + name: bitcoin-p2p-proxy.socket + enabled: yes + state: started + + - name: Allow Bitcoin P2P port through UFW + ufw: + rule: allow + port: "{{ bitcoin_p2p_port | string }}" + proto: tcp + comment: "Bitcoin P2P public access" + + - name: Verify connectivity to knots-box via Tailscale + wait_for: + host: "{{ bitcoin_tailscale_hostname }}" + port: "{{ bitcoin_p2p_port }}" + timeout: 10 + ignore_errors: yes + + - name: Display public endpoint + debug: + msg: "Bitcoin P2P public endpoint: {{ ansible_host }}:{{ bitcoin_p2p_port }}" + + # =========================================== + # Uptime Kuma TCP Monitor for Public P2P + # =========================================== + - name: Create Uptime Kuma TCP monitor setup script for Bitcoin P2P + delegate_to: localhost + become: no + copy: + dest: /tmp/setup_bitcoin_p2p_tcp_monitor.py + content: | + #!/usr/bin/env python3 + import sys + import traceback + import yaml + from uptime_kuma_api import UptimeKumaApi, MonitorType + + try: + with open('/tmp/ansible_bitcoin_p2p_config.yml', 'r') as f: + config = yaml.safe_load(f) + + url = config['uptime_kuma_url'] + username = config['username'] + password = config['password'] + monitor_host = config['monitor_host'] + monitor_port = config['monitor_port'] + monitor_name = config['monitor_name'] + + api = UptimeKumaApi(url, timeout=30) + api.login(username, password) + + monitors = api.get_monitors() + + # Find or create "services" group + group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) + if not group: + api.add_monitor(type='group', name='services') + monitors = api.get_monitors() + group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) + + # Check if monitor already exists + existing = next((m for m in monitors if m.get('name') == monitor_name), None) + + # Get ntfy notification ID + notifications = api.get_notifications() + ntfy_notification_id = None + for notif in notifications: + if notif.get('type') == 'ntfy': + ntfy_notification_id = notif.get('id') + break + + if existing: + print(f"Monitor '{monitor_name}' already exists (ID: {existing['id']})") + print("Skipping - monitor already configured") + else: + print(f"Creating TCP monitor '{monitor_name}'...") + api.add_monitor( + type=MonitorType.PORT, + name=monitor_name, + hostname=monitor_host, + port=monitor_port, + parent=group['id'], + interval=60, + maxretries=3, + retryInterval=60, + notificationIDList={ntfy_notification_id: True} if ntfy_notification_id else {} + ) + + api.disconnect() + print("SUCCESS") + + except Exception as e: + print(f"ERROR: {str(e)}", file=sys.stderr) + traceback.print_exc(file=sys.stderr) + sys.exit(1) + mode: '0755' + + - name: Create temporary config for TCP monitor setup + delegate_to: localhost + become: no + copy: + dest: /tmp/ansible_bitcoin_p2p_config.yml + content: | + uptime_kuma_url: "{{ uptime_kuma_api_url }}" + username: "{{ uptime_kuma_username }}" + password: "{{ uptime_kuma_password }}" + monitor_host: "{{ ansible_host }}" + monitor_port: {{ bitcoin_p2p_port }} + monitor_name: "Bitcoin Knots P2P Public" + mode: '0644' + + - name: Run Uptime Kuma TCP monitor setup + command: python3 /tmp/setup_bitcoin_p2p_tcp_monitor.py + delegate_to: localhost + become: no + register: tcp_monitor_setup + changed_when: "'SUCCESS' in tcp_monitor_setup.stdout" + ignore_errors: yes + + - name: Display TCP monitor setup output + debug: + msg: "{{ tcp_monitor_setup.stdout_lines }}" + when: tcp_monitor_setup.stdout is defined + + - name: Clean up TCP monitor temporary files + delegate_to: localhost + become: no + file: + path: "{{ item }}" + state: absent + loop: + - /tmp/setup_bitcoin_p2p_tcp_monitor.py + - /tmp/ansible_bitcoin_p2p_config.yml + + handlers: + - name: Restart bitcoin-p2p-proxy socket + systemd: + name: bitcoin-p2p-proxy.socket + state: restarted diff --git a/ansible/services/caddy_playbook.yml b/ansible/services/caddy_playbook.yml index 29e74b2..4935424 100644 --- a/ansible/services/caddy_playbook.yml +++ b/ansible/services/caddy_playbook.yml @@ -1,5 +1,5 @@ - name: Install and configure Caddy on Debian 12 - hosts: caddy + hosts: vps become: yes tasks: @@ -13,34 +13,17 @@ state: present update_cache: yes - # Caddy's signing subkey carries an expiry date, and Cloudsmith republishes - # the key with a refreshed binding signature before it lapses. Re-fetch on - # every run: pinning the keyring with `creates:` leaves the host stuck on a - # key that eventually expires and breaks `apt update` with an sqv error. - - name: Ensure apt keyrings directory exists - ansible.builtin.file: - path: /etc/apt/keyrings - state: directory - owner: root - group: root - mode: '0755' - - name: Download Caddy GPG armored key ansible.builtin.get_url: url: https://dl.cloudsmith.io/public/caddy/stable/gpg.key - dest: /etc/apt/keyrings/caddy-stable-archive-keyring.asc + dest: /tmp/caddy-stable-archive-keyring.asc mode: '0644' - register: caddy_key_download - - - name: Check for existing Caddy keyring - ansible.builtin.stat: - path: /usr/share/keyrings/caddy-stable-archive-keyring.gpg - register: caddy_keyring - name: Convert ASCII armored key to binary keyring ansible.builtin.command: - cmd: gpg --batch --yes --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg /etc/apt/keyrings/caddy-stable-archive-keyring.asc - when: caddy_key_download.changed or not caddy_keyring.stat.exists + cmd: gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg /tmp/caddy-stable-archive-keyring.asc + args: + creates: /usr/share/keyrings/caddy-stable-archive-keyring.gpg - name: Ensure permissions on keyring file ansible.builtin.file: diff --git a/ansible/services/datum-gateway/deploy_datum_gateway_playbook.yml b/ansible/services/datum-gateway/deploy_datum_gateway_playbook.yml deleted file mode 100644 index 2e88a27..0000000 --- a/ansible/services/datum-gateway/deploy_datum_gateway_playbook.yml +++ /dev/null @@ -1,58 +0,0 @@ ---- -# DATUM Gateway: solo/pooled mining gateway, built from source on knots-box. -# -# This is HALF OF A SYSTEM. The Bitcoin Knots node on the same host feeds it via -# `blocknotify=killall -USR1 datum_gateway` in bitcoin.conf (see -# roles/bitcoin_knots/README.md). Changing either config means thinking about -# both. Interrupting Stratum costs mining shares, so check for connected miners -# before restarting: -# ss -tn state established '( sport = :23334 )' -- name: Deploy DATUM Gateway on the bitcoin host - hosts: bitcoin - become: yes - vars: - # Preserves the push URL this check reports to. The role knows nothing about - # Uptime Kuma — this is just "a URL that accepts a ping". - healthcheck_push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/probe_datum-gateway/external" - healthcheck_push_token: "{{ gatus_push_tokens[inventory_hostname] }}" - roles: - - datum_gateway - -- name: Configure Caddy reverse proxy for the DATUM Gateway dashboard on the edge host - hosts: edge - become: yes - tasks: - - name: Publish the DATUM Gateway dashboard through Caddy - ansible.builtin.include_role: - name: caddy_site - vars: - caddy_site_name: datum-gateway - caddy_site_domain: "{{ subdomains.datum_gateway }}.{{ root_domain }}" - caddy_site_upstream: "{{ hostvars['knots_box_local'].ansible_host }}:{{ hostvars['knots_box_local'].datum_gateway_api_port }}" - caddy_site_resolvers: "100.100.100.100" - caddy_site_basic_auth: - - user: "{{ datum_dashboard_username }}" - hash: "{{ datum_dashboard_password_hash }}" - - # The role validates its own site fragment; this re-validates the whole - # assembled Caddyfile, which is the only thing that catches a conflict - # between two sites. - - name: Validate the assembled Caddyfile - ansible.builtin.command: caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile - changed_when: false - -- name: Setup public Stratum port forwarding on the edge host - hosts: edge - become: yes - tasks: - - name: Expose the DATUM Stratum port through a socket proxy - ansible.builtin.include_role: - name: socket_proxy - vars: - socket_proxy_name: datum-stratum - socket_proxy_description: "DATUM Stratum" - socket_proxy_listen_port: "{{ hostvars['knots_box_local'].datum_gateway_stratum_port }}" - socket_proxy_upstream_host: "{{ hostvars['knots_box_local'].ansible_host }}" - # Matches the UFW comment already on the edge host; the derived default - # would say "DATUM Stratum" and rewrite the rule. - socket_proxy_ufw_comment: "DATUM Gateway Stratum public access" diff --git a/ansible/services/forgejo-runner/SETUP.md b/ansible/services/forgejo-runner/SETUP.md deleted file mode 100644 index 38f6248..0000000 --- a/ansible/services/forgejo-runner/SETUP.md +++ /dev/null @@ -1,63 +0,0 @@ -# Forgejo Runner Setup - -## Obtaining the Registration Token - -1. Log in to the Forgejo instance at `https://forgejo.contrapeso.xyz` -2. Go to **Site Administration** > **Actions** > **Runners** -3. Click **Create new runner** -4. Copy the registration token - -## Configuring the Token - -Paste the token into `ansible/infra_secrets.yml`: - -```yaml -forgejo_runner_registration_token: "YOUR_TOKEN_HERE" -``` - -## Running the Playbook - -```bash -ansible-playbook ansible/services/forgejo-runner/deploy_forgejo_runner_playbook.yml -``` - -## Verifying - -1. On the VM: `systemctl status forgejo-runner` should show active -2. In Forgejo: **Site Administration** > **Actions** > **Runners** should show the runner as online -3. ~~In Uptime Kuma: the `forgejo-runner-healthcheck` push monitor should be receiving pings~~ - *(Uptime Kuma was decommissioned 2026-09-11 — this check no longer applies. The - healthcheck timer still runs on the host but pushes nowhere. See `archive/uptime_kuma/`.)* - -## Docker in CI jobs - -Jobs get their own Docker daemon (a `docker:dind` sidecar), not this host's. -Enabled by `forgejo_runner_dind_enabled` in the deploy playbook; the mechanism, -the trade-offs and how to turn it off are in `ansible/roles/forgejo_runner/README.md`. - -A workflow needs no special configuration — `DOCKER_HOST` is already set in -every job container: - -```yaml -jobs: - build: - runs-on: docker - steps: - - uses: actions/checkout@v7 - - run: docker build -t my-app . - - run: docker compose up -d --wait -``` - -Verifying after a deploy: - -1. `systemctl status forgejo-dind` — active, and `docker -H tcp://127.0.0.1:2376 version` - answers from the box. -2. `docker ps` on the host shows `forgejo-dind` and, during a run, **no** job - containers: they are inside dind now. `docker exec forgejo-dind docker ps` - shows those. -3. A workflow step running `docker version` reports the dind daemon. - -One caveat worth knowing before writing an end-to-end job: a `docker compose` -bind mount resolves on the dind filesystem, not in the job container, so a -compose file that mounts the checked-out repo will not see it. Named volumes -and `build:` contexts work normally. diff --git a/ansible/services/forgejo-runner/deploy_forgejo_runner_playbook.yml b/ansible/services/forgejo-runner/deploy_forgejo_runner_playbook.yml deleted file mode 100644 index 52c123d..0000000 --- a/ansible/services/forgejo-runner/deploy_forgejo_runner_playbook.yml +++ /dev/null @@ -1,34 +0,0 @@ ---- -- name: Install Forgejo Runner on Debian 13 - hosts: ci_runner - become: yes - vars: - # Preserves the push URL this host has been reporting to all along, so the - # move to a role changes no behaviour. The role itself knows nothing about - # Uptime Kuma — this is just "a URL that accepts a ping", and whatever - # replaces it sets the same variable. - healthcheck_push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/probe_forgejo-runner/external" - healthcheck_push_token: "{{ gatus_push_tokens[inventory_hostname] }}" - - # Give Actions jobs a Docker daemon of their own. - # - # Requested for counterweight/jar-lid-archive, which needs `docker build` - # and `docker compose` in CI. Enabled here rather than in the role defaults - # so that the default stays the safe one and this is a single, visible, - # revertible line. - # - # forgejo-runner-box is the right host for it: a dedicated CI VM on nodito - # that runs the runner and nothing else, which is what "not production" - # asks for — a privileged container is not something to put next to a - # service anyone depends on. - # - # Read before flipping this: the runner is SHARED. arbret, arbret-biz, - # arbretdos and arbret-ops all build here, and far more often than - # jar-lid-archive does. container.docker_host is a per-runner setting, not - # a per-repo one, so this moves every one of those repos' job containers - # off the host daemon and into dind. Nothing about their workflows changes, - # but their first run after this pulls its images again from an empty - # cache. See the role README for what to watch. - forgejo_runner_dind_enabled: true - roles: - - forgejo_runner diff --git a/ansible/services/forgejo/deploy_forgejo_playbook.yml b/ansible/services/forgejo/deploy_forgejo_playbook.yml index 29009ca..e17d08f 100644 --- a/ansible/services/forgejo/deploy_forgejo_playbook.yml +++ b/ansible/services/forgejo/deploy_forgejo_playbook.yml @@ -1,11 +1,16 @@ - name: Install Forgejo on Debian 12 with Caddy reverse proxy - hosts: edge + hosts: vipy become: yes vars_files: + - ../../infra_vars.yml + - ../../services_config.yml + - ../../infra_secrets.yml - ./forgejo_vars.yml vars: forgejo_subdomain: "{{ subdomains.forgejo }}" + caddy_sites_dir: "{{ caddy_sites_dir }}" forgejo_domain: "{{ forgejo_subdomain }}.{{ root_domain }}" + uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}" tasks: - name: Ensure required packages are installed @@ -66,10 +71,6 @@ owner: "{{ forgejo_user }}" group: "{{ forgejo_user }}" mode: '0644' - # Fresh install only: the web installer then writes the real config - # (INTERNAL_TOKEN, JWT secrets, DB path, ROOT_URL) into this file. - # Without force: false every re-run replaced it with this stub. - force: false - name: Download Forgejo systemd service file get_url: @@ -87,10 +88,138 @@ enabled: yes state: started - - name: Publish Forgejo through Caddy - ansible.builtin.include_role: - name: caddy_site - vars: - caddy_site_name: forgejo - caddy_site_domain: "{{ forgejo_domain }}" - caddy_site_upstream: "localhost:{{ forgejo_port }}" + - name: Ensure Caddy sites-enabled directory exists + file: + path: "{{ caddy_sites_dir }}" + state: directory + owner: root + group: root + mode: '0755' + + - name: Ensure Caddyfile includes import directive for sites-enabled + lineinfile: + path: /etc/caddy/Caddyfile + line: 'import sites-enabled/*' + insertafter: EOF + state: present + backup: yes + + - name: Create Caddy reverse proxy configuration for forgejo + copy: + dest: "{{ caddy_sites_dir }}/forgejo.conf" + content: | + {{ forgejo_domain }} { + reverse_proxy localhost:{{ forgejo_port }} + } + owner: root + group: root + mode: '0644' + + - name: Reload Caddy to apply new config + command: systemctl reload caddy + + - name: Create Uptime Kuma monitor setup script for Forgejo + delegate_to: localhost + become: no + copy: + dest: /tmp/setup_forgejo_monitor.py + content: | + #!/usr/bin/env python3 + import sys + import yaml + from uptime_kuma_api import UptimeKumaApi, MonitorType + + try: + with open('/tmp/ansible_config.yml', 'r') as f: + config = yaml.safe_load(f) + + url = config['uptime_kuma_url'] + username = config['username'] + password = config['password'] + monitor_url = config['monitor_url'] + monitor_name = config['monitor_name'] + + api = UptimeKumaApi(url, timeout=30) + api.login(username, password) + + # Get all monitors + monitors = api.get_monitors() + + # Find or create "services" group + group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) + if not group: + group_result = api.add_monitor(type='group', name='services') + # Refresh to get the group with id + monitors = api.get_monitors() + group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) + + # Check if monitor already exists + existing_monitor = None + for monitor in monitors: + if monitor.get('name') == monitor_name: + existing_monitor = monitor + break + + # Get ntfy notification ID + notifications = api.get_notifications() + ntfy_notification_id = None + for notif in notifications: + if notif.get('type') == 'ntfy': + ntfy_notification_id = notif.get('id') + break + + if existing_monitor: + print(f"Monitor '{monitor_name}' already exists (ID: {existing_monitor['id']})") + print("Skipping - monitor already configured") + else: + print(f"Creating monitor '{monitor_name}'...") + api.add_monitor( + type=MonitorType.HTTP, + name=monitor_name, + url=monitor_url, + parent=group['id'], + interval=60, + maxretries=3, + retryInterval=60, + notificationIDList={ntfy_notification_id: True} if ntfy_notification_id else {} + ) + + api.disconnect() + print("SUCCESS") + + except Exception as e: + print(f"ERROR: {str(e)}", file=sys.stderr) + sys.exit(1) + mode: '0755' + + - name: Create temporary config for monitor setup + delegate_to: localhost + become: no + copy: + dest: /tmp/ansible_config.yml + content: | + uptime_kuma_url: "{{ uptime_kuma_api_url }}" + username: "{{ uptime_kuma_username }}" + password: "{{ uptime_kuma_password }}" + monitor_url: "https://{{ forgejo_domain }}/api/healthz" + monitor_name: "Forgejo" + mode: '0644' + + - name: Run Uptime Kuma monitor setup + command: python3 /tmp/setup_forgejo_monitor.py + delegate_to: localhost + become: no + register: monitor_setup + changed_when: "'SUCCESS' in monitor_setup.stdout" + ignore_errors: yes + + - name: Clean up temporary files + delegate_to: localhost + become: no + file: + path: "{{ item }}" + state: absent + loop: + - /tmp/setup_forgejo_monitor.py + - /tmp/ansible_config.yml + diff --git a/ansible/services/forgejo/forgejo_vars.yml b/ansible/services/forgejo/forgejo_vars.yml index 7bba0ed..0bbb5a5 100644 --- a/ansible/services/forgejo/forgejo_vars.yml +++ b/ansible/services/forgejo/forgejo_vars.yml @@ -9,12 +9,15 @@ forgejo_url: "https://codeberg.org/forgejo/forgejo/releases/download/v{{ forgejo forgejo_bin_path: "/usr/local/bin/forgejo" forgejo_user: "git" -# (subdomain in group_vars/all/main.yml, caddy_sites_dir in roles/caddy_site/defaults/) +# (caddy_sites_dir and subdomain now in services_config.yml) # Remote access -remote_host_name: "{{ groups['edge'] | first }}" +remote_host_name: "vipy" remote_host: "{{ hostvars.get(remote_host_name, {}).get('ansible_host', remote_host_name) }}" remote_user: "{{ hostvars.get(remote_host_name, {}).get('ansible_user', 'counterweight') }}" remote_key_file: "{{ hostvars.get(remote_host_name, {}).get('ansible_ssh_private_key_file', '') }}" remote_port: "{{ hostvars.get(remote_host_name, {}).get('ansible_port', 22) }}" +# Local backup +local_backup_dir: "{{ lookup('env', 'HOME') }}/forgejo-backups" +backup_script_path: "{{ lookup('env', 'HOME') }}/.local/bin/forgejo_backup.sh" diff --git a/ansible/services/forgejo/setup_backup_forgejo.yml b/ansible/services/forgejo/setup_backup_forgejo.yml deleted file mode 100644 index 5329e2a..0000000 --- a/ansible/services/forgejo/setup_backup_forgejo.yml +++ /dev/null @@ -1,28 +0,0 @@ ---- -# Forgejo backup: dumps locally on vipy, encrypted with age. -# -# The biggest artefact in the estate (~2.7 G) and the reason retention here is -# short: 7 days locally would be 19 G of vipy's 36 G free. The box keeps 14. -# Forgejo is sqlite3 (DB_TYPE in app.ini), so it is stopped for the dump — the -# old job did the same. -- name: Configure the Forgejo backup on the edge host - hosts: edge - become: yes - vars_files: - - ./forgejo_vars.yml - - tasks: - - name: Ensure Forgejo dumps itself, encrypted, on a timer - ansible.builtin.include_role: - name: backup_source - vars: - backup_source_name: forgejo - backup_source_description: "Forgejo" - backup_source_dump_command: "tar -czf - -C / var/lib/forgejo etc/forgejo" - backup_source_stop_service: forgejo - backup_source_retention_days: 2 - backup_source_on_calendar: "*-*-* 02:30:00" - # Reported to Gatus as backup-dump_forgejo. The token is this HOST's token, - # shared with its other checks - see infra/400_host_monitoring.yml. - backup_source_push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/backup-dump_forgejo/external" - backup_source_push_token: "{{ gatus_push_tokens[inventory_hostname] }}" diff --git a/ansible/services/forgejo/setup_backup_forgejo_to_lapy.yml b/ansible/services/forgejo/setup_backup_forgejo_to_lapy.yml new file mode 100644 index 0000000..b90f0fb --- /dev/null +++ b/ansible/services/forgejo/setup_backup_forgejo_to_lapy.yml @@ -0,0 +1,122 @@ +--- +- name: Configure local backup for Forgejo from remote + hosts: lapy + gather_facts: no + vars_files: + - ../../infra_vars.yml + - ./forgejo_vars.yml + vars: + remote_data_path: "{{ forgejo_data_dir }}" + remote_config_path: "{{ forgejo_config_dir }}" + forgejo_service_name: "forgejo" + gpg_recipient: "{{ hostvars['localhost']['gpg_recipient'] | default('') }}" + gpg_key_id: "{{ hostvars['localhost']['gpg_key_id'] | default('') }}" + + tasks: + - name: Debug Forgejo backup vars + debug: + msg: + - "remote_host={{ remote_host }}" + - "remote_user={{ remote_user }}" + - "remote_data_path='{{ remote_data_path }}'" + - "remote_config_path='{{ remote_config_path }}'" + - "local_backup_dir={{ local_backup_dir }}" + - "gpg_recipient={{ gpg_recipient }}" + - "gpg_key_id={{ gpg_key_id }}" + + - name: Ensure local backup directory exists + ansible.builtin.file: + path: "{{ local_backup_dir }}" + state: directory + mode: "0755" + + - name: Ensure ~/.local/bin exists + ansible.builtin.file: + path: "{{ lookup('env', 'HOME') }}/.local/bin" + state: directory + mode: "0755" + + - name: Create Forgejo backup script + ansible.builtin.copy: + dest: "{{ backup_script_path }}" + mode: "0750" + content: | + #!/bin/bash + set -euo pipefail + + if [ -z "{{ gpg_recipient }}" ]; then + echo "GPG recipient is not configured. Aborting." + exit 1 + fi + + TIMESTAMP=$(date +'%Y-%m-%d') + ENCRYPTED_BACKUP="{{ local_backup_dir }}/forgejo-backup-$TIMESTAMP.tar.gz.gpg" + + {% if remote_key_file %} + SSH_CMD="ssh -i {{ remote_key_file }} -p {{ remote_port }}" + {% else %} + SSH_CMD="ssh -p {{ remote_port }}" + {% endif %} + + echo "Stopping Forgejo service..." + $SSH_CMD {{ remote_user }}@{{ remote_host }} "sudo systemctl stop {{ forgejo_service_name }}" + + echo "Creating encrypted backup archive..." + $SSH_CMD {{ remote_user }}@{{ remote_host }} "sudo tar -czf - {{ remote_data_path }} {{ remote_config_path }}" | \ + gpg --batch --yes --encrypt --recipient "{{ gpg_recipient }}" --output "$ENCRYPTED_BACKUP" + + echo "Starting Forgejo service..." + $SSH_CMD {{ remote_user }}@{{ remote_host }} "sudo systemctl start {{ forgejo_service_name }}" + + # Rotate old backups (keep 3 days) + # Calculate cutoff date (3 days ago) and delete backups older than that + CUTOFF_DATE=$(date -d '3 days ago' +'%Y-%m-%d') + for backup_file in "{{ local_backup_dir }}"/forgejo-backup-*.tar.gz.gpg; do + if [ -f "$backup_file" ]; then + # Extract date from filename: forgejo-backup-YYYY-MM-DD.tar.gz.gpg + file_date=$(basename "$backup_file" | sed -n 's/forgejo-backup-\([0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}\)\.tar\.gz\.gpg/\1/p') + if [ -n "$file_date" ] && [ "$file_date" != "$TIMESTAMP" ] && [ "$file_date" \< "$CUTOFF_DATE" ]; then + rm -f "$backup_file" + fi + fi + done + + echo "Backup completed successfully" + + - name: Ensure cronjob for Forgejo backup exists + ansible.builtin.cron: + name: "Forgejo backup" + user: "{{ lookup('env', 'USER') }}" + job: "{{ backup_script_path }}" + minute: 5 + hour: "9,12,15,18" + + - name: Run Forgejo backup script to create initial backup + ansible.builtin.command: "{{ backup_script_path }}" + + - name: Verify backup was created + block: + - name: Get today's date + command: date +'%Y-%m-%d' + register: today_date + changed_when: false + + - name: Check if backup file exists + stat: + path: "{{ local_backup_dir }}/forgejo-backup-{{ today_date.stdout }}.tar.gz.gpg" + register: backup_file_stat + + - name: Verify backup file exists + assert: + that: + - backup_file_stat.stat.exists + - backup_file_stat.stat.isreg + fail_msg: "Backup file {{ local_backup_dir }}/forgejo-backup-{{ today_date.stdout }}.tar.gz.gpg was not created" + success_msg: "Backup file {{ local_backup_dir }}/forgejo-backup-{{ today_date.stdout }}.tar.gz.gpg exists" + + - name: Verify backup file is not empty + assert: + that: + - backup_file_stat.stat.size > 0 + fail_msg: "Backup file {{ local_backup_dir }}/forgejo-backup-{{ today_date.stdout }}.tar.gz.gpg exists but is empty" + success_msg: "Backup file size is {{ backup_file_stat.stat.size }} bytes" diff --git a/ansible/services/fulcrum/deploy_fulcrum_playbook.yml b/ansible/services/fulcrum/deploy_fulcrum_playbook.yml index 1b7b963..1255cd6 100644 --- a/ansible/services/fulcrum/deploy_fulcrum_playbook.yml +++ b/ansible/services/fulcrum/deploy_fulcrum_playbook.yml @@ -1,27 +1,716 @@ ---- -# Fulcrum: Electrum server indexing the Bitcoin Knots node. -# The index takes days to rebuild, so nothing here touches its data directory -# beyond asserting that it exists. - name: Deploy Fulcrum Electrum Server - hosts: electrum + hosts: fulcrum_box_local become: yes + vars_files: + - ../../infra_vars.yml + - ../../services_config.yml + - ../../infra_secrets.yml + - ./fulcrum_vars.yml vars: - # Preserves the push URL this check has been configured with. The role knows - # nothing about Uptime Kuma — this is just "a URL that accepts a ping". - healthcheck_push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/probe_fulcrum/external" - healthcheck_push_token: "{{ gatus_push_tokens[inventory_hostname] }}" - roles: - - fulcrum + uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}" -- name: Setup public Fulcrum SSL forwarding on the edge host - hosts: edge - become: yes tasks: - - name: Expose Fulcrum SSL through a socket proxy - ansible.builtin.include_role: - name: socket_proxy - vars: - socket_proxy_name: fulcrum-ssl - socket_proxy_description: "Fulcrum SSL" - socket_proxy_listen_port: "{{ hostvars['fulcrum_box_local'].fulcrum_ssl_port }}" - socket_proxy_upstream_host: "{{ hostvars['fulcrum_box_local'].ansible_host }}" + - name: Calculate 75% of system RAM for db_mem + set_fact: + fulcrum_db_mem_mb: "{{ (ansible_memtotal_mb | float * fulcrum_db_mem_percent) | int }}" + changed_when: false + + - name: Display calculated db_mem value + debug: + msg: "Setting db_mem to {{ fulcrum_db_mem_mb }} MB ({{ (fulcrum_db_mem_percent * 100) | int }}% of {{ ansible_memtotal_mb }} MB total RAM)" + + - name: Display Fulcrum version to install + debug: + msg: "Installing Fulcrum version {{ fulcrum_version }}" + + - name: Install required packages + apt: + name: + - curl + - wget + - openssl + state: present + update_cache: yes + + - name: Create fulcrum group + group: + name: "{{ fulcrum_group }}" + system: yes + state: present + + - name: Create fulcrum user + user: + name: "{{ fulcrum_user }}" + group: "{{ fulcrum_group }}" + system: yes + shell: /usr/sbin/nologin + home: /home/{{ fulcrum_user }} + create_home: yes + state: present + + - name: Create Fulcrum database directory (heavy data on special mount) + file: + path: "{{ fulcrum_db_dir }}" + state: directory + owner: "{{ fulcrum_user }}" + group: "{{ fulcrum_group }}" + mode: '0755' + + - name: Create Fulcrum config directory + file: + path: "{{ fulcrum_config_dir }}" + state: directory + owner: root + group: "{{ fulcrum_group }}" + mode: '0755' + + - name: Create Fulcrum lib directory (for banner and other data files) + file: + path: "{{ fulcrum_lib_dir }}" + state: directory + owner: "{{ fulcrum_user }}" + group: "{{ fulcrum_group }}" + mode: '0755' + + # =========================================== + # SSL Certificate Generation + # =========================================== + - name: Check if SSL certificate already exists + stat: + path: "{{ fulcrum_ssl_cert_path }}" + register: fulcrum_ssl_cert_exists + when: fulcrum_ssl_enabled | default(false) + + - name: Generate self-signed SSL certificate for Fulcrum + command: > + openssl req -x509 -newkey rsa:4096 + -keyout {{ fulcrum_ssl_key_path }} + -out {{ fulcrum_ssl_cert_path }} + -sha256 -days {{ fulcrum_ssl_cert_days }} + -nodes + -subj "/C=XX/ST=Decentralized/L=Bitcoin/O=Fulcrum/OU=Electrum/CN=fulcrum.local" + args: + creates: "{{ fulcrum_ssl_cert_path }}" + when: fulcrum_ssl_enabled | default(false) + notify: Restart fulcrum + + - name: Set SSL certificate permissions + file: + path: "{{ fulcrum_ssl_cert_path }}" + owner: "{{ fulcrum_user }}" + group: "{{ fulcrum_group }}" + mode: '0644' + when: fulcrum_ssl_enabled | default(false) and fulcrum_ssl_cert_exists.stat.exists | default(false) or fulcrum_ssl_enabled | default(false) + + - name: Set SSL key permissions + file: + path: "{{ fulcrum_ssl_key_path }}" + owner: "{{ fulcrum_user }}" + group: "{{ fulcrum_group }}" + mode: '0600' + when: fulcrum_ssl_enabled | default(false) + + - name: Check if Fulcrum binary already exists + stat: + path: "{{ fulcrum_binary_path }}" + register: fulcrum_binary_exists + changed_when: false + + - name: Download Fulcrum binary tarball + get_url: + url: "https://github.com/cculianu/Fulcrum/releases/download/v{{ fulcrum_version }}/Fulcrum-{{ fulcrum_version }}-x86_64-linux.tar.gz" + dest: "/tmp/Fulcrum-{{ fulcrum_version }}-x86_64-linux.tar.gz" + mode: '0644' + when: not fulcrum_binary_exists.stat.exists + + - name: Extract Fulcrum binary + unarchive: + src: "/tmp/Fulcrum-{{ fulcrum_version }}-x86_64-linux.tar.gz" + dest: "/tmp" + remote_src: yes + when: not fulcrum_binary_exists.stat.exists + + - name: Install Fulcrum binary + copy: + src: "/tmp/Fulcrum-{{ fulcrum_version }}-x86_64-linux/Fulcrum" + dest: "{{ fulcrum_binary_path }}" + owner: root + group: root + mode: '0755' + remote_src: yes + when: not fulcrum_binary_exists.stat.exists + + - name: Verify Fulcrum binary installation + command: "{{ fulcrum_binary_path }} --version" + register: fulcrum_version_check + changed_when: false + + - name: Display Fulcrum version + debug: + msg: "{{ fulcrum_version_check.stdout_lines }}" + + - name: Create Fulcrum banner file + copy: + dest: "{{ fulcrum_lib_dir }}/fulcrum-banner.txt" + content: | + counterinfra + + PER ASPERA AD ASTRA + owner: "{{ fulcrum_user }}" + group: "{{ fulcrum_group }}" + mode: '0644' + + - name: Create Fulcrum configuration file + copy: + dest: "{{ fulcrum_config_dir }}/fulcrum.conf" + content: | + # Fulcrum Configuration + # Generated by Ansible + + # Bitcoin Core/Knots RPC settings + bitcoind = {{ bitcoin_rpc_host }}:{{ bitcoin_rpc_port }} + rpcuser = {{ bitcoin_rpc_user }} + rpcpassword = {{ bitcoin_rpc_password }} + + # Fulcrum server general settings + datadir = {{ fulcrum_db_dir }} + tcp = {{ fulcrum_tcp_bind }}:{{ fulcrum_tcp_port }} + peering = {{ 'true' if fulcrum_peering else 'false' }} + zmq_allow_hashtx = {{ 'true' if fulcrum_zmq_allow_hashtx else 'false' }} + + # SSL/TLS Configuration + {% if fulcrum_ssl_enabled | default(false) %} + ssl = {{ fulcrum_ssl_bind }}:{{ fulcrum_ssl_port }} + cert = {{ fulcrum_ssl_cert_path }} + key = {{ fulcrum_ssl_key_path }} + {% endif %} + + # Anonymize client IP addresses and TxIDs in logs + anon_logs = {{ 'true' if fulcrum_anon_logs else 'false' }} + + # Max RocksDB Memory in MiB + db_mem = {{ fulcrum_db_mem_mb }}.0 + + # Banner + banner = {{ fulcrum_lib_dir }}/fulcrum-banner.txt + owner: "{{ fulcrum_user }}" + group: "{{ fulcrum_group }}" + mode: '0640' + notify: Restart fulcrum + + - name: Create systemd service file for Fulcrum + copy: + dest: /etc/systemd/system/fulcrum.service + content: | + # MiniBolt: systemd unit for Fulcrum + # /etc/systemd/system/fulcrum.service + + [Unit] + Description=Fulcrum + After=network.target + + StartLimitBurst=2 + StartLimitIntervalSec=20 + + [Service] + ExecStart={{ fulcrum_binary_path }} {{ fulcrum_config_dir }}/fulcrum.conf + + User={{ fulcrum_user }} + Group={{ fulcrum_group }} + + # Process management + #################### + Type=simple + KillSignal=SIGINT + TimeoutStopSec=300 + + [Install] + WantedBy=multi-user.target + owner: root + group: root + mode: '0644' + notify: Restart fulcrum + + - name: Reload systemd daemon + systemd: + daemon_reload: yes + + - name: Enable and start Fulcrum service + systemd: + name: fulcrum + enabled: yes + state: started + + - name: Wait for Fulcrum to start + wait_for: + port: "{{ fulcrum_tcp_port }}" + host: "{{ fulcrum_tcp_bind }}" + delay: 5 + timeout: 30 + ignore_errors: yes + + - name: Check Fulcrum service status + systemd: + name: fulcrum + register: fulcrum_service_status + changed_when: false + + - name: Display Fulcrum service status + debug: + msg: "Fulcrum service is {{ 'running' if fulcrum_service_status.status.ActiveState == 'active' else 'not running' }}" + + - name: Create Fulcrum health check and push script + copy: + dest: /usr/local/bin/fulcrum-healthcheck-push.sh + content: | + #!/bin/bash + # + # Fulcrum Health Check and Push to Uptime Kuma + # Checks if Fulcrum TCP port is responding and pushes status to Uptime Kuma + # + + FULCRUM_HOST="{{ fulcrum_tcp_bind }}" + FULCRUM_PORT={{ fulcrum_tcp_port }} + UPTIME_KUMA_PUSH_URL="${UPTIME_KUMA_PUSH_URL}" + + # Check if Fulcrum TCP port is responding + check_fulcrum() { + # Try to connect to TCP port + timeout 5 bash -c "echo > /dev/tcp/${FULCRUM_HOST}/${FULCRUM_PORT}" 2>/dev/null + return $? + } + + # Push status to Uptime Kuma + push_to_uptime_kuma() { + local status=$1 + local msg=$2 + + if [ -z "$UPTIME_KUMA_PUSH_URL" ]; then + echo "ERROR: UPTIME_KUMA_PUSH_URL not set" + return 1 + fi + + # URL encode spaces in message + local encoded_msg="${msg// /%20}" + + if ! curl -s --max-time 10 --retry 2 -o /dev/null \ + "${UPTIME_KUMA_PUSH_URL}?status=${status}&msg=${encoded_msg}&ping="; then + echo "ERROR: Failed to push to Uptime Kuma" + return 1 + fi + } + + # Main health check + if check_fulcrum; then + push_to_uptime_kuma "up" "OK" + exit 0 + else + push_to_uptime_kuma "down" "Fulcrum TCP port not responding" + exit 1 + fi + owner: root + group: root + mode: '0755' + + - name: Create systemd timer for Fulcrum health check + copy: + dest: /etc/systemd/system/fulcrum-healthcheck.timer + content: | + [Unit] + Description=Fulcrum Health Check Timer + Requires=fulcrum.service + + [Timer] + OnBootSec=1min + OnUnitActiveSec=1min + Persistent=true + + [Install] + WantedBy=timers.target + owner: root + group: root + mode: '0644' + + - name: Create systemd service for Fulcrum health check + copy: + dest: /etc/systemd/system/fulcrum-healthcheck.service + content: | + [Unit] + Description=Fulcrum Health Check and Push to Uptime Kuma + After=network.target fulcrum.service + + [Service] + Type=oneshot + User=root + ExecStart=/usr/local/bin/fulcrum-healthcheck-push.sh + Environment=UPTIME_KUMA_PUSH_URL= + StandardOutput=journal + StandardError=journal + + [Install] + WantedBy=multi-user.target + owner: root + group: root + mode: '0644' + + - name: Reload systemd daemon for health check + systemd: + daemon_reload: yes + + - name: Enable and start Fulcrum health check timer + systemd: + name: fulcrum-healthcheck.timer + enabled: yes + state: started + + - name: Create Uptime Kuma push monitor setup script for Fulcrum + delegate_to: localhost + become: no + copy: + dest: /tmp/setup_fulcrum_monitor.py + content: | + #!/usr/bin/env python3 + import sys + import traceback + import yaml + from uptime_kuma_api import UptimeKumaApi, MonitorType + + try: + # Load configs + with open('/tmp/ansible_config.yml', 'r') as f: + config = yaml.safe_load(f) + + url = config['uptime_kuma_url'] + username = config['username'] + password = config['password'] + monitor_name = config['monitor_name'] + + # Connect to Uptime Kuma + api = UptimeKumaApi(url, timeout=30) + api.login(username, password) + + # Get all monitors + monitors = api.get_monitors() + + # Find or create "services" group + group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) + if not group: + group_result = api.add_monitor(type='group', name='services') + # Refresh to get the group with id + monitors = api.get_monitors() + group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) + + # Check if monitor already exists + existing_monitor = None + for monitor in monitors: + if monitor.get('name') == monitor_name: + existing_monitor = monitor + break + + # Get ntfy notification ID + notifications = api.get_notifications() + ntfy_notification_id = None + for notif in notifications: + if notif.get('type') == 'ntfy': + ntfy_notification_id = notif.get('id') + break + + if existing_monitor: + print(f"Monitor '{monitor_name}' already exists (ID: {existing_monitor['id']})") + push_token = existing_monitor.get('pushToken') or existing_monitor.get('push_token') + if not push_token: + raise ValueError("Could not find push token for monitor") + push_url = f"{url}/api/push/{push_token}" + print(f"Push URL: {push_url}") + else: + print(f"Creating push monitor '{monitor_name}'...") + api.add_monitor( + type=MonitorType.PUSH, + name=monitor_name, + parent=group['id'], + interval=60, + maxretries=3, + retryInterval=60, + notificationIDList={ntfy_notification_id: True} if ntfy_notification_id else {} + ) + monitors = api.get_monitors() + new_monitor = next((m for m in monitors if m.get('name') == monitor_name), None) + if new_monitor: + push_token = new_monitor.get('pushToken') or new_monitor.get('push_token') + if not push_token: + raise ValueError("Could not find push token for new monitor") + push_url = f"{url}/api/push/{push_token}" + print(f"Push URL: {push_url}") + + api.disconnect() + print("SUCCESS") + + except Exception as e: + error_msg = str(e) if str(e) else repr(e) + print(f"ERROR: {error_msg}", file=sys.stderr) + traceback.print_exc(file=sys.stderr) + sys.exit(1) + mode: '0755' + + - name: Create temporary config for monitor setup + delegate_to: localhost + become: no + copy: + dest: /tmp/ansible_config.yml + content: | + uptime_kuma_url: "{{ uptime_kuma_api_url }}" + username: "{{ uptime_kuma_username }}" + password: "{{ uptime_kuma_password }}" + monitor_name: "Fulcrum" + mode: '0644' + + - name: Run Uptime Kuma push monitor setup + command: python3 /tmp/setup_fulcrum_monitor.py + delegate_to: localhost + become: no + register: monitor_setup + changed_when: "'SUCCESS' in monitor_setup.stdout" + ignore_errors: yes + + - name: Extract push URL from monitor setup output + set_fact: + uptime_kuma_push_url: "{{ monitor_setup.stdout | regex_search('Push URL: (https?://[^\\s]+)', '\\1') | first | default('') }}" + delegate_to: localhost + become: no + when: monitor_setup.stdout is defined + + - name: Display extracted push URL + debug: + msg: "Uptime Kuma Push URL: {{ uptime_kuma_push_url }}" + when: uptime_kuma_push_url | default('') != '' + + - name: Set push URL in systemd service environment + lineinfile: + path: /etc/systemd/system/fulcrum-healthcheck.service + regexp: '^Environment=UPTIME_KUMA_PUSH_URL=' + line: "Environment=UPTIME_KUMA_PUSH_URL={{ uptime_kuma_push_url }}" + state: present + insertafter: '^\[Service\]' + when: uptime_kuma_push_url | default('') != '' + + - name: Reload systemd daemon after push URL update + systemd: + daemon_reload: yes + when: uptime_kuma_push_url | default('') != '' + + - name: Restart health check timer to pick up new environment + systemd: + name: fulcrum-healthcheck.timer + state: restarted + when: uptime_kuma_push_url | default('') != '' + + - name: Clean up temporary files + delegate_to: localhost + become: no + file: + path: "{{ item }}" + state: absent + loop: + - /tmp/setup_fulcrum_monitor.py + - /tmp/ansible_config.yml + - /tmp/Fulcrum-{{ fulcrum_version }}-x86_64-linux.tar.gz + - /tmp/Fulcrum-{{ fulcrum_version }}-x86_64-linux + + handlers: + - name: Restart fulcrum + systemd: + name: fulcrum + state: restarted + + +- name: Setup public Fulcrum SSL forwarding on vipy via systemd-socket-proxyd + hosts: vipy + become: yes + vars_files: + - ../../infra_vars.yml + - ../../services_config.yml + - ../../infra_secrets.yml + - ./fulcrum_vars.yml + vars: + uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}" + + tasks: + - name: Create Fulcrum SSL proxy socket unit + copy: + dest: /etc/systemd/system/fulcrum-ssl-proxy.socket + content: | + [Unit] + Description=Fulcrum SSL Proxy Socket + + [Socket] + ListenStream={{ fulcrum_ssl_port }} + + [Install] + WantedBy=sockets.target + owner: root + group: root + mode: '0644' + notify: Restart fulcrum-ssl-proxy socket + + - name: Create Fulcrum SSL proxy service unit + copy: + dest: /etc/systemd/system/fulcrum-ssl-proxy.service + content: | + [Unit] + Description=Fulcrum SSL Proxy to {{ fulcrum_tailscale_hostname }} + Requires=fulcrum-ssl-proxy.socket + After=network.target + + [Service] + Type=notify + ExecStart=/lib/systemd/systemd-socket-proxyd {{ fulcrum_tailscale_hostname }}:{{ fulcrum_ssl_port }} + owner: root + group: root + mode: '0644' + + - name: Reload systemd daemon + systemd: + daemon_reload: yes + + - name: Enable and start Fulcrum SSL proxy socket + systemd: + name: fulcrum-ssl-proxy.socket + enabled: yes + state: started + + - name: Allow Fulcrum SSL port through UFW + ufw: + rule: allow + port: "{{ fulcrum_ssl_port | string }}" + proto: tcp + comment: "Fulcrum SSL public access" + + - name: Verify connectivity to fulcrum-box via Tailscale + wait_for: + host: "{{ fulcrum_tailscale_hostname }}" + port: "{{ fulcrum_ssl_port }}" + timeout: 10 + ignore_errors: yes + + - name: Display public endpoint + debug: + msg: "Fulcrum SSL public endpoint: {{ ansible_host }}:{{ fulcrum_ssl_port }}" + + # =========================================== + # Uptime Kuma TCP Monitor for Public SSL Port + # =========================================== + - name: Create Uptime Kuma TCP monitor setup script for Fulcrum SSL + delegate_to: localhost + become: no + copy: + dest: /tmp/setup_fulcrum_ssl_tcp_monitor.py + content: | + #!/usr/bin/env python3 + import sys + import traceback + import yaml + from uptime_kuma_api import UptimeKumaApi, MonitorType + + try: + with open('/tmp/ansible_fulcrum_ssl_config.yml', 'r') as f: + config = yaml.safe_load(f) + + url = config['uptime_kuma_url'] + username = config['username'] + password = config['password'] + monitor_host = config['monitor_host'] + monitor_port = config['monitor_port'] + monitor_name = config['monitor_name'] + + api = UptimeKumaApi(url, timeout=30) + api.login(username, password) + + monitors = api.get_monitors() + + # Find or create "services" group + group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) + if not group: + api.add_monitor(type='group', name='services') + monitors = api.get_monitors() + group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) + + # Check if monitor already exists + existing = next((m for m in monitors if m.get('name') == monitor_name), None) + + # Get ntfy notification ID + notifications = api.get_notifications() + ntfy_notification_id = None + for notif in notifications: + if notif.get('type') == 'ntfy': + ntfy_notification_id = notif.get('id') + break + + if existing: + print(f"Monitor '{monitor_name}' already exists (ID: {existing['id']})") + print("Skipping - monitor already configured") + else: + print(f"Creating TCP monitor '{monitor_name}'...") + api.add_monitor( + type=MonitorType.PORT, + name=monitor_name, + hostname=monitor_host, + port=monitor_port, + parent=group['id'], + interval=60, + maxretries=3, + retryInterval=60, + notificationIDList={ntfy_notification_id: True} if ntfy_notification_id else {} + ) + + api.disconnect() + print("SUCCESS") + + except Exception as e: + print(f"ERROR: {str(e)}", file=sys.stderr) + traceback.print_exc(file=sys.stderr) + sys.exit(1) + mode: '0755' + + - name: Create temporary config for TCP monitor setup + delegate_to: localhost + become: no + copy: + dest: /tmp/ansible_fulcrum_ssl_config.yml + content: | + uptime_kuma_url: "{{ uptime_kuma_api_url }}" + username: "{{ uptime_kuma_username }}" + password: "{{ uptime_kuma_password }}" + monitor_host: "{{ ansible_host }}" + monitor_port: {{ fulcrum_ssl_port }} + monitor_name: "Fulcrum SSL Public" + mode: '0644' + + - name: Run Uptime Kuma TCP monitor setup + command: python3 /tmp/setup_fulcrum_ssl_tcp_monitor.py + delegate_to: localhost + become: no + register: tcp_monitor_setup + changed_when: "'SUCCESS' in tcp_monitor_setup.stdout" + ignore_errors: yes + + - name: Display TCP monitor setup output + debug: + msg: "{{ tcp_monitor_setup.stdout_lines }}" + when: tcp_monitor_setup.stdout is defined + + - name: Clean up TCP monitor temporary files + delegate_to: localhost + become: no + file: + path: "{{ item }}" + state: absent + loop: + - /tmp/setup_fulcrum_ssl_tcp_monitor.py + - /tmp/ansible_fulcrum_ssl_config.yml + + handlers: + - name: Restart fulcrum-ssl-proxy socket + systemd: + name: fulcrum-ssl-proxy.socket + state: restarted + diff --git a/ansible/services/fulcrum/fulcrum_vars.yml b/ansible/services/fulcrum/fulcrum_vars.yml new file mode 100644 index 0000000..8486b14 --- /dev/null +++ b/ansible/services/fulcrum/fulcrum_vars.yml @@ -0,0 +1,51 @@ +# Fulcrum Configuration Variables + +# Version - Pinned to specific release +fulcrum_version: "2.1.0" # Fulcrum version to install + +# Directories +fulcrum_db_dir: /mnt/fulcrum_data/fulcrum_db # Database directory (heavy data on special mount) +fulcrum_config_dir: /etc/fulcrum # Config file location (standard OS path) +fulcrum_lib_dir: /var/lib/fulcrum # Other data files (banner, etc.) on OS disk +fulcrum_binary_path: /usr/local/bin/Fulcrum + +# Network - Bitcoin RPC connection +# Bitcoin Knots is on a different host (knots_box_local) +# Using RPC user/password authentication (credentials from infra_secrets.yml) +bitcoin_rpc_host: "192.168.1.140" # Bitcoin Knots RPC host (IP of knots_box_local) +bitcoin_rpc_port: 8332 # Bitcoin Knots RPC port +# Note: bitcoin_rpc_user and bitcoin_rpc_password are loaded from infra_secrets.yml + +# Network - Fulcrum server +fulcrum_tcp_port: 50001 +fulcrum_ssl_port: 50002 +# Binding address for Fulcrum TCP/SSL server: +# - "127.0.0.1" = localhost only (use when Caddy is on the same box) +# - "0.0.0.0" = all interfaces (use when Caddy is on a different box) +# - Specific IP = bind to specific network interface +fulcrum_tcp_bind: "0.0.0.0" # Default: localhost (change to "0.0.0.0" if Caddy is on different box) +fulcrum_ssl_bind: "0.0.0.0" # Binding address for SSL port +# If Caddy is on a different box, set this to the IP address that Caddy will use to connect + +# SSL/TLS Configuration +fulcrum_ssl_enabled: true +fulcrum_ssl_cert_path: "{{ fulcrum_config_dir }}/fulcrum.crt" +fulcrum_ssl_key_path: "{{ fulcrum_config_dir }}/fulcrum.key" +fulcrum_ssl_cert_days: 3650 # 10 years validity for self-signed cert + +# Port forwarding configuration (for public access via VPS) +fulcrum_tailscale_hostname: "fulcrum-box" + +# Performance +# db_mem will be calculated as 75% of available RAM automatically in playbook +fulcrum_db_mem_percent: 0.75 # 75% of RAM for database cache + +# Configuration options +fulcrum_anon_logs: true # Anonymize client IPs and TxIDs in logs +fulcrum_peering: false # Disable peering with other Fulcrum servers +fulcrum_zmq_allow_hashtx: true # Allow ZMQ hashtx notifications + +# Service user +fulcrum_user: fulcrum +fulcrum_group: fulcrum + diff --git a/ansible/services/gatus/deploy_gatus_playbook.yml b/ansible/services/gatus/deploy_gatus_playbook.yml deleted file mode 100644 index 26b8c8e..0000000 --- a/ansible/services/gatus/deploy_gatus_playbook.yml +++ /dev/null @@ -1,122 +0,0 @@ ---- -# Gatus: health checks, status page and alerting for the whole estate. -# -# Built from source and run under systemd - upstream publishes no binaries, and -# their Dockerfile shows the runtime needs nothing but the static binary and a -# CA bundle. See roles/gatus/README.md. -- name: Deploy Gatus on the observability host - hosts: observability - become: yes - vars: - gatus_alerting: - signal: - # NOTE: the key is `api-url`, not `url` as upstream's own README table - # says - see alerting/provider/signal/signal.go. Gatus appends /v2/send - # itself if the suffix is missing. - # - # Reached by service name over the shared docker network. Gatus runs in - # a container, so 127.0.0.1 here would be the Gatus container, not the - # host - and the Signal API deliberately publishes no ports because it - # has no authentication. - api-url: "http://signal-api:8080" - number: "{{ signal_number }}" - recipients: "{{ signal_recipients }}" - default-alert: - # ── SIGNAL ALERTING IS ON ─────────────────────────────────────── - # Flip to false to silence the phone. Nothing else needs to - # change: this ONE key switches every Signal alert in the estate. - # (Re-enabled 2026-10-18 after relinking signal-api post-rebuild.) - # - # It works because `enabled` is the only alert field no registration - # playbook sets. Gatus merges a provider's default-alert into each - # endpoint's alert for every field the endpoint left unset, so this - # reaches all ~90 of them without editing a single endpoint - and - # without unregistering anything. The dashboard keeps checking and - # keeps showing red; it just stops sending. - # - # Deliberately NOT done by deleting the `signal:` provider block: - # that would strand ~90 endpoints referencing a provider that no - # longer exists, and lose the number and recipients. - enabled: true - # Overridden per group by the registration playbooks; these are the - # values that apply if a caller sets nothing. - failure-threshold: 1 - success-threshold: 2 - send-on-resolved: true - # An ongoing outage should not become an ongoing phone buzz. - minimum-reminder-interval: 6h - - roles: - - gatus - -# The dashboard is bound to loopback; Caddy publishes it. -# -# Auth is done HERE, at the edge, and not with Gatus's own `security.basic`. -# Gatus's security middleware protects exactly four routes (api/api.go): -# -# /api/v1/endpoints/statuses -# /api/v1/endpoints/:key/statuses -# /api/v1/suites/statuses -# /api/v1/suites/:key/statuses -# -# Everything else is registered on the UNPROTECTED router, including -# /api/v1/config, every badge, and - the part that matters - -# /api/v1/endpoints/:key/uptimes/:duration and .../response-times/:duration/history, -# which return real per-endpoint data to anyone who can guess a key. Keys are -# just "_". So Gatus's own auth makes the dashboard render empty -# while leaving the data readable, which is worse than it looks. -# -# The one route that must NOT sit behind basic auth is the external-endpoint -# push API. It authenticates with `Authorization: Bearer `, and basic -# auth wants `Authorization: Basic <...>` - same header, two schemes, and the -# push clients lose. It is not actually unauthenticated: the handler 401s on a -# missing prefix, an empty token, or a token that does not match that endpoint's -# own. Upstream's comment on the route says exactly that. -- name: Publish the Gatus status page through Caddy - hosts: observability - become: yes - tasks: - - name: Require the dashboard credentials to be set - ansible.builtin.assert: - that: - - gatus_dashboard_username is defined - - gatus_dashboard_username | length > 0 - - gatus_dashboard_password_hash is defined - - gatus_dashboard_password_hash.startswith('$2') - fail_msg: >- - gatus_dashboard_username and gatus_dashboard_password_hash must be in - the vault. Generate the hash on the observability host, which runs - Caddy natively, so the bcrypt cost and format match what verifies it: - caddy hash-password --plaintext 'your-password' - then: ansible-vault edit group_vars/all/vault.yml - - - name: Configure the Caddy vhost for Gatus - ansible.builtin.include_role: - name: caddy_site - vars: - caddy_site_name: gatus - caddy_site_domain: "{{ subdomains.gatus }}.{{ root_domain }}" - # caddy_site_body rather than caddy_site_upstream + caddy_site_basic_auth, - # because that pair applies auth to the whole site with no way to carve - # out the push path. `handle` blocks are mutually exclusive and first - # match wins, so the push API gets a route of its own. - caddy_site_body: | - @push { - path /api/v1/endpoints/*/external - method POST - } - - # Push API: Bearer-authenticated by Gatus itself. No basic auth here, - # or the Authorization header collides. - handle @push { - reverse_proxy 127.0.0.1:{{ gatus_port | default(8080) }} - } - - # Everything else: the dashboard, the config endpoint, the badges and - # the uptime/response-time history. - handle { - basic_auth { - {{ gatus_dashboard_username }} {{ gatus_dashboard_password_hash }} - } - reverse_proxy 127.0.0.1:{{ gatus_port | default(8080) }} - } diff --git a/ansible/services/headscale/deploy_headscale_playbook.yml b/ansible/services/headscale/deploy_headscale_playbook.yml index 29de9b6..1bcf5bf 100644 --- a/ansible/services/headscale/deploy_headscale_playbook.yml +++ b/ansible/services/headscale/deploy_headscale_playbook.yml @@ -1,12 +1,18 @@ - name: Deploy headscale and configure Caddy reverse proxy - hosts: vpn_control + hosts: spacey become: no vars_files: + - ../../infra_vars.yml + - ../../services_config.yml + - ../../infra_secrets.yml - ./headscale_vars.yml vars: headscale_subdomain: "{{ subdomains.headscale }}" + caddy_sites_dir: "{{ caddy_sites_dir }}" headscale_domain: "{{ headscale_subdomain }}.{{ root_domain }}" headscale_base_domain: "tailnet.{{ root_domain }}" + headscale_namespace: "{{ service_settings.headscale.namespace }}" + uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}" tasks: - name: Install required packages @@ -79,24 +85,15 @@ groups: headscale append: yes - # This used to write `{}` - an empty policy - over the real ACL, which was - # maintained by hand on the box. It had already flattened it once - # (/etc/headscale/acl.json.937464.2026-09-12@14:03:08~ is Ansible's own - # backup of the 1596-byte policy it replaced), and every subsequent - # site.yml run would have taken the whole tailnet's access rules with it. - # - # The policy now lives in templates/acl.json.j2, captured from the live - # file on 2026-09-19. validate first: a malformed policy is one headscale - # refuses to load, which is the same outage as an empty one. - - name: Deploy the Headscale ACL policy + - name: Create ACL policies file become: yes - ansible.builtin.template: - src: acl.json.j2 + copy: dest: /etc/headscale/acl.json + content: | + {} owner: headscale group: headscale mode: '0640' - validate: "python3 -m json.tool %s" notify: Restart headscale - name: Deploy headscale configuration file @@ -233,20 +230,144 @@ port: '3478' proto: udp - - name: Publish headscale through Caddy - ansible.builtin.include_role: - name: caddy_site - # This play is become: no and elevates per task. `apply` is how an - # include_role passes become down to the role's tasks - `become:` on - # the include itself is rejected. The role's handler sets its own. - apply: - become: yes - vars: - caddy_site_name: headscale - caddy_site_domain: "{{ headscale_domain }}" - # The Headscale API authenticates every request itself (Bearer API - # key), so a plain proxy is all it needs. - caddy_site_upstream: "localhost:{{ headscale_port }}" + - name: Ensure Caddy sites-enabled directory exists + become: yes + file: + path: "{{ caddy_sites_dir }}" + state: directory + owner: root + group: root + mode: '0755' + + - name: Ensure Caddyfile includes import directive for sites-enabled + become: yes + lineinfile: + path: /etc/caddy/Caddyfile + line: 'import sites-enabled/*' + insertafter: EOF + state: present + backup: yes + + - name: Create Caddy reverse proxy configuration for headscale + become: yes + copy: + dest: "{{ caddy_sites_dir }}/headscale.conf" + content: | + {{ headscale_domain }} { + reverse_proxy localhost:{{ headscale_port }} + } + owner: root + group: root + mode: '0644' + + - name: Reload Caddy to apply new config + become: yes + command: systemctl reload caddy + + - name: Create Uptime Kuma monitor setup script for Headscale + delegate_to: localhost + become: no + copy: + dest: /tmp/setup_headscale_monitor.py + content: | + #!/usr/bin/env python3 + import sys + import yaml + from uptime_kuma_api import UptimeKumaApi, MonitorType + + try: + with open('/tmp/ansible_config.yml', 'r') as f: + config = yaml.safe_load(f) + + url = config['uptime_kuma_url'] + username = config['username'] + password = config['password'] + monitor_url = config['monitor_url'] + monitor_name = config['monitor_name'] + + api = UptimeKumaApi(url, timeout=30) + api.login(username, password) + + # Get all monitors + monitors = api.get_monitors() + + # Find or create "services" group + group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) + if not group: + group_result = api.add_monitor(type='group', name='services') + # Refresh to get the group with id + monitors = api.get_monitors() + group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) + + # Check if monitor already exists + existing_monitor = None + for monitor in monitors: + if monitor.get('name') == monitor_name: + existing_monitor = monitor + break + + # Get ntfy notification ID + notifications = api.get_notifications() + ntfy_notification_id = None + for notif in notifications: + if notif.get('type') == 'ntfy': + ntfy_notification_id = notif.get('id') + break + + if existing_monitor: + print(f"Monitor '{monitor_name}' already exists (ID: {existing_monitor['id']})") + print("Skipping - monitor already configured") + else: + print(f"Creating monitor '{monitor_name}'...") + api.add_monitor( + type=MonitorType.HTTP, + name=monitor_name, + url=monitor_url, + parent=group['id'], + interval=60, + maxretries=3, + retryInterval=60, + notificationIDList={ntfy_notification_id: True} if ntfy_notification_id else {} + ) + + api.disconnect() + print("SUCCESS") + + except Exception as e: + print(f"ERROR: {str(e)}", file=sys.stderr) + sys.exit(1) + mode: '0755' + + - name: Create temporary config for monitor setup + delegate_to: localhost + become: no + copy: + dest: /tmp/ansible_config.yml + content: | + uptime_kuma_url: "{{ uptime_kuma_api_url }}" + username: "{{ uptime_kuma_username }}" + password: "{{ uptime_kuma_password }}" + monitor_url: "https://{{ headscale_domain }}/health" + monitor_name: "Headscale" + mode: '0644' + + - name: Run Uptime Kuma monitor setup + command: python3 /tmp/setup_headscale_monitor.py + delegate_to: localhost + become: no + register: monitor_setup + changed_when: "'SUCCESS' in monitor_setup.stdout" + ignore_errors: yes + + - name: Clean up temporary files + delegate_to: localhost + become: no + file: + path: "{{ item }}" + state: absent + loop: + - /tmp/setup_headscale_monitor.py + - /tmp/ansible_config.yml handlers: - name: Restart headscale diff --git a/ansible/services/headscale/headscale_vars.yml b/ansible/services/headscale/headscale_vars.yml index 7118cc7..653c175 100644 --- a/ansible/services/headscale/headscale_vars.yml +++ b/ansible/services/headscale/headscale_vars.yml @@ -1,5 +1,5 @@ # Headscale service configuration -# (subdomain in group_vars/all/main.yml, caddy_sites_dir in roles/caddy_site/defaults/) +# (subdomain and caddy_sites_dir now in services_config.yml) headscale_port: 8080 headscale_grpc_port: 50443 @@ -10,11 +10,15 @@ headscale_version: "0.26.1" # Data directory headscale_data_dir: /var/lib/headscale -# Namespace is headscale_namespace in group_vars/all/main.yml +# Namespace now configured in services_config.yml under service_settings.headscale.namespace # Remote access -remote_host_name: "{{ groups['vpn_control'] | first }}" +remote_host_name: "spacey" remote_host: "{{ hostvars.get(remote_host_name, {}).get('ansible_host', remote_host_name) }}" remote_user: "{{ hostvars.get(remote_host_name, {}).get('ansible_user', 'counterweight') }}" remote_key_file: "{{ hostvars.get(remote_host_name, {}).get('ansible_ssh_private_key_file', '') }}" remote_port: "{{ hostvars.get(remote_host_name, {}).get('ansible_port', 22) }}" + +# Local backup +local_backup_dir: "{{ lookup('env', 'HOME') }}/headscale-backups" +backup_script_path: "{{ lookup('env', 'HOME') }}/.local/bin/headscale_backup.sh" diff --git a/ansible/services/headscale/setup_backup_headscale.yml b/ansible/services/headscale/setup_backup_headscale.yml deleted file mode 100644 index 15ba5b7..0000000 --- a/ansible/services/headscale/setup_backup_headscale.yml +++ /dev/null @@ -1,24 +0,0 @@ ---- -- name: Configure the Headscale backup on the vpn_control host - hosts: vpn_control - become: yes - vars_files: - - ./headscale_vars.yml - - tasks: - - name: Ensure Headscale dumps itself, encrypted, on a timer - ansible.builtin.include_role: - name: backup_source - vars: - backup_source_name: headscale - backup_source_description: "Headscale" - # -C / with relative paths: avoids tar's "removing leading /" and makes - # the restore target explicit. - backup_source_dump_command: "tar -czf - -C / var/lib/headscale etc/headscale" - backup_source_stop_service: headscale - backup_source_retention_days: 7 - backup_source_on_calendar: "*-*-* 02:00:00" - # Reported to Gatus as backup-dump_headscale. The token is this HOST's token, - # shared with its other checks - see infra/400_host_monitoring.yml. - backup_source_push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/backup-dump_headscale/external" - backup_source_push_token: "{{ gatus_push_tokens[inventory_hostname] }}" diff --git a/ansible/services/headscale/setup_backup_headscale_to_lapy.yml b/ansible/services/headscale/setup_backup_headscale_to_lapy.yml new file mode 100644 index 0000000..5f9a764 --- /dev/null +++ b/ansible/services/headscale/setup_backup_headscale_to_lapy.yml @@ -0,0 +1,75 @@ +- name: Configure local backup for Headscale from remote + hosts: lapy + gather_facts: no + vars_files: + - ../../infra_vars.yml + - ./headscale_vars.yml + vars: + remote_data_path: "{{ headscale_data_dir }}" + remote_config_path: "/etc/headscale" + + tasks: + - name: Debug remote backup vars + debug: + msg: + - "remote_host={{ remote_host }}" + - "remote_user={{ remote_user }}" + - "remote_data_path='{{ remote_data_path }}'" + - "remote_config_path='{{ remote_config_path }}'" + - "local_backup_dir={{ local_backup_dir }}" + + - name: Ensure local backup directory exists + file: + path: "{{ local_backup_dir }}" + state: directory + mode: '0755' + + - name: Ensure ~/.local/bin exists + file: + path: "{{ lookup('env', 'HOME') }}/.local/bin" + state: directory + mode: '0755' + + - name: Create backup script + copy: + dest: "{{ backup_script_path }}" + mode: '0750' + content: | + #!/bin/bash + set -euo pipefail + + TIMESTAMP=$(date +'%Y-%m-%d') + BACKUP_DIR="{{ local_backup_dir }}/$TIMESTAMP" + mkdir -p "$BACKUP_DIR" + + {% if remote_key_file %} + SSH_CMD="ssh -i {{ remote_key_file }} -p {{ remote_port }}" + {% else %} + SSH_CMD="ssh -p {{ remote_port }}" + {% endif %} + + # Stop headscale service for consistent backup + $SSH_CMD {{ remote_user }}@{{ remote_host }} "sudo systemctl stop headscale" + + # Backup data directory + rsync -az -e "$SSH_CMD" --delete {{ remote_user }}@{{ remote_host }}:{{ remote_data_path }}/ "$BACKUP_DIR/data/" + + # Backup config directory + rsync -az -e "$SSH_CMD" --delete {{ remote_user }}@{{ remote_host }}:{{ remote_config_path }}/ "$BACKUP_DIR/config/" + + # Start headscale service again + $SSH_CMD {{ remote_user }}@{{ remote_host }} "sudo systemctl start headscale" + + # Rotate old backups (keep 14 days) + find "{{ local_backup_dir }}" -maxdepth 1 -type d -name '20*' -mtime +13 -exec rm -rf {} \; + + - name: Ensure cronjob for backup exists + cron: + name: "Headscale backup" + user: "{{ lookup('env', 'USER') }}" + job: "{{ backup_script_path }}" + minute: 5 + hour: "9,12,15,18" + + - name: Run the backup script to make the first backup + command: "{{ backup_script_path }}" diff --git a/ansible/services/headscale/templates/acl.json.j2 b/ansible/services/headscale/templates/acl.json.j2 deleted file mode 100644 index 710c7a8..0000000 --- a/ansible/services/headscale/templates/acl.json.j2 +++ /dev/null @@ -1,128 +0,0 @@ -{# - Headscale ACL policy. - - CAPTURED FROM THE LIVE FILE on spacey, 2026-09-19. Until then this playbook - wrote `{}` here - an empty policy - while the real rules were maintained by - hand on the box. Ansible had already flattened them once: - /etc/headscale/acl.json.937464.2026-09-12@14:03:08~ is its own backup of a - 1596-byte policy it replaced with {}. The next full site.yml run would have - done it again, taking the whole tailnet's access rules with it. - - So this is the source of truth now. It is a template and not inline `content:` - (the style used elsewhere in this playbook) precisely because it needs to be - diffable in review - an empty policy is the kind of change that must never - again slip through as three invisible characters. - - ADDING A HOST: it needs a tagOwners entry, and an acls rule naming the exact - ports its peers may reach. A node with no rule is INVISIBLE to the others - - it will not appear in `tailscale status` and MagicDNS will not resolve it, - which surfaces as a Caddy 502 with "no such host" rather than as a refused - connection. -#} -{ - "acls": [ - { - "action": "accept", - "dst": [ - "*:22", - "tag:arbret-staging-box:4117", - "tag:prd-vipy:4117", - "tag:nodito:8006" - ], - "src": [ - "tag:framie" - ] - }, - { - "action": "accept", - "dst": [ - "tag:knots-box:8333", - "tag:knots-box:7152", - "tag:knots-box:23334", - "tag:fulcrum-box:50002", - "tag:mempool-box:8080", - "tag:memos-box:5230", - "tag:music-box:4533", - "tag:jar-lid-archive-staging-box:8000", - "tag:monerod-box:18089", - "tag:monerod-box:8081", - "tag:arbret-staging-box:80" - ], - "src": [ - "tag:prd-vipy" - ] - }, - { - "action": "accept", - "dst": [ - "tag:knots-box:8332", - "tag:knots-box:28332-28335" - ], - "src": [ - "tag:fulcrum-box" - ] - }, - { - "action": "accept", - "dst": [ - "tag:knots-box:8332", - "tag:fulcrum-box:50001" - ], - "src": [ - "tag:mempool-box" - ] - }, - { - "action": "accept", - "dst": [ - "tag:prd-vipy:22", - "tag:memos-box:22", - "tag:jar-lid-archive-staging-box:22" - ], - "src": [ - "tag:small-backups-box" - ] - } - ], - "tagOwners": { - "tag:arbret-staging-box": [ - "{{ headscale_namespace }}@" - ], - "tag:forgejo-runner-box": [ - "{{ headscale_namespace }}@" - ], - "tag:framie": [ - "{{ headscale_namespace }}@" - ], - "tag:fulcrum-box": [ - "{{ headscale_namespace }}@" - ], - "tag:jar-lid-archive-staging-box": [ - "{{ headscale_namespace }}@" - ], - "tag:knots-box": [ - "{{ headscale_namespace }}@" - ], - "tag:memos-box": [ - "{{ headscale_namespace }}@" - ], - "tag:mempool-box": [ - "{{ headscale_namespace }}@" - ], - "tag:monerod-box": [ - "{{ headscale_namespace }}@" - ], - "tag:music-box": [ - "{{ headscale_namespace }}@" - ], - "tag:nodito": [ - "{{ headscale_namespace }}@" - ], - "tag:prd-vipy": [ - "{{ headscale_namespace }}@" - ], - "tag:small-backups-box": [ - "{{ headscale_namespace }}@" - ] - } -} diff --git a/ansible/services/jar-lid-archive/deploy_jar_lid_archive_playbook.yml b/ansible/services/jar-lid-archive/deploy_jar_lid_archive_playbook.yml deleted file mode 100644 index 4e9a9d6..0000000 --- a/ansible/services/jar-lid-archive/deploy_jar_lid_archive_playbook.yml +++ /dev/null @@ -1,51 +0,0 @@ ---- -# Jar Lid Archive on its own box, published from the edge host. -# -# See roles/jar_lid_archive/README.md. Per-host values (domain, port, seed users, -# secrets) are in host_vars//, so a production host is a new inventory -# entry in [jar_lid_archive], not a new playbook. -- name: Deploy Jar Lid Archive - hosts: jar_lid_archive - become: yes - roles: - - jar_lid_archive - # "Does the app answer", as opposed to "are the containers up". /health/ - # runs SELECT 1 and touches the bucket, and returns 503 if either fails. - - role: healthcheck - vars: - healthcheck_name: jar-lid-archive - healthcheck_description: "Jar Lid Archive /health/ (app, database and photo store)" - healthcheck_command: "curl -fsS --max-time 10 -o /dev/null http://127.0.0.1:{{ jar_lid_archive_port }}/health/" - healthcheck_push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/probe_{{ jar_lid_archive_instance }}/external" - healthcheck_push_token: "{{ gatus_push_tokens[inventory_hostname] }}" - -- name: Publish Jar Lid Archive from the edge host - hosts: edge - become: yes - tasks: - # No basic auth: the one-off links (/links/...) are opened by people who have - # no account and no proxy credential. The app's own login guards the rest. - # - # Caddy sets X-Forwarded-Proto: https and overwrites any value the client - # sent, which is what lets the app trust it (DJANGO_BEHIND_TLS_PROXY). - # - # No request size limit is set, deliberately: Caddy has none by default, and - # phones upload multi-megabyte photos. - - name: Expose Jar Lid Archive over HTTPS - ansible.builtin.include_role: - name: caddy_site - vars: - caddy_site_name: "{{ hostvars[jla_host].jar_lid_archive_instance }}" - caddy_site_domain: "{{ hostvars[jla_host].jar_lid_archive_domain }}" - # Tailscale MagicDNS: the box has no public address. - caddy_site_upstream: "{{ hostvars[jla_host].ansible_host }}:{{ hostvars[jla_host].jar_lid_archive_port }}" - caddy_site_resolvers: "100.100.100.100" - loop: "{{ groups['jar_lid_archive'] }}" - # Not `item`: the role's own tasks loop, and would shadow it. - loop_control: - loop_var: jla_host - - # The per-site validate cannot catch a conflict between sites. - - name: Validate the whole Caddyfile - ansible.builtin.command: caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile - changed_when: false diff --git a/ansible/services/jar-lid-archive/setup_backup_jar_lid_archive.yml b/ansible/services/jar-lid-archive/setup_backup_jar_lid_archive.yml deleted file mode 100644 index d0a0147..0000000 --- a/ansible/services/jar-lid-archive/setup_backup_jar_lid_archive.yml +++ /dev/null @@ -1,34 +0,0 @@ ---- -# Jar Lid Archive backup: the database and the photo store, as ONE artefact, -# dumped and age-encrypted on the box itself. small-backups-box pulls it. -# -# Only `web` is stopped. It is the only writer to both the database and the -# photo store, so with it down both are quiescent and are captured at the same -# instant - no row points at a photo the tar lacks. db stays up for pg_dump. -# -# storage is deliberately NOT stopped, though an earlier version did: on this -# VM's contended HDD-backed disk SeaweedFS can take minutes to come back, and the -# first test run left the app down for 26 minutes waiting on it. -- name: Configure the Jar Lid Archive backup on its own host - hosts: jar_lid_archive - become: yes - - tasks: - - name: Ensure Jar Lid Archive dumps itself, encrypted, on a timer - ansible.builtin.include_role: - name: backup_source - vars: - backup_source_name: "{{ jar_lid_archive_instance }}" - backup_source_description: "Jar Lid Archive ({{ jar_lid_archive_instance }})" - backup_source_dump_command: /usr/local/bin/jar-lid-archive-dump.sh - # Uncompressed: the bulk is JPEGs. See roles/jar_lid_archive/templates/dump.sh.j2. - backup_source_artifact_suffix: "tar.age" - backup_source_stop_command: "docker compose -f /opt/jar-lid-archive/docker-compose.yml stop web" - # `up -d --wait`, not `start`: it also repairs a db or storage that died - # meanwhile, and only returns once /health/ passes. - backup_source_start_command: "docker compose -f /opt/jar-lid-archive/docker-compose.yml up -d --wait --wait-timeout 300" - backup_source_retention_days: "{{ jar_lid_archive_backup_retention_days }}" - backup_source_on_calendar: "*-*-* 02:20:00" - # Reported to Gatus as backup-dump_. - backup_source_push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/backup-dump_{{ jar_lid_archive_instance }}/external" - backup_source_push_token: "{{ gatus_push_tokens[inventory_hostname] }}" diff --git a/ansible/services/lnbits/deploy_lnbits_playbook.yml b/ansible/services/lnbits/deploy_lnbits_playbook.yml index d0e1a77..e5d546b 100644 --- a/ansible/services/lnbits/deploy_lnbits_playbook.yml +++ b/ansible/services/lnbits/deploy_lnbits_playbook.yml @@ -1,11 +1,16 @@ - name: Deploy LNBits with Poetry and configure Caddy reverse proxy - hosts: edge + hosts: vipy become: yes vars_files: + - ../../infra_vars.yml + - ../../services_config.yml + - ../../infra_secrets.yml - ./lnbits_vars.yml vars: lnbits_subdomain: "{{ subdomains.lnbits }}" + caddy_sites_dir: "{{ caddy_sites_dir }}" lnbits_domain: "{{ lnbits_subdomain }}.{{ root_domain }}" + uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}" tasks: - name: Create lnbits directory @@ -85,10 +90,6 @@ owner: "{{ ansible_user }}" group: "{{ ansible_user }}" mode: '0644' - # Seed a fresh install only. Without this every re-run replaced the real - # .env (AUTH_SECRET_KEY and all) with the example, logging every user - # out - and would have clobbered a restored one. - force: false - name: Configure LNBits environment variables @@ -146,12 +147,142 @@ enabled: yes state: started - - name: Publish LNBits through Caddy - ansible.builtin.include_role: - name: caddy_site - vars: - caddy_site_name: lnbits - caddy_site_domain: "{{ lnbits_domain }}" - caddy_site_upstream: "localhost:{{ lnbits_port }}" - caddy_site_headers_up: - X-Forwarded-Host: "{{ lnbits_domain }}" + - name: Ensure Caddy sites-enabled directory exists + file: + path: "{{ caddy_sites_dir }}" + state: directory + owner: root + group: root + mode: '0755' + + - name: Ensure Caddyfile includes import directive for sites-enabled + lineinfile: + path: /etc/caddy/Caddyfile + line: 'import sites-enabled/*' + insertafter: EOF + state: present + backup: yes + create: yes + mode: '0644' + + - name: Create Caddy reverse proxy configuration for lnbits + copy: + dest: "{{ caddy_sites_dir }}/lnbits.conf" + content: | + {{ lnbits_domain }} { + reverse_proxy localhost:{{ lnbits_port }} { + header_up X-Forwarded-Host {{ lnbits_domain }} + } + } + owner: root + group: root + mode: '0644' + + - name: Reload Caddy to apply new config + command: systemctl reload caddy + + - name: Create Uptime Kuma monitor setup script for LNBits + delegate_to: localhost + become: no + copy: + dest: /tmp/setup_lnbits_monitor.py + content: | + #!/usr/bin/env python3 + import sys + import yaml + from uptime_kuma_api import UptimeKumaApi, MonitorType + + try: + with open('/tmp/ansible_config.yml', 'r') as f: + config = yaml.safe_load(f) + + url = config['uptime_kuma_url'] + username = config['username'] + password = config['password'] + monitor_url = config['monitor_url'] + monitor_name = config['monitor_name'] + + api = UptimeKumaApi(url, timeout=30) + api.login(username, password) + + # Get all monitors + monitors = api.get_monitors() + + # Find or create "services" group + group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) + if not group: + group_result = api.add_monitor(type='group', name='services') + # Refresh to get the group with id + monitors = api.get_monitors() + group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) + + # Check if monitor already exists + existing_monitor = None + for monitor in monitors: + if monitor.get('name') == monitor_name: + existing_monitor = monitor + break + + # Get ntfy notification ID + notifications = api.get_notifications() + ntfy_notification_id = None + for notif in notifications: + if notif.get('type') == 'ntfy': + ntfy_notification_id = notif.get('id') + break + + if existing_monitor: + print(f"Monitor '{monitor_name}' already exists (ID: {existing_monitor['id']})") + print("Skipping - monitor already configured") + else: + print(f"Creating monitor '{monitor_name}'...") + api.add_monitor( + type=MonitorType.HTTP, + name=monitor_name, + url=monitor_url, + parent=group['id'], + interval=60, + maxretries=3, + retryInterval=60, + notificationIDList={ntfy_notification_id: True} if ntfy_notification_id else {} + ) + + api.disconnect() + print("SUCCESS") + + except Exception as e: + print(f"ERROR: {str(e)}", file=sys.stderr) + sys.exit(1) + mode: '0755' + + - name: Create temporary config for monitor setup + delegate_to: localhost + become: no + copy: + dest: /tmp/ansible_config.yml + content: | + uptime_kuma_url: "{{ uptime_kuma_api_url }}" + username: "{{ uptime_kuma_username }}" + password: "{{ uptime_kuma_password }}" + monitor_url: "https://{{ lnbits_domain }}/api/v1/health" + monitor_name: "LNBits" + mode: '0644' + + - name: Run Uptime Kuma monitor setup + command: python3 /tmp/setup_lnbits_monitor.py + delegate_to: localhost + become: no + register: monitor_setup + changed_when: "'SUCCESS' in monitor_setup.stdout" + ignore_errors: yes + + - name: Clean up temporary files + delegate_to: localhost + become: no + file: + path: "{{ item }}" + state: absent + loop: + - /tmp/setup_lnbits_monitor.py + - /tmp/ansible_config.yml + diff --git a/ansible/services/lnbits/lnbits_vars.yml b/ansible/services/lnbits/lnbits_vars.yml index 466ce78..bdb97df 100644 --- a/ansible/services/lnbits/lnbits_vars.yml +++ b/ansible/services/lnbits/lnbits_vars.yml @@ -3,12 +3,15 @@ lnbits_dir: /opt/lnbits lnbits_data_dir: "{{ lnbits_dir }}/data" lnbits_port: 8765 -# (subdomain in group_vars/all/main.yml, caddy_sites_dir in roles/caddy_site/defaults/) +# (caddy_sites_dir and subdomain now in services_config.yml) # Remote access -remote_host_name: "{{ groups['edge'] | first }}" +remote_host_name: "vipy" remote_host: "{{ hostvars.get(remote_host_name, {}).get('ansible_host', remote_host_name) }}" remote_user: "{{ hostvars.get(remote_host_name, {}).get('ansible_user', 'counterweight') }}" remote_key_file: "{{ hostvars.get(remote_host_name, {}).get('ansible_ssh_private_key_file', '') }}" remote_port: "{{ hostvars.get(remote_host_name, {}).get('ansible_port', 22) }}" +# Local backup +local_backup_dir: "{{ lookup('env', 'HOME') }}/lnbits-backups" +backup_script_path: "{{ lookup('env', 'HOME') }}/.local/bin/lnbits_backup.sh" diff --git a/ansible/services/lnbits/setup_backup_lnbits.yml b/ansible/services/lnbits/setup_backup_lnbits.yml deleted file mode 100644 index 2da0ac0..0000000 --- a/ansible/services/lnbits/setup_backup_lnbits.yml +++ /dev/null @@ -1,28 +0,0 @@ ---- -# LNBits backup: dumps locally on vipy, encrypted with age. -# The old job produced TWO gpg artefacts (data, then .env separately). They are -# folded into one tar here so the wallet database and the .env that configures -# it are always the same point in time; two artefacts written by two runs can -# drift. Pulling one file back out needs no unpacking: -# age -d -i | tar -xzO opt/lnbits/lnbits/.env -- name: Configure the LNBits backup on the edge host - hosts: edge - become: yes - vars_files: - - ./lnbits_vars.yml - - tasks: - - name: Ensure LNBits dumps itself, encrypted, on a timer - ansible.builtin.include_role: - name: backup_source - vars: - backup_source_name: lnbits - backup_source_description: "LNBits" - backup_source_dump_command: "tar -czf - -C / opt/lnbits/data opt/lnbits/lnbits/.env" - backup_source_stop_service: lnbits - backup_source_retention_days: 7 - backup_source_on_calendar: "*-*-* 02:20:00" - # Reported to Gatus as backup-dump_lnbits. The token is this HOST's token, - # shared with its other checks - see infra/400_host_monitoring.yml. - backup_source_push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/backup-dump_lnbits/external" - backup_source_push_token: "{{ gatus_push_tokens[inventory_hostname] }}" diff --git a/ansible/services/lnbits/setup_backup_lnbits_to_lapy.yml b/ansible/services/lnbits/setup_backup_lnbits_to_lapy.yml new file mode 100644 index 0000000..5d10dec --- /dev/null +++ b/ansible/services/lnbits/setup_backup_lnbits_to_lapy.yml @@ -0,0 +1,104 @@ +- name: Configure local backup for LNBits from remote + hosts: lapy + gather_facts: no + vars_files: + - ../../infra_vars.yml + - ./lnbits_vars.yml + vars: + remote_data_path: "{{ lnbits_data_dir }}" + remote_lnbits_dir: "{{ lnbits_dir }}/lnbits" + gpg_recipient: "{{ hostvars['localhost']['gpg_recipient'] | default('') }}" + gpg_key_id: "{{ hostvars['localhost']['gpg_key_id'] | default('') }}" + + tasks: + - name: Debug remote backup vars + debug: + msg: + - "remote_host={{ remote_host }}" + - "remote_user={{ remote_user }}" + - "remote_data_path='{{ remote_data_path }}'" + - "local_backup_dir={{ local_backup_dir }}" + - "gpg_recipient={{ gpg_recipient }}" + - "gpg_key_id={{ gpg_key_id }}" + + - name: Ensure local backup directory exists + file: + path: "{{ local_backup_dir }}" + state: directory + mode: '0755' + + - name: Ensure ~/.local/bin exists + file: + path: "{{ lookup('env', 'HOME') }}/.local/bin" + state: directory + mode: '0755' + + - name: Create backup script + copy: + dest: "{{ backup_script_path }}" + mode: '0750' + content: | + #!/bin/bash + set -euo pipefail + + TIMESTAMP=$(date +'%Y-%m-%d') + ENCRYPTED_BACKUP="{{ local_backup_dir }}/lnbits-backup-$TIMESTAMP.tar.gz.gpg" + + {% if remote_key_file %} + SSH_CMD="ssh -i {{ remote_key_file }} -p {{ remote_port }}" + {% else %} + SSH_CMD="ssh -p {{ remote_port }}" + {% endif %} + + # Stop LNBits service before backup + echo "Stopping LNBits service..." + $SSH_CMD {{ remote_user }}@{{ remote_host }} "sudo systemctl stop lnbits.service" + + # Create encrypted backup on the fly + # First, create a tar archive of the data directory and pipe it through gpg + echo "Creating backup..." + $SSH_CMD {{ remote_user }}@{{ remote_host }} "cd {{ remote_data_path }} && tar -czf - ." | \ + gpg --batch --yes --encrypt --recipient "{{ gpg_recipient }}" --output "$ENCRYPTED_BACKUP" + + # Also backup the .env file separately (smaller, might need quick access) + $SSH_CMD {{ remote_user }}@{{ remote_host }} "cat {{ remote_lnbits_dir }}/.env" | \ + gpg --batch --yes --encrypt --recipient "{{ gpg_recipient }}" --output "{{ local_backup_dir }}/lnbits-env-$TIMESTAMP.gpg" + + # Start LNBits service after backup + echo "Starting LNBits service..." + $SSH_CMD {{ remote_user }}@{{ remote_host }} "sudo systemctl start lnbits.service" + + # Rotate old backups (keep 14 days) + # Calculate cutoff date (14 days ago) and delete backups older than that + CUTOFF_DATE=$(date -d '14 days ago' +'%Y-%m-%d') + for backup_file in "{{ local_backup_dir }}"/lnbits-backup-*.tar.gz.gpg; do + if [ -f "$backup_file" ]; then + # Extract date from filename: lnbits-backup-YYYY-MM-DD.tar.gz.gpg + file_date=$(basename "$backup_file" | sed -n 's/lnbits-backup-\([0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}\)\.tar\.gz\.gpg/\1/p') + if [ -n "$file_date" ] && [ "$file_date" != "$TIMESTAMP" ] && [ "$file_date" \< "$CUTOFF_DATE" ]; then + rm -f "$backup_file" + fi + fi + done + for env_file in "{{ local_backup_dir }}"/lnbits-env-*.gpg; do + if [ -f "$env_file" ]; then + # Extract date from filename: lnbits-env-YYYY-MM-DD.gpg + file_date=$(basename "$env_file" | sed -n 's/lnbits-env-\([0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}\)\.gpg/\1/p') + if [ -n "$file_date" ] && [ "$file_date" != "$TIMESTAMP" ] && [ "$file_date" \< "$CUTOFF_DATE" ]; then + rm -f "$env_file" + fi + fi + done + + echo "Backup completed successfully" + + - name: Ensure cronjob for backup exists + cron: + name: "LNBits backup" + user: "{{ lookup('env', 'USER') }}" + job: "{{ backup_script_path }}" + minute: 5 + hour: "9,12,15,18" + + - name: Run the backup script to make the first backup + command: "{{ backup_script_path }}" diff --git a/ansible/services/memos/deploy_memos_playbook.yml b/ansible/services/memos/deploy_memos_playbook.yml index 8b21b85..da56bd6 100644 --- a/ansible/services/memos/deploy_memos_playbook.yml +++ b/ansible/services/memos/deploy_memos_playbook.yml @@ -1,7 +1,10 @@ - name: Deploy Memos on memos-box - hosts: memos + hosts: memos_box_local become: yes vars_files: + - ../../infra_vars.yml + - ../../services_config.yml + - ../../infra_secrets.yml - ./memos_vars.yml vars: memos_subdomain: "{{ subdomains.memos }}" @@ -62,6 +65,14 @@ owner: root group: root + - name: Clean up temporary files + file: + path: "{{ item }}" + state: absent + loop: + - /tmp/memos.tar.gz + - /tmp/memos + - name: Create memos environment file copy: dest: "{{ memos_config_dir }}/memos.env" @@ -134,21 +145,162 @@ state: restarted -- name: Configure Caddy reverse proxy for Memos on the edge host (via Tailscale) - hosts: edge +- name: Configure Caddy reverse proxy for Memos on vipy (proxying via Tailscale) + hosts: vipy become: yes vars_files: + - ../../infra_vars.yml + - ../../services_config.yml + - ../../infra_secrets.yml - ./memos_vars.yml vars: memos_subdomain: "{{ subdomains.memos }}" + caddy_sites_dir: "{{ caddy_sites_dir }}" memos_domain: "{{ memos_subdomain }}.{{ root_domain }}" + uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}" tasks: - - name: Publish Memos through Caddy (via Tailscale) - ansible.builtin.include_role: - name: caddy_site - vars: - caddy_site_name: memos - caddy_site_domain: "{{ memos_domain }}" - caddy_site_upstream: "{{ memos_tailscale_hostname }}:{{ memos_port }}" - caddy_site_resolvers: "100.100.100.100" + - name: Ensure Caddy sites-enabled directory exists + file: + path: "{{ caddy_sites_dir }}" + state: directory + owner: root + group: root + mode: '0755' + + - name: Ensure Caddyfile includes import directive for sites-enabled + lineinfile: + path: /etc/caddy/Caddyfile + line: 'import sites-enabled/*' + insertafter: EOF + state: present + backup: yes + + - name: Create Caddy reverse proxy configuration for memos (via Tailscale) + copy: + dest: "{{ caddy_sites_dir }}/memos.conf" + content: | + {{ memos_domain }} { + reverse_proxy {{ memos_tailscale_hostname }}:{{ memos_port }} { + # Use Tailscale MagicDNS to resolve the upstream hostname + transport http { + resolvers 100.100.100.100 + } + } + } + owner: root + group: root + mode: '0644' + + - name: Reload Caddy to apply new config + command: systemctl reload caddy + + - name: Create Uptime Kuma monitor setup script for Memos + delegate_to: localhost + become: no + copy: + dest: /tmp/setup_memos_monitor.py + content: | + #!/usr/bin/env python3 + import sys + import traceback + import yaml + from uptime_kuma_api import UptimeKumaApi, MonitorType + + try: + # Load configs + with open('/tmp/ansible_memos_config.yml', 'r') as f: + config = yaml.safe_load(f) + + url = config['uptime_kuma_url'] + username = config['username'] + password = config['password'] + monitor_url = config['monitor_url'] + monitor_name = config['monitor_name'] + + # Connect to Uptime Kuma + api = UptimeKumaApi(url, timeout=30) + api.login(username, password) + + # Get all monitors + monitors = api.get_monitors() + + # Find or create "services" group + group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) + if not group: + group_result = api.add_monitor(type='group', name='services') + # Refresh to get the group with id + monitors = api.get_monitors() + group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) + + # Check if monitor already exists + existing_monitor = None + for monitor in monitors: + if monitor.get('name') == monitor_name: + existing_monitor = monitor + break + + # Get ntfy notification ID + notifications = api.get_notifications() + ntfy_notification_id = None + for notif in notifications: + if notif.get('type') == 'ntfy': + ntfy_notification_id = notif.get('id') + break + + if existing_monitor: + print(f"Monitor '{monitor_name}' already exists (ID: {existing_monitor['id']})") + print("Skipping - monitor already configured") + else: + print(f"Creating monitor '{monitor_name}'...") + api.add_monitor( + type=MonitorType.HTTP, + name=monitor_name, + url=monitor_url, + parent=group['id'], + interval=60, + maxretries=3, + retryInterval=60, + notificationIDList={ntfy_notification_id: True} if ntfy_notification_id else {} + ) + + api.disconnect() + print("SUCCESS") + + except Exception as e: + error_msg = str(e) if str(e) else repr(e) + print(f"ERROR: {error_msg}", file=sys.stderr) + traceback.print_exc(file=sys.stderr) + sys.exit(1) + mode: '0755' + + - name: Create temporary config for monitor setup + delegate_to: localhost + become: no + copy: + dest: /tmp/ansible_memos_config.yml + content: | + uptime_kuma_url: "{{ uptime_kuma_api_url }}" + username: "{{ uptime_kuma_username }}" + password: "{{ uptime_kuma_password }}" + monitor_url: "https://{{ memos_domain }}/healthz" + monitor_name: "Memos" + mode: '0644' + + - name: Run Uptime Kuma monitor setup + command: python3 /tmp/setup_memos_monitor.py + delegate_to: localhost + become: no + register: monitor_setup + changed_when: "'SUCCESS' in monitor_setup.stdout" + ignore_errors: yes + + - name: Clean up temporary files + delegate_to: localhost + become: no + file: + path: "{{ item }}" + state: absent + loop: + - /tmp/setup_memos_monitor.py + - /tmp/ansible_memos_config.yml diff --git a/ansible/services/memos/memos_vars.yml b/ansible/services/memos/memos_vars.yml index 94c6de7..99618db 100644 --- a/ansible/services/memos/memos_vars.yml +++ b/ansible/services/memos/memos_vars.yml @@ -12,7 +12,15 @@ memos_url: "https://github.com/usememos/memos/releases/download/v{{ memos_versio memos_tailscale_hostname: "memos-box" memos_tailscale_ip: "100.64.0.4" -# (subdomain in group_vars/all/main.yml, caddy_sites_dir in roles/caddy_site/defaults/) +# (caddy_sites_dir and subdomain in services_config.yml) # Remote access (for backup from lapy via Tailscale) +backup_host: "{{ memos_tailscale_hostname }}" +backup_user: "counterweight" +backup_key_file: "~/.ssh/counterganzua" +backup_port: 22 + +# Local backup +local_backup_dir: "{{ lookup('env', 'HOME') }}/memos-backups" +backup_script_path: "{{ lookup('env', 'HOME') }}/.local/bin/memos_backup.sh" diff --git a/ansible/services/memos/setup_backup_memos.yml b/ansible/services/memos/setup_backup_memos.yml deleted file mode 100644 index 53ab85d..0000000 --- a/ansible/services/memos/setup_backup_memos.yml +++ /dev/null @@ -1,29 +0,0 @@ ---- -# Memos backup: dumps locally on memos-box, encrypted with age. -# Replaces the lapy pull, which had been writing EMPTY directories since -# 2025-12-27 — its script hardcoded 192.168.1.130, which DHCP later reassigned -# to a different machine that has no rsync. -- name: Configure the Memos backup on its own host - hosts: memos - become: yes - vars_files: - - ./memos_vars.yml - - tasks: - - name: Ensure Memos dumps itself, encrypted, on a timer - ansible.builtin.include_role: - name: backup_source - vars: - backup_source_name: memos - backup_source_description: "Memos" - backup_source_dump_command: "tar -czf - -C / var/lib/memos" - # sqlite in WAL mode: stopping checkpoints the WAL, so the artefact is a - # consistent database rather than a torn mid-write copy. The old rsync - # job did not stop it. - backup_source_stop_service: memos - backup_source_retention_days: 7 - backup_source_on_calendar: "*-*-* 02:00:00" - # Reported to Gatus as backup-dump_memos. The token is this HOST's token, - # shared with its other checks - see infra/400_host_monitoring.yml. - backup_source_push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/backup-dump_memos/external" - backup_source_push_token: "{{ gatus_push_tokens[inventory_hostname] }}" diff --git a/ansible/services/memos/setup_backup_memos_to_lapy.yml b/ansible/services/memos/setup_backup_memos_to_lapy.yml new file mode 100644 index 0000000..6d9c161 --- /dev/null +++ b/ansible/services/memos/setup_backup_memos_to_lapy.yml @@ -0,0 +1,106 @@ +- name: Configure local backup for Memos from memos-box + hosts: lapy + gather_facts: no + vars_files: + - ../../infra_vars.yml + - ./memos_vars.yml + vars: + backup_data_path: "{{ memos_data_dir }}" + + tasks: + - name: Debug remote backup vars + debug: + msg: + - "backup_host={{ backup_host }}" + - "backup_user={{ backup_user }}" + - "backup_data_path='{{ backup_data_path }}'" + - "local_backup_dir={{ local_backup_dir }}" + + - name: Ensure local backup directory exists + file: + path: "{{ local_backup_dir }}" + state: directory + mode: '0755' + + - name: Ensure ~/.local/bin exists + file: + path: "{{ lookup('env', 'HOME') }}/.local/bin" + state: directory + mode: '0755' + + - name: Create backup script + copy: + dest: "{{ backup_script_path }}" + mode: '0750' + content: | + #!/bin/bash + set -euo pipefail + + TIMESTAMP=$(date +'%Y-%m-%d') + BACKUP_DIR="{{ local_backup_dir }}/$TIMESTAMP" + mkdir -p "$BACKUP_DIR" + + {% if backup_key_file %} + SSH_CMD="ssh -i {{ backup_key_file }} -p {{ backup_port }}" + {% else %} + SSH_CMD="ssh -p {{ backup_port }}" + {% endif %} + + rsync -az -e "$SSH_CMD" --rsync-path="sudo rsync" --delete {{ backup_user }}@{{ backup_host }}:{{ backup_data_path }}/ "$BACKUP_DIR/" + + # Rotate old backups (keep 14 days) + # Calculate cutoff date (14 days ago) and delete backups older than that + CUTOFF_DATE=$(date -d '14 days ago' +'%Y-%m-%d') + for dir in "{{ local_backup_dir }}"/20*; do + if [ -d "$dir" ]; then + dir_date=$(basename "$dir") + if [ "$dir_date" != "$TIMESTAMP" ] && [ "$dir_date" \< "$CUTOFF_DATE" ]; then + rm -rf "$dir" + fi + fi + done + + - name: Ensure cronjob for backup exists + cron: + name: "Memos backup" + user: "{{ lookup('env', 'USER') }}" + job: "{{ backup_script_path }}" + minute: 15 + hour: "9,12,15,18" + + - name: Run the backup script to make the first backup + command: "{{ backup_script_path }}" + + - name: Verify backup was created + block: + - name: Get today's date + command: date +'%Y-%m-%d' + register: today_date + changed_when: false + + - name: Check backup directory exists and contains files + stat: + path: "{{ local_backup_dir }}/{{ today_date.stdout }}" + register: backup_dir_stat + + - name: Verify backup directory exists + assert: + that: + - backup_dir_stat.stat.exists + - backup_dir_stat.stat.isdir + fail_msg: "Backup directory {{ local_backup_dir }}/{{ today_date.stdout }} was not created" + success_msg: "Backup directory {{ local_backup_dir }}/{{ today_date.stdout }} exists" + + - name: Check if backup directory contains files + find: + paths: "{{ local_backup_dir }}/{{ today_date.stdout }}" + recurse: yes + register: backup_files + + - name: Verify backup directory is not empty + assert: + that: + - backup_files.files | length > 0 + fail_msg: "Backup directory {{ local_backup_dir }}/{{ today_date.stdout }} exists but is empty" + success_msg: "Backup directory contains {{ backup_files.files | length }} file(s)" + diff --git a/ansible/services/mempool/deploy_mempool_playbook.yml b/ansible/services/mempool/deploy_mempool_playbook.yml index 0042fb0..658180a 100644 --- a/ansible/services/mempool/deploy_mempool_playbook.yml +++ b/ansible/services/mempool/deploy_mempool_playbook.yml @@ -1,33 +1,751 @@ ---- - name: Deploy Mempool Block Explorer with Docker - hosts: mempool + hosts: mempool_box_local become: yes + vars_files: + - ../../infra_vars.yml + - ../../services_config.yml + - ../../infra_secrets.yml + - ./mempool_vars.yml vars: - # Preserves the three push URLs these checks have been reporting to all - # along, so the move to a role changes no behaviour. The role knows nothing - # about Uptime Kuma — these are just "URLs that accept a ping", and whatever - # replaces it sets the same values. - mempool_healthchecks: - - {name: mariadb, label: MariaDB, push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/probe_mempool-mariadb/external"} - - {name: backend, label: Backend, push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/probe_mempool-backend/external"} - - {name: frontend, label: Frontend, push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/probe_mempool-frontend/external"} - # One token for all three components: they run on the same host, so the - # blast radius is already that host. - healthcheck_push_token: "{{ gatus_push_tokens[inventory_hostname] }}" - roles: - - mempool + mempool_subdomain: "{{ subdomains.mempool }}" + mempool_domain: "{{ mempool_subdomain }}.{{ root_domain }}" + uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}" -- name: Configure Caddy reverse proxy for Mempool on the edge host - hosts: edge - become: yes - vars: - mempool_domain: "{{ subdomains.mempool }}.{{ root_domain }}" tasks: - - name: Publish Mempool through Caddy (via Tailscale) - ansible.builtin.include_role: - name: caddy_site - vars: - caddy_site_name: mempool - caddy_site_domain: "{{ mempool_domain }}" - caddy_site_upstream: "{{ hostvars['mempool_box_local'].ansible_host }}:{{ hostvars['mempool_box_local'].mempool_frontend_port }}" - caddy_site_resolvers: "100.100.100.100" + # =========================================== + # Docker Installation (from 910_docker_playbook.yml) + # =========================================== + - name: Remove old Docker-related packages + apt: + name: + - docker.io + - docker-doc + - docker-compose + - podman-docker + - containerd + - runc + state: absent + purge: yes + autoremove: yes + + - name: Update apt cache + apt: + update_cache: yes + + - name: Install prerequisites + apt: + name: + - ca-certificates + - curl + state: present + + - name: Create directory for Docker GPG key + file: + path: /etc/apt/keyrings + state: directory + mode: '0755' + + - name: Download Docker GPG key + get_url: + url: https://download.docker.com/linux/debian/gpg + dest: /etc/apt/keyrings/docker.asc + mode: '0644' + + - name: Get Debian architecture + command: dpkg --print-architecture + register: deb_arch + changed_when: false + + - name: Add Docker repository + apt_repository: + repo: "deb [arch={{ deb_arch.stdout }} signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian {{ ansible_distribution_release }} stable" + filename: docker + state: present + update_cache: yes + + - name: Install Docker packages + apt: + name: + - docker-ce + - docker-ce-cli + - containerd.io + - docker-buildx-plugin + - docker-compose-plugin + state: present + update_cache: yes + + - name: Ensure Docker is started and enabled + systemd: + name: docker + enabled: yes + state: started + + - name: Add user to docker group + user: + name: "{{ ansible_user }}" + groups: docker + append: yes + + # =========================================== + # Mempool Deployment + # =========================================== + - name: Create mempool directories + file: + path: "{{ item }}" + state: directory + owner: "{{ ansible_user }}" + group: "{{ ansible_user }}" + mode: '0755' + loop: + - "{{ mempool_dir }}" + - "{{ mempool_data_dir }}" + - "{{ mempool_mysql_dir }}" + + - name: Create docker-compose.yml for Mempool + copy: + dest: "{{ mempool_dir }}/docker-compose.yml" + content: | + # All containers use host network for Tailscale MagicDNS resolution + services: + mariadb: + image: mariadb:10.11 + container_name: mempool-db + restart: unless-stopped + network_mode: host + environment: + MYSQL_DATABASE: "{{ mariadb_database }}" + MYSQL_USER: "{{ mariadb_user }}" + MYSQL_PASSWORD: "{{ mariadb_mempool_password }}" + MYSQL_ROOT_PASSWORD: "{{ mariadb_mempool_password }}" + volumes: + - {{ mempool_mysql_dir }}:/var/lib/mysql + healthcheck: + test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"] + interval: 10s + timeout: 5s + retries: 5 + start_period: 30s + + mempool-backend: + image: mempool/backend:{{ mempool_version }} + container_name: mempool-backend + restart: unless-stopped + network_mode: host + environment: + # Database (localhost since all containers share host network) + DATABASE_ENABLED: "true" + DATABASE_HOST: "127.0.0.1" + DATABASE_DATABASE: "{{ mariadb_database }}" + DATABASE_USERNAME: "{{ mariadb_user }}" + DATABASE_PASSWORD: "{{ mariadb_mempool_password }}" + # Bitcoin Core/Knots (via Tailnet MagicDNS) + CORE_RPC_HOST: "{{ bitcoin_host }}" + CORE_RPC_PORT: "{{ bitcoin_rpc_port }}" + CORE_RPC_USERNAME: "{{ bitcoin_rpc_user }}" + CORE_RPC_PASSWORD: "{{ bitcoin_rpc_password }}" + # Electrum (Fulcrum via Tailnet MagicDNS) + ELECTRUM_HOST: "{{ fulcrum_host }}" + ELECTRUM_PORT: "{{ fulcrum_port }}" + ELECTRUM_TLS_ENABLED: "{{ fulcrum_tls }}" + # Mempool settings + MEMPOOL_NETWORK: "{{ mempool_network }}" + MEMPOOL_BACKEND: "electrum" + MEMPOOL_CLEAR_PROTECTION_MINUTES: "20" + MEMPOOL_INDEXING_BLOCKS_AMOUNT: "52560" + volumes: + - {{ mempool_data_dir }}:/backend/cache + depends_on: + mariadb: + condition: service_healthy + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:8999/api/v1/backend-info"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 60s + + mempool-frontend: + image: mempool/frontend:{{ mempool_version }} + container_name: mempool-frontend + restart: unless-stopped + network_mode: host + environment: + FRONTEND_HTTP_PORT: "{{ mempool_frontend_port }}" + BACKEND_MAINNET_HTTP_HOST: "127.0.0.1" + depends_on: + - mempool-backend + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:{{ mempool_frontend_port }}"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 30s + owner: "{{ ansible_user }}" + group: "{{ ansible_user }}" + mode: '0644' + + - name: Pull Mempool images + command: docker compose pull + args: + chdir: "{{ mempool_dir }}" + + - name: Deploy Mempool containers with docker compose + command: docker compose up -d + args: + chdir: "{{ mempool_dir }}" + + - name: Wait for MariaDB to be healthy + command: docker inspect --format='{{ '{{' }}.State.Health.Status{{ '}}' }}' mempool-db + register: mariadb_health + until: mariadb_health.stdout == 'healthy' + retries: 30 + delay: 10 + changed_when: false + + - name: Wait for Mempool backend to start + uri: + url: "http://localhost:{{ mempool_backend_port }}/api/v1/backend-info" + method: GET + status_code: 200 + timeout: 10 + register: backend_check + until: backend_check.status == 200 + retries: 30 + delay: 10 + ignore_errors: yes + + - name: Wait for Mempool frontend to be available + uri: + url: "http://localhost:{{ mempool_frontend_port }}" + method: GET + status_code: 200 + timeout: 10 + register: frontend_check + until: frontend_check.status == 200 + retries: 20 + delay: 5 + ignore_errors: yes + + - name: Display deployment status + debug: + msg: + - "Mempool deployment complete!" + - "Frontend: http://localhost:{{ mempool_frontend_port }}" + - "Backend API: http://localhost:{{ mempool_backend_port }}/api/v1/backend-info" + - "Backend check: {{ 'OK' if backend_check.status == 200 else 'Still initializing...' }}" + - "Frontend check: {{ 'OK' if frontend_check.status == 200 else 'Still initializing...' }}" + + # =========================================== + # Health Check Scripts for Uptime Kuma Push Monitors + # =========================================== + - name: Create Mempool MariaDB health check script + copy: + dest: /usr/local/bin/mempool-mariadb-healthcheck-push.sh + content: | + #!/bin/bash + UPTIME_KUMA_PUSH_URL="${UPTIME_KUMA_PUSH_URL}" + + check_container() { + local status=$(docker inspect --format='{{ '{{' }}.State.Health.Status{{ '}}' }}' mempool-db 2>/dev/null) + [ "$status" = "healthy" ] + } + + push_to_uptime_kuma() { + local status=$1 + local msg=$2 + if [ -z "$UPTIME_KUMA_PUSH_URL" ]; then + echo "ERROR: UPTIME_KUMA_PUSH_URL not set" + return 1 + fi + curl -s --max-time 10 --retry 2 -o /dev/null \ + "${UPTIME_KUMA_PUSH_URL}?status=${status}&msg=${msg// /%20}&ping=" || true + } + + if check_container; then + push_to_uptime_kuma "up" "OK" + exit 0 + else + push_to_uptime_kuma "down" "MariaDB container unhealthy" + exit 1 + fi + owner: root + group: root + mode: '0755' + + - name: Create Mempool backend health check script + copy: + dest: /usr/local/bin/mempool-backend-healthcheck-push.sh + content: | + #!/bin/bash + UPTIME_KUMA_PUSH_URL="${UPTIME_KUMA_PUSH_URL}" + BACKEND_PORT={{ mempool_backend_port }} + + check_backend() { + curl -sf --max-time 5 "http://localhost:${BACKEND_PORT}/api/v1/backend-info" > /dev/null 2>&1 + } + + push_to_uptime_kuma() { + local status=$1 + local msg=$2 + if [ -z "$UPTIME_KUMA_PUSH_URL" ]; then + echo "ERROR: UPTIME_KUMA_PUSH_URL not set" + return 1 + fi + curl -s --max-time 10 --retry 2 -o /dev/null \ + "${UPTIME_KUMA_PUSH_URL}?status=${status}&msg=${msg// /%20}&ping=" || true + } + + if check_backend; then + push_to_uptime_kuma "up" "OK" + exit 0 + else + push_to_uptime_kuma "down" "Backend API not responding" + exit 1 + fi + owner: root + group: root + mode: '0755' + + - name: Create Mempool frontend health check script + copy: + dest: /usr/local/bin/mempool-frontend-healthcheck-push.sh + content: | + #!/bin/bash + UPTIME_KUMA_PUSH_URL="${UPTIME_KUMA_PUSH_URL}" + FRONTEND_PORT={{ mempool_frontend_port }} + + check_frontend() { + curl -sf --max-time 5 "http://localhost:${FRONTEND_PORT}" > /dev/null 2>&1 + } + + push_to_uptime_kuma() { + local status=$1 + local msg=$2 + if [ -z "$UPTIME_KUMA_PUSH_URL" ]; then + echo "ERROR: UPTIME_KUMA_PUSH_URL not set" + return 1 + fi + curl -s --max-time 10 --retry 2 -o /dev/null \ + "${UPTIME_KUMA_PUSH_URL}?status=${status}&msg=${msg// /%20}&ping=" || true + } + + if check_frontend; then + push_to_uptime_kuma "up" "OK" + exit 0 + else + push_to_uptime_kuma "down" "Frontend not responding" + exit 1 + fi + owner: root + group: root + mode: '0755' + + # =========================================== + # Systemd Timers for Health Checks + # =========================================== + - name: Create systemd services for health checks + copy: + dest: "/etc/systemd/system/mempool-{{ item.name }}-healthcheck.service" + content: | + [Unit] + Description=Mempool {{ item.label }} Health Check + After=network.target docker.service + + [Service] + Type=oneshot + User=root + ExecStart=/usr/local/bin/mempool-{{ item.name }}-healthcheck-push.sh + Environment=UPTIME_KUMA_PUSH_URL= + StandardOutput=journal + StandardError=journal + + [Install] + WantedBy=multi-user.target + owner: root + group: root + mode: '0644' + loop: + - { name: "mariadb", label: "MariaDB" } + - { name: "backend", label: "Backend" } + - { name: "frontend", label: "Frontend" } + + - name: Create systemd timers for health checks + copy: + dest: "/etc/systemd/system/mempool-{{ item }}-healthcheck.timer" + content: | + [Unit] + Description=Mempool {{ item }} Health Check Timer + + [Timer] + OnBootSec=2min + OnUnitActiveSec=1min + Persistent=true + + [Install] + WantedBy=timers.target + owner: root + group: root + mode: '0644' + loop: + - mariadb + - backend + - frontend + + - name: Reload systemd daemon + systemd: + daemon_reload: yes + + - name: Enable and start health check timers + systemd: + name: "mempool-{{ item }}-healthcheck.timer" + enabled: yes + state: started + loop: + - mariadb + - backend + - frontend + + # =========================================== + # Uptime Kuma Push Monitor Setup + # =========================================== + - name: Create Uptime Kuma push monitor setup script for Mempool + delegate_to: localhost + become: no + copy: + dest: /tmp/setup_mempool_monitors.py + content: | + #!/usr/bin/env python3 + import sys + import traceback + import yaml + from uptime_kuma_api import UptimeKumaApi, MonitorType + + try: + with open('/tmp/ansible_mempool_config.yml', 'r') as f: + config = yaml.safe_load(f) + + url = config['uptime_kuma_url'] + username = config['username'] + password = config['password'] + monitors_to_create = config['monitors'] + + api = UptimeKumaApi(url, timeout=30) + api.login(username, password) + + monitors = api.get_monitors() + + # Find or create "services" group + group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) + if not group: + api.add_monitor(type='group', name='services') + monitors = api.get_monitors() + group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) + + # Get ntfy notification ID + notifications = api.get_notifications() + ntfy_notification_id = None + for notif in notifications: + if notif.get('type') == 'ntfy': + ntfy_notification_id = notif.get('id') + break + + results = {} + for monitor_name in monitors_to_create: + existing = next((m for m in monitors if m.get('name') == monitor_name), None) + + if existing: + print(f"Monitor '{monitor_name}' already exists (ID: {existing['id']})") + push_token = existing.get('pushToken') or existing.get('push_token') + if push_token: + results[monitor_name] = f"{url}/api/push/{push_token}" + print(f"Push URL ({monitor_name}): {results[monitor_name]}") + else: + print(f"Creating push monitor '{monitor_name}'...") + api.add_monitor( + type=MonitorType.PUSH, + name=monitor_name, + parent=group['id'], + interval=90, + maxretries=3, + retryInterval=60, + notificationIDList={ntfy_notification_id: True} if ntfy_notification_id else {} + ) + monitors = api.get_monitors() + new_monitor = next((m for m in monitors if m.get('name') == monitor_name), None) + if new_monitor: + push_token = new_monitor.get('pushToken') or new_monitor.get('push_token') + if push_token: + results[monitor_name] = f"{url}/api/push/{push_token}" + print(f"Push URL ({monitor_name}): {results[monitor_name]}") + + api.disconnect() + print("SUCCESS") + + # Write results to file for Ansible to read + with open('/tmp/mempool_push_urls.yml', 'w') as f: + yaml.dump(results, f) + + except Exception as e: + print(f"ERROR: {str(e)}", file=sys.stderr) + traceback.print_exc(file=sys.stderr) + sys.exit(1) + mode: '0755' + + - name: Create temporary config for monitor setup + delegate_to: localhost + become: no + copy: + dest: /tmp/ansible_mempool_config.yml + content: | + uptime_kuma_url: "{{ uptime_kuma_api_url }}" + username: "{{ uptime_kuma_username }}" + password: "{{ uptime_kuma_password }}" + monitors: + - "Mempool MariaDB" + - "Mempool Backend" + - "Mempool Frontend" + mode: '0644' + + - name: Run Uptime Kuma push monitor setup + command: python3 /tmp/setup_mempool_monitors.py + delegate_to: localhost + become: no + register: monitor_setup + changed_when: "'SUCCESS' in monitor_setup.stdout" + ignore_errors: yes + + - name: Display monitor setup output + debug: + msg: "{{ monitor_setup.stdout_lines }}" + when: monitor_setup.stdout is defined + + - name: Read push URLs from file + slurp: + src: /tmp/mempool_push_urls.yml + delegate_to: localhost + become: no + register: push_urls_file + ignore_errors: yes + + - name: Parse push URLs + set_fact: + push_urls: "{{ push_urls_file.content | b64decode | from_yaml }}" + when: push_urls_file.content is defined + ignore_errors: yes + + - name: Update MariaDB health check service with push URL + lineinfile: + path: /etc/systemd/system/mempool-mariadb-healthcheck.service + regexp: '^Environment=UPTIME_KUMA_PUSH_URL=' + line: "Environment=UPTIME_KUMA_PUSH_URL={{ push_urls['Mempool MariaDB'] }}" + insertafter: '^\[Service\]' + when: push_urls is defined and push_urls['Mempool MariaDB'] is defined + + - name: Update Backend health check service with push URL + lineinfile: + path: /etc/systemd/system/mempool-backend-healthcheck.service + regexp: '^Environment=UPTIME_KUMA_PUSH_URL=' + line: "Environment=UPTIME_KUMA_PUSH_URL={{ push_urls['Mempool Backend'] }}" + insertafter: '^\[Service\]' + when: push_urls is defined and push_urls['Mempool Backend'] is defined + + - name: Update Frontend health check service with push URL + lineinfile: + path: /etc/systemd/system/mempool-frontend-healthcheck.service + regexp: '^Environment=UPTIME_KUMA_PUSH_URL=' + line: "Environment=UPTIME_KUMA_PUSH_URL={{ push_urls['Mempool Frontend'] }}" + insertafter: '^\[Service\]' + when: push_urls is defined and push_urls['Mempool Frontend'] is defined + + - name: Reload systemd after push URL updates + systemd: + daemon_reload: yes + when: push_urls is defined + + - name: Restart health check timers + systemd: + name: "mempool-{{ item }}-healthcheck.timer" + state: restarted + loop: + - mariadb + - backend + - frontend + when: push_urls is defined + + - name: Clean up temporary files + delegate_to: localhost + become: no + file: + path: "{{ item }}" + state: absent + loop: + - /tmp/setup_mempool_monitors.py + - /tmp/ansible_mempool_config.yml + - /tmp/mempool_push_urls.yml + +- name: Configure Caddy reverse proxy for Mempool on vipy + hosts: vipy + become: yes + vars_files: + - ../../infra_vars.yml + - ../../services_config.yml + - ../../infra_secrets.yml + - ./mempool_vars.yml + vars: + mempool_subdomain: "{{ subdomains.mempool }}" + mempool_domain: "{{ mempool_subdomain }}.{{ root_domain }}" + caddy_sites_dir: "{{ caddy_sites_dir }}" + + tasks: + - name: Ensure Caddy sites-enabled directory exists + file: + path: "{{ caddy_sites_dir }}" + state: directory + owner: root + group: root + mode: '0755' + + - name: Ensure Caddyfile includes import directive for sites-enabled + lineinfile: + path: /etc/caddy/Caddyfile + line: 'import sites-enabled/*' + insertafter: EOF + state: present + backup: yes + create: yes + mode: '0644' + + - name: Create Caddy reverse proxy configuration for Mempool + copy: + dest: "{{ caddy_sites_dir }}/mempool.conf" + content: | + {{ mempool_domain }} { + reverse_proxy mempool-box:{{ mempool_frontend_port }} { + # Use Tailscale MagicDNS to resolve the upstream hostname + transport http { + resolvers 100.100.100.100 + } + } + } + owner: root + group: root + mode: '0644' + + - name: Reload Caddy to apply new config + systemd: + name: caddy + state: reloaded + + - name: Display Mempool URL + debug: + msg: "Mempool is now available at https://{{ mempool_domain }}" + + # =========================================== + # Uptime Kuma HTTP Monitor for Public Endpoint + # =========================================== + - name: Create Uptime Kuma HTTP monitor setup script for Mempool + delegate_to: localhost + become: no + copy: + dest: /tmp/setup_mempool_http_monitor.py + content: | + #!/usr/bin/env python3 + import sys + import traceback + import yaml + from uptime_kuma_api import UptimeKumaApi, MonitorType + + try: + with open('/tmp/ansible_mempool_http_config.yml', 'r') as f: + config = yaml.safe_load(f) + + url = config['uptime_kuma_url'] + username = config['username'] + password = config['password'] + monitor_url = config['monitor_url'] + monitor_name = config['monitor_name'] + + api = UptimeKumaApi(url, timeout=30) + api.login(username, password) + + monitors = api.get_monitors() + + # Find or create "services" group + group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) + if not group: + api.add_monitor(type='group', name='services') + monitors = api.get_monitors() + group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) + + # Check if monitor already exists + existing = next((m for m in monitors if m.get('name') == monitor_name), None) + + # Get ntfy notification ID + notifications = api.get_notifications() + ntfy_notification_id = None + for notif in notifications: + if notif.get('type') == 'ntfy': + ntfy_notification_id = notif.get('id') + break + + if existing: + print(f"Monitor '{monitor_name}' already exists (ID: {existing['id']})") + print("Skipping - monitor already configured") + else: + print(f"Creating HTTP monitor '{monitor_name}'...") + api.add_monitor( + type=MonitorType.HTTP, + name=monitor_name, + url=monitor_url, + parent=group['id'], + interval=60, + maxretries=3, + retryInterval=60, + notificationIDList={ntfy_notification_id: True} if ntfy_notification_id else {} + ) + + api.disconnect() + print("SUCCESS") + + except Exception as e: + print(f"ERROR: {str(e)}", file=sys.stderr) + traceback.print_exc(file=sys.stderr) + sys.exit(1) + mode: '0755' + + - name: Create temporary config for HTTP monitor setup + delegate_to: localhost + become: no + copy: + dest: /tmp/ansible_mempool_http_config.yml + content: | + uptime_kuma_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}" + username: "{{ uptime_kuma_username }}" + password: "{{ uptime_kuma_password }}" + monitor_url: "https://{{ mempool_domain }}" + monitor_name: "Mempool Public Access" + mode: '0644' + + - name: Run Uptime Kuma HTTP monitor setup + command: python3 /tmp/setup_mempool_http_monitor.py + delegate_to: localhost + become: no + register: http_monitor_setup + changed_when: "'SUCCESS' in http_monitor_setup.stdout" + ignore_errors: yes + + - name: Display HTTP monitor setup output + debug: + msg: "{{ http_monitor_setup.stdout_lines }}" + when: http_monitor_setup.stdout is defined + + - name: Clean up HTTP monitor temporary files + delegate_to: localhost + become: no + file: + path: "{{ item }}" + state: absent + loop: + - /tmp/setup_mempool_http_monitor.py + - /tmp/ansible_mempool_http_config.yml + diff --git a/ansible/services/mempool/mempool_vars.yml b/ansible/services/mempool/mempool_vars.yml new file mode 100644 index 0000000..d3051c3 --- /dev/null +++ b/ansible/services/mempool/mempool_vars.yml @@ -0,0 +1,33 @@ +# Mempool Configuration Variables + +# Version - Pinned to specific release +mempool_version: "v3.2.1" + +# Directories +mempool_dir: /opt/mempool +mempool_data_dir: "{{ mempool_dir }}/data" +mempool_mysql_dir: "{{ mempool_dir }}/mysql" + +# Network - Bitcoin Core/Knots connection (via Tailnet Magic DNS) +bitcoin_host: "knots-box" +bitcoin_rpc_port: 8332 +# Note: bitcoin_rpc_user and bitcoin_rpc_password are loaded from infra_secrets.yml + +# Network - Fulcrum Electrum server (via Tailnet Magic DNS) +fulcrum_host: "fulcrum-box" +fulcrum_port: 50001 +fulcrum_tls: "false" + +# Mempool network mode +mempool_network: "mainnet" + +# Container ports (internal) +mempool_frontend_port: 8080 +mempool_backend_port: 8999 + +# MariaDB settings +mariadb_database: "mempool" +mariadb_user: "mempool" +# Note: mariadb_mempool_password is loaded from infra_secrets.yml + + diff --git a/ansible/services/monero/deploy_monero_playbook.yml b/ansible/services/monero/deploy_monero_playbook.yml deleted file mode 100644 index ca90693..0000000 --- a/ansible/services/monero/deploy_monero_playbook.yml +++ /dev/null @@ -1,39 +0,0 @@ ---- -# Monero: a full node, reachable from Feather over the WAN. -# -# The chain lives on a dedicated 500 GB disk mounted at /var/lib/monero; the -# role creates the filesystem and the fstab entry, and monerod's unit carries -# RequiresMountsFor so it cannot start without it. An initial sync takes days -# and nothing here touches the data directory afterwards. -# -# Exposure is one HTTPS vhost and no new open ports. See roles/monero/README.md. -- name: Deploy the Monero node - hosts: monero - become: yes - vars: - healthcheck_push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/probe_monerod/external" - healthcheck_push_token: "{{ gatus_push_tokens[inventory_hostname] }}" - roles: - - monero - -- name: Publish the restricted Monero RPC from the edge host - hosts: edge - become: yes - tasks: - # No basic auth here on purpose. monerod's own --rpc-login is HTTP Digest - # and guards this endpoint; Caddy's basic_auth would put a second, Basic - # challenge in front of it that Feather would answer with the wrong scheme. - # Caddy's job is TLS and routing, nothing else. - - name: Expose the restricted Monero RPC over HTTPS - ansible.builtin.include_role: - name: caddy_site - vars: - caddy_site_name: monero - caddy_site_domain: "{{ subdomains.monero }}.{{ root_domain }}" - caddy_site_upstream: "{{ hostvars['monerod_box_local'].ansible_host }}:{{ hostvars['monerod_box_local'].monero_restricted_rpc_port }}" - caddy_site_resolvers: "100.100.100.100" - - # The per-site validate cannot catch a conflict between sites. - - name: Validate the whole Caddyfile - ansible.builtin.command: caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile - changed_when: false diff --git a/ansible/services/navidrome/deploy_navidrome_playbook.yml b/ansible/services/navidrome/deploy_navidrome_playbook.yml deleted file mode 100644 index 56d792b..0000000 --- a/ansible/services/navidrome/deploy_navidrome_playbook.yml +++ /dev/null @@ -1,163 +0,0 @@ ---- -# Navidrome + beets on music-box, published at music.contrapeso.xyz. -# -# See roles/navidrome/README.md for the exposure and backup decisions. The two -# that matter here: no basic_auth in front (the Subsonic API the mobile clients -# speak has nowhere to carry a second credential), and the fail2ban jail below -# covers the web UI only. -- name: Deploy Navidrome on music-box - hosts: music - become: yes - vars: - # Vault key is navidrome_encryption_key, not the role's own variable name - - # assigning a var to itself is a recursive reference, not an override. - navidrome_password_encryption_key: "{{ navidrome_encryption_key }}" - healthcheck_push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/probe_navidrome/external" - healthcheck_push_token: "{{ gatus_push_tokens[inventory_hostname] }}" - roles: - - navidrome - -- name: Publish Navidrome from the edge host - hosts: edge - become: yes - vars: - navidrome_access_log: /var/log/caddy/music.log - tasks: - # The Debian package usually creates this, but `caddy validate` passes with - # the directory missing and the RELOAD is what fails - so the vhost would - # look deployed and serve nothing. Make it explicit. - - name: Ensure the Caddy log directory exists - ansible.builtin.file: - path: "{{ navidrome_access_log | dirname }}" - state: directory - owner: caddy - group: caddy - mode: '0750' - - # The FILE, not just the directory, and this is load-bearing. - # - # `caddy validate` below runs as root and provisions the logging module, - # which CREATES this file root:root 0600. Caddy itself runs as the caddy - # user, so the subsequent reload then dies with: - # loading new config: setting up custom log 'log0': ... - # open /var/log/caddy/music.log: permission denied - # and - the nasty part - `systemctl reload` hangs rather than failing, - # leaving the unit stuck in "reloading" while the OLD config keeps serving. - # So the vhost looks deployed, validate passes, nothing reports an error, - # and the site is simply absent. - # - # Creating it up front with the right owner makes validate's open a no-op. - - name: Ensure the Caddy access log file exists and is owned by caddy - ansible.builtin.file: - path: "{{ navidrome_access_log }}" - state: touch - owner: caddy - group: caddy - mode: '0640' - access_time: preserve - modification_time: preserve - - # caddy_site_body rather than caddy_site_upstream + caddy_site_resolvers: - # this vhost needs a `log` directive so fail2ban has something to read, and - # the role's template has no knob for one. The escape hatch is the - # documented way to do this - gatus and headscale already use it - and it - # leaves the other vhosts untouched. - - name: Expose Navidrome over HTTPS - ansible.builtin.include_role: - name: caddy_site - vars: - caddy_site_name: music - caddy_site_domain: "{{ subdomains.music }}.{{ root_domain }}" - caddy_site_body: | - log { - output file {{ navidrome_access_log }} { - roll_size 10MiB - roll_keep 5 - } - format json - } - reverse_proxy {{ hostvars['music_box_local'].ansible_host }}:{{ hostvars['music_box_local'].navidrome_port }} { - # Tailscale MagicDNS: music-box has no public address. - transport http { - resolvers 100.100.100.100 - } - } - - # The per-site validate cannot catch a conflict between sites. - - name: Validate the whole Caddyfile - ansible.builtin.command: caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile - changed_when: false - - # ── fail2ban ──────────────────────────────────────────────────────────── - # The estate's first jail: 02_firewall_and_fail2ban_playbook.yml installs - # fail2ban everywhere but leaves it on stock config. Kept inline here - # because one jail does not justify a role; if a second service ever wants - # one, that is the moment to promote this to roles/fail2ban_jail alongside - # caddy_site and gatus_endpoint. - # - # Scope, stated so it is not mistaken for more than it is: this catches web - # UI brute force (401 on /auth/login). It does NOT catch Subsonic API brute - # force, because Subsonic answers a bad password with HTTP 200 and the error - # in the body. See roles/navidrome/README.md. - - name: Install the navidrome fail2ban filter - ansible.builtin.copy: - dest: /etc/fail2ban/filter.d/navidrome-caddy.conf - owner: root - group: root - mode: '0644' - content: | - # Managed by Ansible - services/navidrome/deploy_navidrome_playbook.yml - # - # The log file is per-vhost, so any 401 in it is a failed Navidrome - # login and needs no path matching. - [Definition] - failregex = "remote_ip":"".*"status":401 - ignoreregex = - notify: Restart fail2ban - - - name: Install the navidrome fail2ban jail - ansible.builtin.copy: - dest: /etc/fail2ban/jail.d/navidrome.conf - owner: root - group: root - mode: '0644' - content: | - # Managed by Ansible - services/navidrome/deploy_navidrome_playbook.yml - [navidrome] - enabled = true - filter = navidrome-caddy - logpath = {{ navidrome_access_log }} - # ufw, not the Debian default iptables-multiport: UFW is what manages - # the firewall on every host here (infra/02_firewall_and_fail2ban). - banaction = ufw - maxretry = 5 - findtime = 10m - bantime = 1h - notify: Restart fail2ban - - - name: Ensure fail2ban is enabled and running - systemd: - name: fail2ban - enabled: yes - state: started - - - name: Flush handlers so the jail is loaded before we assert on it - meta: flush_handlers - - # A jail that silently failed to load is worse than no jail: it reads as - # protection in `ls /etc/fail2ban/jail.d` and bans nothing. - - name: Assert the navidrome jail is actually active - ansible.builtin.command: fail2ban-client status navidrome - register: navidrome_jail_status - changed_when: false - failed_when: navidrome_jail_status.rc != 0 - - - name: Display the navidrome jail status - ansible.builtin.debug: - var: navidrome_jail_status.stdout_lines - - handlers: - - name: Restart fail2ban - systemd: - name: fail2ban - state: restarted diff --git a/ansible/services/ntfy-emergency-app/deploy_ntfy_emergency_app_playbook.yml b/ansible/services/ntfy-emergency-app/deploy_ntfy_emergency_app_playbook.yml new file mode 100644 index 0000000..b8c0064 --- /dev/null +++ b/ansible/services/ntfy-emergency-app/deploy_ntfy_emergency_app_playbook.yml @@ -0,0 +1,190 @@ +- name: Deploy ntfy-emergency-app with Docker Compose and configure Caddy reverse proxy + hosts: vipy + become: yes + vars_files: + - ../../infra_vars.yml + - ../../infra_secrets.yml + - ../../services_config.yml + - ./ntfy_emergency_app_vars.yml + vars: + ntfy_emergency_app_subdomain: "{{ subdomains.ntfy_emergency_app }}" + caddy_sites_dir: "{{ caddy_sites_dir }}" + ntfy_emergency_app_domain: "{{ ntfy_emergency_app_subdomain }}.{{ root_domain }}" + ntfy_service_domain: "{{ subdomains.ntfy }}.{{ root_domain }}" + ntfy_emergency_app_ntfy_url: "https://{{ ntfy_service_domain }}" + ntfy_emergency_app_ntfy_user: "{{ ntfy_username | default('') }}" + ntfy_emergency_app_ntfy_password: "{{ ntfy_password | default('') }}" + uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}" + + tasks: + - name: Create ntfy-emergency-app directory + file: + path: "{{ ntfy_emergency_app_dir }}" + state: directory + owner: "{{ ansible_user }}" + group: "{{ ansible_user }}" + mode: '0755' + + - name: Create docker-compose.yml for ntfy-emergency-app + copy: + dest: "{{ ntfy_emergency_app_dir }}/docker-compose.yml" + content: | + version: "3" + services: + ntfy-emergency-app: + image: ghcr.io/pmartincalvo/ntfy-emergency-app:latest + container_name: ntfy-emergency-app + restart: unless-stopped + ports: + - "{{ ntfy_emergency_app_port }}:3000" + environment: + NTFY_TOPIC: "{{ ntfy_emergency_app_topic }}" + NTFY_URL: "{{ ntfy_emergency_app_ntfy_url }}" + NTFY_USER: "{{ ntfy_emergency_app_ntfy_user }}" + NTFY_PASSWORD: "{{ ntfy_emergency_app_ntfy_password }}" + UI_MESSAGE: "{{ ntfy_emergency_app_ui_message }}" + + - name: Deploy ntfy-emergency-app container with docker compose + command: docker compose up -d + args: + chdir: "{{ ntfy_emergency_app_dir }}" + + - name: Ensure Caddy sites-enabled directory exists + file: + path: "{{ caddy_sites_dir }}" + state: directory + owner: root + group: root + mode: '0755' + + - name: Ensure Caddyfile includes import directive for sites-enabled + lineinfile: + path: /etc/caddy/Caddyfile + line: 'import sites-enabled/*' + insertafter: EOF + state: present + backup: yes + + - name: Create Caddy reverse proxy configuration for ntfy-emergency-app + copy: + dest: "{{ caddy_sites_dir }}/ntfy-emergency-app.conf" + content: | + {{ ntfy_emergency_app_domain }} { + reverse_proxy localhost:{{ ntfy_emergency_app_port }} + } + owner: root + group: root + mode: '0644' + + - name: Reload Caddy to apply new config + command: systemctl reload caddy + + - name: Create Uptime Kuma monitor setup script for ntfy-emergency-app + delegate_to: localhost + become: no + copy: + dest: /tmp/setup_ntfy_emergency_app_monitor.py + content: | + #!/usr/bin/env python3 + import sys + import traceback + import yaml + from uptime_kuma_api import UptimeKumaApi, MonitorType + + try: + # Load configs + with open('/tmp/ansible_config.yml', 'r') as f: + config = yaml.safe_load(f) + + url = config['uptime_kuma_url'] + username = config['username'] + password = config['password'] + monitor_url = config['monitor_url'] + monitor_name = config['monitor_name'] + + # Connect to Uptime Kuma + api = UptimeKumaApi(url, timeout=30) + api.login(username, password) + + # Get all monitors + monitors = api.get_monitors() + + # Find or create "services" group + group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) + if not group: + group_result = api.add_monitor(type='group', name='services') + # Refresh to get the group with id + monitors = api.get_monitors() + group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) + + # Check if monitor already exists + existing_monitor = None + for monitor in monitors: + if monitor.get('name') == monitor_name: + existing_monitor = monitor + break + + # Get ntfy notification ID + notifications = api.get_notifications() + ntfy_notification_id = None + for notif in notifications: + if notif.get('type') == 'ntfy': + ntfy_notification_id = notif.get('id') + break + + if existing_monitor: + print(f"Monitor '{monitor_name}' already exists (ID: {existing_monitor['id']})") + print("Skipping - monitor already configured") + else: + print(f"Creating monitor '{monitor_name}'...") + api.add_monitor( + type=MonitorType.HTTP, + name=monitor_name, + url=monitor_url, + parent=group['id'], + interval=60, + maxretries=3, + retryInterval=60, + notificationIDList={ntfy_notification_id: True} if ntfy_notification_id else {} + ) + + api.disconnect() + print("SUCCESS") + + except Exception as e: + error_msg = str(e) if str(e) else repr(e) + print(f"ERROR: {error_msg}", file=sys.stderr) + traceback.print_exc(file=sys.stderr) + sys.exit(1) + mode: '0755' + + - name: Create temporary config for monitor setup + delegate_to: localhost + become: no + copy: + dest: /tmp/ansible_config.yml + content: | + uptime_kuma_url: "{{ uptime_kuma_api_url }}" + username: "{{ uptime_kuma_username }}" + password: "{{ uptime_kuma_password }}" + monitor_url: "https://{{ ntfy_emergency_app_domain }}" + monitor_name: "ntfy-emergency-app" + mode: '0644' + + - name: Run Uptime Kuma monitor setup + command: python3 /tmp/setup_ntfy_emergency_app_monitor.py + delegate_to: localhost + become: no + register: monitor_setup + changed_when: "'SUCCESS' in monitor_setup.stdout" + ignore_errors: yes + + - name: Clean up temporary files + delegate_to: localhost + become: no + file: + path: "{{ item }}" + state: absent + loop: + - /tmp/setup_ntfy_emergency_app_monitor.py + - /tmp/ansible_config.yml diff --git a/ansible/services/ntfy-emergency-app/ntfy_emergency_app_vars.yml b/ansible/services/ntfy-emergency-app/ntfy_emergency_app_vars.yml new file mode 100644 index 0000000..415bc4d --- /dev/null +++ b/ansible/services/ntfy-emergency-app/ntfy_emergency_app_vars.yml @@ -0,0 +1,16 @@ +# General +ntfy_emergency_app_dir: /opt/ntfy-emergency-app +ntfy_emergency_app_port: 3000 + +# (caddy_sites_dir and subdomain now in services_config.yml) + +# ntfy configuration +ntfy_emergency_app_topic: "emergencia" +ntfy_emergency_app_ui_message: "Leave Pablo a message, he will respond as soon as possible" + +# Remote access +remote_host_name: "vipy" +remote_host: "{{ hostvars.get(remote_host_name, {}).get('ansible_host', remote_host_name) }}" +remote_user: "{{ hostvars.get(remote_host_name, {}).get('ansible_user', 'counterweight') }}" +remote_key_file: "{{ hostvars.get(remote_host_name, {}).get('ansible_ssh_private_key_file', '') }}" +remote_port: "{{ hostvars.get(remote_host_name, {}).get('ansible_port', 22) }}" diff --git a/ansible/services/ntfy/deploy_ntfy_playbook.yml b/ansible/services/ntfy/deploy_ntfy_playbook.yml new file mode 100644 index 0000000..0729baa --- /dev/null +++ b/ansible/services/ntfy/deploy_ntfy_playbook.yml @@ -0,0 +1,120 @@ +- name: Deploy ntfy and configure Caddy reverse proxy + hosts: watchtower + become: yes + vars_files: + - ../../infra_vars.yml + - ../../infra_secrets.yml + - ../../services_config.yml + - ./ntfy_vars.yml + vars: + ntfy_subdomain: "{{ subdomains.ntfy }}" + caddy_sites_dir: "{{ caddy_sites_dir }}" + ntfy_domain: "{{ ntfy_subdomain }}.{{ root_domain }}" + + tasks: + - name: Ensure /etc/apt/keyrings exists + file: + path: /etc/apt/keyrings + state: directory + mode: '0755' + + - name: Download and dearmor ntfy GPG key + shell: curl -fsSL https://archive.heckel.io/apt/pubkey.txt | gpg --dearmor -o /etc/apt/keyrings/archive.heckel.io.gpg + args: + creates: /etc/apt/keyrings/archive.heckel.io.gpg + + - name: Add ntfy APT repository + copy: + dest: /etc/apt/sources.list.d/archive.heckel.io.list + content: | + deb [arch=amd64 signed-by=/etc/apt/keyrings/archive.heckel.io.gpg] https://archive.heckel.io/apt debian main + mode: '0644' + + - name: Update APT cache + apt: + update_cache: yes + + - name: Install ntfy + apt: + name: ntfy + state: present + + - name: Ensure ntfy cache directories exist + file: + path: "{{ item }}" + state: directory + owner: ntfy + group: ntfy + mode: '0755' + loop: + - /var/cache/ntfy + - /var/cache/ntfy/attachments + + - name: Deploy ntfy configuration file + copy: + dest: /etc/ntfy/server.yml + content: | + base-url: "http://{{ ntfy_domain }}" + listen-http: ":{{ ntfy_port }}" + cache-file: "/var/cache/ntfy/cache.db" + attachment-cache-dir: "/var/cache/ntfy/attachments" + behind-proxy: true + auth-file: "/var/lib/ntfy/user.db" + auth-default-access: "deny-all" + owner: root + group: root + mode: '0644' + notify: Restart ntfy + + - name: Enable and start ntfy service + systemd: + name: ntfy + enabled: yes + state: started + + - name: Create ntfy admin user + shell: | + (echo "{{ ntfy_password }}"; echo "{{ ntfy_password }}") | ntfy user add --role=admin "{{ ntfy_username }}" + + - name: Ensure Caddy sites-enabled directory exists + file: + path: "{{ caddy_sites_dir }}" + state: directory + owner: root + group: root + mode: '0755' + + - name: Ensure Caddyfile includes import directive for sites-enabled + lineinfile: + path: /etc/caddy/Caddyfile + line: 'import sites-enabled/*' + insertafter: EOF + state: present + backup: yes + + - name: Create Caddy reverse proxy configuration for ntfy + copy: + dest: "{{ caddy_sites_dir }}/ntfy.conf" + content: | + {{ ntfy_domain }}, http://{{ ntfy_domain }} { + reverse_proxy 127.0.0.1:{{ ntfy_port }} + + @httpget { + protocol http + method GET + path_regexp ^/([-_a-z0-9]{0,64}$|docs/|static/) + } + redir @httpget https://{host}{uri} + } + owner: root + group: root + mode: '0644' + + - name: Reload Caddy to apply new config + command: systemctl reload caddy + + handlers: + - name: Restart ntfy + systemd: + name: ntfy + state: restarted \ No newline at end of file diff --git a/ansible/services/ntfy/ntfy_vars.yml b/ansible/services/ntfy/ntfy_vars.yml new file mode 100644 index 0000000..5ebec37 --- /dev/null +++ b/ansible/services/ntfy/ntfy_vars.yml @@ -0,0 +1,3 @@ +ntfy_port: 6674 + +# ntfy_topic now lives in services_config.yml under service_settings.ntfy.topic \ No newline at end of file diff --git a/ansible/services/ntfy/setup_ntfy_uptime_kuma_notification.yml b/ansible/services/ntfy/setup_ntfy_uptime_kuma_notification.yml new file mode 100644 index 0000000..5ba03f1 --- /dev/null +++ b/ansible/services/ntfy/setup_ntfy_uptime_kuma_notification.yml @@ -0,0 +1,155 @@ +- name: Setup ntfy as Uptime Kuma Notification Channel + hosts: watchtower + become: no + vars_files: + - ../../infra_vars.yml + - ../../services_config.yml + - ../../infra_secrets.yml + - ./ntfy_vars.yml + + vars: + ntfy_subdomain: "{{ subdomains.ntfy }}" + ntfy_topic: "{{ service_settings.ntfy.topic }}" + uptime_kuma_subdomain: "{{ subdomains.uptime_kuma }}" + ntfy_domain: "{{ ntfy_subdomain }}.{{ root_domain }}" + ntfy_server_url: "https://{{ ntfy_domain }}" + ntfy_priority: 4 # 1=min, 2=low, 3=default, 4=high, 5=max + uptime_kuma_api_url: "https://{{ uptime_kuma_subdomain }}.{{ root_domain }}" + + tasks: + - name: Validate Uptime Kuma configuration + assert: + that: + - uptime_kuma_api_url is defined + - uptime_kuma_api_url != "" + - uptime_kuma_username is defined + - uptime_kuma_username != "" + - uptime_kuma_password is defined + - uptime_kuma_password != "" + fail_msg: "uptime_kuma_api_url, uptime_kuma_username and uptime_kuma_password must be set" + + - name: Validate ntfy configuration + assert: + that: + - ntfy_domain is defined + - ntfy_domain != "" + - ntfy_topic is defined + - ntfy_topic != "" + - ntfy_username is defined + - ntfy_username != "" + - ntfy_password is defined + - ntfy_password != "" + fail_msg: "ntfy_domain, ntfy_topic, ntfy_username and ntfy_password must be set" + + - name: Create Uptime Kuma ntfy notification setup script + copy: + dest: /tmp/setup_uptime_kuma_ntfy_notification.py + content: | + #!/usr/bin/env python3 + import sys + import json + from uptime_kuma_api import UptimeKumaApi + + def main(): + api_url = sys.argv[1] + username = sys.argv[2] + password = sys.argv[3] + notification_name = sys.argv[4] + ntfy_server_url = sys.argv[5] + ntfy_topic = sys.argv[6] + ntfy_username = sys.argv[7] + ntfy_password = sys.argv[8] + ntfy_priority = int(sys.argv[9]) + + api = UptimeKumaApi(api_url, timeout=60, wait_events=2.0) + api.login(username, password) + + # Get all notifications + notifications = api.get_notifications() + + # Find existing ntfy notification by name + existing_notification = next((n for n in notifications if n.get('name') == notification_name), None) + + notification_data = { + 'name': notification_name, + 'type': 'ntfy', + 'isDefault': True, # Apply to all monitors by default + 'applyExisting': True, # Apply to existing monitors + 'ntfyserverurl': ntfy_server_url, + 'ntfytopic': ntfy_topic, + 'ntfyusername': ntfy_username, + 'ntfypassword': ntfy_password, + 'ntfyPriority': ntfy_priority + } + + if existing_notification: + notification = api.edit_notification(existing_notification['id'], **notification_data) + action = "updated" + else: + notification = api.add_notification(**notification_data) + action = "created" + + # Output result as JSON + result = { + 'notification_id': notification['id'], + 'notification_name': notification_name, + 'ntfy_server': ntfy_server_url, + 'ntfy_topic': ntfy_topic, + 'action': action + } + print(json.dumps(result)) + + api.disconnect() + + if __name__ == '__main__': + main() + mode: '0755' + delegate_to: localhost + become: no + + - name: Run Uptime Kuma ntfy notification setup script + command: > + {{ ansible_playbook_python }} + /tmp/setup_uptime_kuma_ntfy_notification.py + "{{ uptime_kuma_api_url }}" + "{{ uptime_kuma_username }}" + "{{ uptime_kuma_password }}" + "ntfy ({{ ntfy_topic }})" + "{{ ntfy_server_url }}" + "{{ ntfy_topic }}" + "{{ ntfy_username }}" + "{{ ntfy_password }}" + "{{ ntfy_priority }}" + register: notification_setup_result + delegate_to: localhost + become: no + changed_when: false + + - name: Parse notification setup result + set_fact: + notification_info_parsed: "{{ notification_setup_result.stdout | from_json }}" + + - name: Display notification information + debug: + msg: | + ✓ ntfy notification channel {{ notification_info_parsed.action }} successfully! + + Notification Name: ntfy ({{ ntfy_topic }}) + ntfy Server: {{ ntfy_server_url }} + ntfy Topic: {{ ntfy_topic }} + Priority: {{ ntfy_priority }} (4=high) + Default for all monitors: Yes + Applied to existing monitors: Yes + + All Uptime Kuma monitors will now send alerts to your ntfy server + on the "{{ ntfy_topic }}" topic. + + You can subscribe to alerts at: {{ ntfy_server_url }}/{{ ntfy_topic }} + + - name: Clean up temporary Uptime Kuma setup script + file: + path: /tmp/setup_uptime_kuma_ntfy_notification.py + state: absent + delegate_to: localhost + become: no + diff --git a/ansible/services/personal-blog/deploy_personal_blog_playbook.yml b/ansible/services/personal-blog/deploy_personal_blog_playbook.yml index d3a9951..f4ee8ec 100644 --- a/ansible/services/personal-blog/deploy_personal_blog_playbook.yml +++ b/ansible/services/personal-blog/deploy_personal_blog_playbook.yml @@ -1,11 +1,16 @@ - name: Deploy personal blog static site with Caddy file server - hosts: edge + hosts: vipy become: yes vars_files: + - ../../infra_vars.yml + - ../../services_config.yml + - ../../infra_secrets.yml - ./personal_blog_vars.yml vars: personal_blog_subdomain: "{{ subdomains.personal_blog }}" + caddy_sites_dir: "{{ caddy_sites_dir }}" personal_blog_domain: "{{ personal_blog_subdomain }}.{{ root_domain }}" + uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}" tasks: - name: Ensure user is in www-data group @@ -45,14 +50,140 @@ owner: "{{ ansible_user }}" group: www-data mode: '0664' - # Only on a fresh web root. Without this, every re-run (and site.yml) - # overwrote the real deployed index.html with the placeholder. - force: false - - name: Publish the blog through Caddy - ansible.builtin.include_role: - name: caddy_site - vars: - caddy_site_name: personal-blog - caddy_site_domain: "{{ personal_blog_domain }}" - caddy_site_root: "{{ personal_blog_web_root }}" + - name: Ensure Caddy sites-enabled directory exists + file: + path: "{{ caddy_sites_dir }}" + state: directory + owner: root + group: root + mode: '0755' + + - name: Ensure Caddyfile includes import directive for sites-enabled + lineinfile: + path: /etc/caddy/Caddyfile + line: 'import sites-enabled/*' + insertafter: EOF + state: present + backup: yes + + - name: Create Caddy file server configuration for personal blog + copy: + dest: "{{ caddy_sites_dir }}/personal-blog.conf" + content: | + {{ personal_blog_domain }} { + root * {{ personal_blog_web_root }} + file_server + } + owner: root + group: root + mode: '0644' + + - name: Reload Caddy to apply new config + command: systemctl reload caddy + + - name: Create Uptime Kuma monitor setup script for Personal Blog + delegate_to: localhost + become: no + copy: + dest: /tmp/setup_personal_blog_monitor.py + content: | + #!/usr/bin/env python3 + import sys + import yaml + from uptime_kuma_api import UptimeKumaApi, MonitorType + + try: + with open('/tmp/ansible_config.yml', 'r') as f: + config = yaml.safe_load(f) + + url = config['uptime_kuma_url'] + username = config['username'] + password = config['password'] + monitor_url = config['monitor_url'] + monitor_name = config['monitor_name'] + + api = UptimeKumaApi(url, timeout=30) + api.login(username, password) + + # Get all monitors + monitors = api.get_monitors() + + # Find or create "services" group + group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) + if not group: + group_result = api.add_monitor(type='group', name='services') + # Refresh to get the group with id + monitors = api.get_monitors() + group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) + + # Check if monitor already exists + existing_monitor = None + for monitor in monitors: + if monitor.get('name') == monitor_name: + existing_monitor = monitor + break + + # Get ntfy notification ID + notifications = api.get_notifications() + ntfy_notification_id = None + for notif in notifications: + if notif.get('type') == 'ntfy': + ntfy_notification_id = notif.get('id') + break + + if existing_monitor: + print(f"Monitor '{monitor_name}' already exists (ID: {existing_monitor['id']})") + print("Skipping - monitor already configured") + else: + print(f"Creating monitor '{monitor_name}'...") + api.add_monitor( + type=MonitorType.HTTP, + name=monitor_name, + url=monitor_url, + parent=group['id'], + interval=60, + maxretries=3, + retryInterval=60, + notificationIDList={ntfy_notification_id: True} if ntfy_notification_id else {} + ) + + api.disconnect() + print("SUCCESS") + + except Exception as e: + print(f"ERROR: {str(e)}", file=sys.stderr) + sys.exit(1) + mode: '0755' + + - name: Create temporary config for monitor setup + delegate_to: localhost + become: no + copy: + dest: /tmp/ansible_config.yml + content: | + uptime_kuma_url: "{{ uptime_kuma_api_url }}" + username: "{{ uptime_kuma_username }}" + password: "{{ uptime_kuma_password }}" + monitor_url: "https://{{ personal_blog_domain }}" + monitor_name: "Personal Blog" + mode: '0644' + + - name: Run Uptime Kuma monitor setup + command: python3 /tmp/setup_personal_blog_monitor.py + delegate_to: localhost + become: no + register: monitor_setup + changed_when: "'SUCCESS' in monitor_setup.stdout" + ignore_errors: yes + + - name: Clean up temporary files + delegate_to: localhost + become: no + file: + path: "{{ item }}" + state: absent + loop: + - /tmp/setup_personal_blog_monitor.py + - /tmp/ansible_config.yml + diff --git a/ansible/services/personal-blog/personal_blog_vars.yml b/ansible/services/personal-blog/personal_blog_vars.yml index a1f34e3..59e0921 100644 --- a/ansible/services/personal-blog/personal_blog_vars.yml +++ b/ansible/services/personal-blog/personal_blog_vars.yml @@ -4,7 +4,7 @@ personal_blog_web_root: "/var/www/pablohere.contrapeso.xyz" # Remote access for deployment -remote_host_name: "{{ groups['edge'] | first }}" +remote_host_name: "vipy" remote_host: "{{ hostvars.get(remote_host_name, {}).get('ansible_host', remote_host_name) }}" remote_user: "{{ hostvars.get(remote_host_name, {}).get('ansible_user', 'counterweight') }}" remote_key_file: "{{ hostvars.get(remote_host_name, {}).get('ansible_ssh_private_key_file', '') }}" diff --git a/ansible/services/personal-blog/setup_deploy_alias_lapy.yml b/ansible/services/personal-blog/setup_deploy_alias_lapy.yml index bd9b715..2b90d68 100644 --- a/ansible/services/personal-blog/setup_deploy_alias_lapy.yml +++ b/ansible/services/personal-blog/setup_deploy_alias_lapy.yml @@ -1,7 +1,8 @@ - name: Configure deployment alias for personal blog in lapy .bashrc - hosts: control + hosts: lapy gather_facts: no vars_files: + - ../../infra_vars.yml - ./personal_blog_vars.yml vars: bashrc_path: "{{ lookup('env', 'HOME') }}/.bashrc" diff --git a/ansible/services/phoenixd/deploy_phoenixd_playbook.yml b/ansible/services/phoenixd/deploy_phoenixd_playbook.yml deleted file mode 100644 index b9cf243..0000000 --- a/ansible/services/phoenixd/deploy_phoenixd_playbook.yml +++ /dev/null @@ -1,15 +0,0 @@ ---- -# phoenixd: Lightning node on the edge host, used by LNBits as a wallet backend. -# Never exposed through Caddy — the HTTP API stays on loopback. -- name: Deploy phoenixd on the edge host - hosts: edge - become: yes - vars: - # phoenixd's health check has never reported anywhere since the Uptime Kuma - # decommissioning — its systemd Environment= was left empty. Leaving it empty - # preserves that; the check still runs and its exit code is still the answer. - # Set this to plug in whatever monitoring replaces it. - healthcheck_push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/probe_phoenixd/external" - healthcheck_push_token: "{{ gatus_push_tokens[inventory_hostname] }}" - roles: - - phoenixd diff --git a/ansible/services/signal-api/deploy_signal_api_playbook.yml b/ansible/services/signal-api/deploy_signal_api_playbook.yml deleted file mode 100644 index d4690f9..0000000 --- a/ansible/services/signal-api/deploy_signal_api_playbook.yml +++ /dev/null @@ -1,44 +0,0 @@ ---- -# The Signal transport for Gatus alerts. -# -# Deliberately NOT published and NOT fronted by Caddy: the API has no -# authentication of any kind, so it is reachable only from the shared docker -# network that Gatus is on. See roles/signal_api/README.md, including the -# one-time manual step to link the device. -- name: Deploy the Signal API on the observability host - hosts: observability - become: yes - roles: - - signal_api - - # post_tasks, not a second play: the registration below needs the role's - # defaults (service name, port) in scope, and a separate play would not have - # them. - post_tasks: - # Monitored, because a dead alert transport is the worst kind of dead: every - # check could be failing and nothing would tell you. Gatus polls it over the - # shared network - the same path the alerts take - so this proves the actual - # delivery route rather than merely that a container is running. - # - # Deliberately NOT backed up: the data directory holds Signal private keys, - # and the recovery path is to link the device again from the phone. Backing - # it up would copy a credential off the host to buy nothing. - - name: Register the signal-api health endpoint with Gatus - ansible.builtin.include_role: - name: gatus_endpoint - vars: - gatus_endpoint_name: signal-api - gatus_endpoint_pulled: - - name: signal-api - group: infrastructure - url: "http://{{ signal_api_service_name }}:{{ signal_api_port }}/v1/health" - interval: 5m - # Deliberately NOT alerted via Signal: if this endpoint is down, - # Signal is exactly what cannot deliver the alert. It is visible on - # the dashboard, and its failure shows up indirectly as every other - # alert going missing. - conditions: - # /v1/health answers 204 No Content, not 200 - checked live. Any - # 2xx is asserted rather than the exact code, so an upstream - # change from 204 to 200 does not read as an outage. - - "[STATUS] < 300" diff --git a/ansible/services/uptime_kuma/deploy_uptime_kuma_playbook.yml b/ansible/services/uptime_kuma/deploy_uptime_kuma_playbook.yml new file mode 100644 index 0000000..4af3858 --- /dev/null +++ b/ansible/services/uptime_kuma/deploy_uptime_kuma_playbook.yml @@ -0,0 +1,74 @@ +- name: Deploy Uptime Kuma with Docker Compose and configure Caddy reverse proxy + hosts: watchtower + become: yes + vars_files: + - ../../infra_vars.yml + - ../../services_config.yml + - ./uptime_kuma_vars.yml + vars: + uptime_kuma_subdomain: "{{ subdomains.uptime_kuma }}" + caddy_sites_dir: "{{ caddy_sites_dir }}" + uptime_kuma_domain: "{{ uptime_kuma_subdomain }}.{{ root_domain }}" + + tasks: + - name: Create uptime kuma directory + file: + path: "{{ uptime_kuma_dir }}" + state: directory + owner: "{{ ansible_user }}" + group: "{{ ansible_user }}" + mode: '0755' + + - name: Create docker-compose.yml for uptime kuma + copy: + dest: "{{ uptime_kuma_dir }}/docker-compose.yml" + content: | + version: "3" + services: + uptime-kuma: + image: louislam/uptime-kuma:latest + container_name: uptime-kuma + restart: unless-stopped + ports: + - "{{ uptime_kuma_port }}:3001" + volumes: + - ./data:/app/data + dns: + - 1.1.1.1 + - 9.9.9.9 + - 8.8.8.8 + + - name: Deploy uptime kuma container with docker compose + command: docker compose up -d + args: + chdir: "{{ uptime_kuma_dir }}" + + - name: Ensure Caddy sites-enabled directory exists + file: + path: /etc/caddy/sites-enabled + state: directory + owner: root + group: root + mode: '0755' + + - name: Ensure Caddyfile includes import directive for sites-enabled + lineinfile: + path: /etc/caddy/Caddyfile + line: 'import sites-enabled/*' + insertafter: EOF + state: present + backup: yes + + - name: Create Caddy reverse proxy configuration for uptime kuma + copy: + dest: "{{ caddy_sites_dir }}/uptime-kuma.conf" + content: | + {{ uptime_kuma_domain }} { + reverse_proxy localhost:{{ uptime_kuma_port }} + } + owner: root + group: root + mode: '0644' + + - name: Reload Caddy to apply new config + command: systemctl reload caddy \ No newline at end of file diff --git a/ansible/services/uptime_kuma/setup_backup_uptime_kuma_to_lapy.yml b/ansible/services/uptime_kuma/setup_backup_uptime_kuma_to_lapy.yml new file mode 100644 index 0000000..9ae9713 --- /dev/null +++ b/ansible/services/uptime_kuma/setup_backup_uptime_kuma_to_lapy.yml @@ -0,0 +1,107 @@ +- name: Configure local backup for Uptime Kuma from remote + hosts: lapy + gather_facts: no + vars_files: + - ../../infra_vars.yml + - ./uptime_kuma_vars.yml + vars: + remote_data_path: "{{ uptime_kuma_data_dir }}" + local_backup_dir: "{{ lookup('env', 'HOME') }}/uptime-kuma-backups" + backup_script_path: "{{ lookup('env', 'HOME') }}/.local/bin/uptime_kuma_backup.sh" + + tasks: + - name: Debug remote backup vars + debug: + msg: + - "remote_host={{ remote_host }}" + - "remote_user={{ remote_user }}" + - "remote_data_path='{{ remote_data_path }}'" + - "local_backup_dir={{ local_backup_dir }}" + + - name: Ensure local backup directory exists + file: + path: "{{ local_backup_dir }}" + state: directory + mode: '0755' + + - name: Ensure ~/.local/bin exists + file: + path: "{{ lookup('env', 'HOME') }}/.local/bin" + state: directory + mode: '0755' + + - name: Create backup script + copy: + dest: "{{ backup_script_path }}" + mode: '0750' + content: | + #!/bin/bash + set -euo pipefail + + TIMESTAMP=$(date +'%Y-%m-%d') + BACKUP_DIR="{{ local_backup_dir }}/$TIMESTAMP" + mkdir -p "$BACKUP_DIR" + + {% if remote_key_file %} + SSH_CMD="ssh -i {{ remote_key_file }} -p {{ remote_port }}" + {% else %} + SSH_CMD="ssh -p {{ remote_port }}" + {% endif %} + + rsync -az -e "$SSH_CMD" --delete {{ remote_user }}@{{ remote_host }}:{{ remote_data_path }}/ "$BACKUP_DIR/" + + # Rotate old backups (keep 14 days) + # Calculate cutoff date (14 days ago) and delete backups older than that + CUTOFF_DATE=$(date -d '14 days ago' +'%Y-%m-%d') + for dir in "{{ local_backup_dir }}"/20*; do + if [ -d "$dir" ]; then + dir_date=$(basename "$dir") + if [ "$dir_date" != "$TIMESTAMP" ] && [ "$dir_date" \< "$CUTOFF_DATE" ]; then + rm -rf "$dir" + fi + fi + done + + - name: Ensure cronjob for backup exists + cron: + name: "Uptime Kuma backup" + user: "{{ lookup('env', 'USER') }}" + job: "{{ backup_script_path }}" + minute: 0 + hour: "9,12,15,18" + + - name: Run the backup script to make the first backup + command: "{{ backup_script_path }}" + + - name: Verify backup was created + block: + - name: Get today's date + command: date +'%Y-%m-%d' + register: today_date + changed_when: false + + - name: Check backup directory exists and contains files + stat: + path: "{{ local_backup_dir }}/{{ today_date.stdout }}" + register: backup_dir_stat + + - name: Verify backup directory exists + assert: + that: + - backup_dir_stat.stat.exists + - backup_dir_stat.stat.isdir + fail_msg: "Backup directory {{ local_backup_dir }}/{{ today_date.stdout }} was not created" + success_msg: "Backup directory {{ local_backup_dir }}/{{ today_date.stdout }} exists" + + - name: Check if backup directory contains files + find: + paths: "{{ local_backup_dir }}/{{ today_date.stdout }}" + recurse: yes + register: backup_files + + - name: Verify backup directory is not empty + assert: + that: + - backup_files.files | length > 0 + fail_msg: "Backup directory {{ local_backup_dir }}/{{ today_date.stdout }} exists but is empty" + success_msg: "Backup directory contains {{ backup_files.files | length }} file(s)" diff --git a/ansible/services/uptime_kuma/uptime_kuma_vars.yml b/ansible/services/uptime_kuma/uptime_kuma_vars.yml new file mode 100644 index 0000000..3263f49 --- /dev/null +++ b/ansible/services/uptime_kuma/uptime_kuma_vars.yml @@ -0,0 +1,15 @@ +# General +uptime_kuma_dir: /opt/uptime-kuma +uptime_kuma_data_dir: "{{ uptime_kuma_dir }}/data" +uptime_kuma_port: 3001 + +# Remote access +remote_host_name: "watchtower" +remote_host: "{{ hostvars.get(remote_host_name, {}).get('ansible_host', remote_host_name) }}" +remote_user: "{{ hostvars.get(remote_host_name, {}).get('ansible_user', 'counterweight') }}" +remote_key_file: "{{ hostvars.get(remote_host_name, {}).get('ansible_ssh_private_key_file', '') }}" +remote_port: "{{ hostvars.get(remote_host_name, {}).get('ansible_port', 22) }}" + +# Local backup +local_backup_dir: "{{ lookup('env', 'HOME') }}/uptime-kuma-backups" +backup_script_path: "{{ lookup('env', 'HOME') }}/.local/bin/uptime_kuma_backup.sh" diff --git a/ansible/services/vaultwarden/deploy_vaultwarden_playbook.yml b/ansible/services/vaultwarden/deploy_vaultwarden_playbook.yml index 0a4a080..0340538 100644 --- a/ansible/services/vaultwarden/deploy_vaultwarden_playbook.yml +++ b/ansible/services/vaultwarden/deploy_vaultwarden_playbook.yml @@ -1,11 +1,16 @@ - name: Deploy Vaultwarden with Docker Compose and configure Caddy reverse proxy - hosts: edge + hosts: vipy become: yes vars_files: + - ../../infra_vars.yml + - ../../services_config.yml + - ../../infra_secrets.yml - ./vaultwarden_vars.yml vars: vaultwarden_subdomain: "{{ subdomains.vaultwarden }}" + caddy_sites_dir: "{{ caddy_sites_dir }}" vaultwarden_domain: "{{ vaultwarden_subdomain }}.{{ root_domain }}" + uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}" tasks: - name: Create vaultwarden directory @@ -27,16 +32,13 @@ container_name: vaultwarden restart: unless-stopped ports: - # Loopback only. Docker writes its own iptables rules AHEAD of - # UFW, so a bare "8222:80" was reachable from the internet as - # plain HTTP, bypassing Caddy and TLS. Caddy is the only client. - - "127.0.0.1:{{ vaultwarden_port }}:80" + - "{{ vaultwarden_port }}:80" volumes: - ./data:/data environment: WEBSOCKET_ENABLED: 'true' DOMAIN: "https://{{ vaultwarden_domain }}" - SIGNUPS_ALLOWED: '{{ vaultwarden_signups_allowed | bool | lower }}' + SIGNUPS_ALLOWED: 'true' LOG_FILE: /data/vaultwarden.log - name: Deploy vaultwarden container with docker compose @@ -79,10 +81,143 @@ name: fail2ban state: restarted - - name: Publish Vaultwarden through Caddy - ansible.builtin.include_role: - name: caddy_site - vars: - caddy_site_name: vaultwarden - caddy_site_domain: "{{ vaultwarden_domain }}" - caddy_site_upstream: "localhost:{{ vaultwarden_port }}" + - name: Ensure Caddy sites-enabled directory exists + file: + path: "{{ caddy_sites_dir }}" + state: directory + owner: root + group: root + mode: '0755' + + - name: Ensure Caddyfile includes import directive for sites-enabled + lineinfile: + path: /etc/caddy/Caddyfile + line: 'import sites-enabled/*' + insertafter: EOF + state: present + backup: yes + + - name: Create Caddy reverse proxy configuration for vaultwarden + copy: + dest: "{{ caddy_sites_dir }}/vaultwarden.conf" + content: | + {{ vaultwarden_domain }} { + reverse_proxy localhost:{{ vaultwarden_port }} + } + owner: root + group: root + mode: '0644' + + - name: Reload Caddy to apply new config + command: systemctl reload caddy + + - name: Create Uptime Kuma monitor setup script for Vaultwarden + delegate_to: localhost + become: no + copy: + dest: /tmp/setup_vaultwarden_monitor.py + content: | + #!/usr/bin/env python3 + import sys + import traceback + import yaml + from uptime_kuma_api import UptimeKumaApi, MonitorType + + try: + # Load configs + with open('/tmp/ansible_config.yml', 'r') as f: + config = yaml.safe_load(f) + + url = config['uptime_kuma_url'] + username = config['username'] + password = config['password'] + monitor_url = config['monitor_url'] + monitor_name = config['monitor_name'] + + # Connect to Uptime Kuma + api = UptimeKumaApi(url, timeout=30) + api.login(username, password) + + # Get all monitors + monitors = api.get_monitors() + + # Find or create "services" group + group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) + if not group: + group_result = api.add_monitor(type='group', name='services') + # Refresh to get the group with id + monitors = api.get_monitors() + group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None) + + # Check if monitor already exists + existing_monitor = None + for monitor in monitors: + if monitor.get('name') == monitor_name: + existing_monitor = monitor + break + + # Get ntfy notification ID + notifications = api.get_notifications() + ntfy_notification_id = None + for notif in notifications: + if notif.get('type') == 'ntfy': + ntfy_notification_id = notif.get('id') + break + + if existing_monitor: + print(f"Monitor '{monitor_name}' already exists (ID: {existing_monitor['id']})") + print("Skipping - monitor already configured") + else: + print(f"Creating monitor '{monitor_name}'...") + api.add_monitor( + type=MonitorType.HTTP, + name=monitor_name, + url=monitor_url, + parent=group['id'], + interval=60, + maxretries=3, + retryInterval=60, + notificationIDList={ntfy_notification_id: True} if ntfy_notification_id else {} + ) + + api.disconnect() + print("SUCCESS") + + except Exception as e: + error_msg = str(e) if str(e) else repr(e) + print(f"ERROR: {error_msg}", file=sys.stderr) + traceback.print_exc(file=sys.stderr) + sys.exit(1) + mode: '0755' + + - name: Create temporary config for monitor setup + delegate_to: localhost + become: no + copy: + dest: /tmp/ansible_config.yml + content: | + uptime_kuma_url: "{{ uptime_kuma_api_url }}" + username: "{{ uptime_kuma_username }}" + password: "{{ uptime_kuma_password }}" + monitor_url: "https://{{ vaultwarden_domain }}/alive" + monitor_name: "Vaultwarden" + mode: '0644' + + - name: Run Uptime Kuma monitor setup + command: python3 /tmp/setup_vaultwarden_monitor.py + delegate_to: localhost + become: no + register: monitor_setup + changed_when: "'SUCCESS' in monitor_setup.stdout" + ignore_errors: yes + + - name: Clean up temporary files + delegate_to: localhost + become: no + file: + path: "{{ item }}" + state: absent + loop: + - /tmp/setup_vaultwarden_monitor.py + - /tmp/ansible_config.yml + diff --git a/ansible/services/vaultwarden/disable_vaultwarden_sign_ups_playbook.yml b/ansible/services/vaultwarden/disable_vaultwarden_sign_ups_playbook.yml index bccc2cd..b041e8e 100644 --- a/ansible/services/vaultwarden/disable_vaultwarden_sign_ups_playbook.yml +++ b/ansible/services/vaultwarden/disable_vaultwarden_sign_ups_playbook.yml @@ -1,7 +1,8 @@ - name: Disable Vaultwarden Signups - hosts: edge + hosts: vipy become: yes vars_files: + - ../../infra_vars.yml - ./vaultwarden_vars.yml tasks: diff --git a/ansible/services/vaultwarden/setup_backup_vaultwarden.yml b/ansible/services/vaultwarden/setup_backup_vaultwarden.yml deleted file mode 100644 index d0dfdfa..0000000 --- a/ansible/services/vaultwarden/setup_backup_vaultwarden.yml +++ /dev/null @@ -1,28 +0,0 @@ ---- -# Vaultwarden backup: dumps locally on vipy, encrypted with age. -# Previously rsynced to lapy in the CLEAR; the artefact now never exists -# unencrypted, on disk or on the wire. -- name: Configure the Vaultwarden backup on the edge host - hosts: edge - become: yes - vars_files: - - ./vaultwarden_vars.yml - - tasks: - - name: Ensure Vaultwarden dumps itself, encrypted, on a timer - ansible.builtin.include_role: - name: backup_source - vars: - backup_source_name: vaultwarden - backup_source_description: "Vaultwarden" - backup_source_dump_command: "tar -czf - -C / opt/vaultwarden/data" - # Not systemd — a docker compose stack — so stop/start explicitly. - # sqlite in WAL mode, hence stopping at all. - backup_source_stop_command: "docker compose -f /opt/vaultwarden/docker-compose.yml stop" - backup_source_start_command: "docker compose -f /opt/vaultwarden/docker-compose.yml start" - backup_source_retention_days: 7 - backup_source_on_calendar: "*-*-* 02:10:00" - # Reported to Gatus as backup-dump_vaultwarden. The token is this HOST's token, - # shared with its other checks - see infra/400_host_monitoring.yml. - backup_source_push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/backup-dump_vaultwarden/external" - backup_source_push_token: "{{ gatus_push_tokens[inventory_hostname] }}" diff --git a/ansible/services/vaultwarden/setup_backup_vaultwarden_to_lapy.yml b/ansible/services/vaultwarden/setup_backup_vaultwarden_to_lapy.yml new file mode 100644 index 0000000..064d633 --- /dev/null +++ b/ansible/services/vaultwarden/setup_backup_vaultwarden_to_lapy.yml @@ -0,0 +1,105 @@ +- name: Configure local backup for Vaultwarden from remote + hosts: lapy + gather_facts: no + vars_files: + - ../../infra_vars.yml + - ./vaultwarden_vars.yml + vars: + remote_data_path: "{{ vaultwarden_data_dir }}" + + tasks: + - name: Debug remote backup vars + debug: + msg: + - "remote_host={{ remote_host }}" + - "remote_user={{ remote_user }}" + - "remote_data_path='{{ remote_data_path }}'" + - "local_backup_dir={{ local_backup_dir }}" + + - name: Ensure local backup directory exists + file: + path: "{{ local_backup_dir }}" + state: directory + mode: '0755' + + - name: Ensure ~/.local/bin exists + file: + path: "{{ lookup('env', 'HOME') }}/.local/bin" + state: directory + mode: '0755' + + - name: Create backup script + copy: + dest: "{{ backup_script_path }}" + mode: '0750' + content: | + #!/bin/bash + set -euo pipefail + + TIMESTAMP=$(date +'%Y-%m-%d') + BACKUP_DIR="{{ local_backup_dir }}/$TIMESTAMP" + mkdir -p "$BACKUP_DIR" + + {% if remote_key_file %} + SSH_CMD="ssh -i {{ remote_key_file }} -p {{ remote_port }}" + {% else %} + SSH_CMD="ssh -p {{ remote_port }}" + {% endif %} + + rsync -az -e "$SSH_CMD" --delete {{ remote_user }}@{{ remote_host }}:{{ remote_data_path }}/ "$BACKUP_DIR/" + + # Rotate old backups (keep 14 days) + # Calculate cutoff date (14 days ago) and delete backups older than that + CUTOFF_DATE=$(date -d '14 days ago' +'%Y-%m-%d') + for dir in "{{ local_backup_dir }}"/20*; do + if [ -d "$dir" ]; then + dir_date=$(basename "$dir") + if [ "$dir_date" != "$TIMESTAMP" ] && [ "$dir_date" \< "$CUTOFF_DATE" ]; then + rm -rf "$dir" + fi + fi + done + + - name: Ensure cronjob for backup exists + cron: + name: "Vaultwarden backup" + user: "{{ lookup('env', 'USER') }}" + job: "{{ backup_script_path }}" + minute: 5 + hour: "9,12,15,18" + + - name: Run the backup script to make the first backup + command: "{{ backup_script_path }}" + + - name: Verify backup was created + block: + - name: Get today's date + command: date +'%Y-%m-%d' + register: today_date + changed_when: false + + - name: Check backup directory exists and contains files + stat: + path: "{{ local_backup_dir }}/{{ today_date.stdout }}" + register: backup_dir_stat + + - name: Verify backup directory exists + assert: + that: + - backup_dir_stat.stat.exists + - backup_dir_stat.stat.isdir + fail_msg: "Backup directory {{ local_backup_dir }}/{{ today_date.stdout }} was not created" + success_msg: "Backup directory {{ local_backup_dir }}/{{ today_date.stdout }} exists" + + - name: Check if backup directory contains files + find: + paths: "{{ local_backup_dir }}/{{ today_date.stdout }}" + recurse: yes + register: backup_files + + - name: Verify backup directory is not empty + assert: + that: + - backup_files.files | length > 0 + fail_msg: "Backup directory {{ local_backup_dir }}/{{ today_date.stdout }} exists but is empty" + success_msg: "Backup directory contains {{ backup_files.files | length }} file(s)" diff --git a/ansible/services/vaultwarden/vaultwarden_vars.yml b/ansible/services/vaultwarden/vaultwarden_vars.yml index c7470f5..75e527d 100644 --- a/ansible/services/vaultwarden/vaultwarden_vars.yml +++ b/ansible/services/vaultwarden/vaultwarden_vars.yml @@ -3,19 +3,15 @@ vaultwarden_dir: /opt/vaultwarden vaultwarden_data_dir: "{{ vaultwarden_dir }}/data" vaultwarden_port: 8222 -# Public registration. OFF: the accounts exist, and this vault is on the open -# internet. It used to be hardcoded 'true' in the compose file and turned off by -# a separate manual playbook - so every routine run of the deploy (site.yml -# included) silently re-opened sign-ups. Flip to true only to add an account, -# then back. -vaultwarden_signups_allowed: false - -# (subdomain in group_vars/all/main.yml, caddy_sites_dir in roles/caddy_site/defaults/) +# (caddy_sites_dir and subdomain now in services_config.yml) # Remote access -remote_host_name: "{{ groups['edge'] | first }}" +remote_host_name: "vipy" remote_host: "{{ hostvars.get(remote_host_name, {}).get('ansible_host', remote_host_name) }}" remote_user: "{{ hostvars.get(remote_host_name, {}).get('ansible_user', 'counterweight') }}" remote_key_file: "{{ hostvars.get(remote_host_name, {}).get('ansible_ssh_private_key_file', '') }}" remote_port: "{{ hostvars.get(remote_host_name, {}).get('ansible_port', 22) }}" +# Local backup +local_backup_dir: "{{ lookup('env', 'HOME') }}/vaultwarden-backups" +backup_script_path: "{{ lookup('env', 'HOME') }}/.local/bin/vaultwarden_backup.sh" diff --git a/ansible/services/xmrblocks/deploy_xmrblocks_playbook.yml b/ansible/services/xmrblocks/deploy_xmrblocks_playbook.yml deleted file mode 100644 index c3dcc29..0000000 --- a/ansible/services/xmrblocks/deploy_xmrblocks_playbook.yml +++ /dev/null @@ -1,39 +0,0 @@ ---- -# The Monero block explorer, on the same box as the node. -# -# xmrblocks reads monerod's LMDB directly rather than going through RPC, which -# is why it lives here and not on its own VM. It is a 26 MB binary and 132 KB -# of templates; the expensive part is building it, and that happens elsewhere -# (scripts/build-xmrblocks.sh) because this box has 2 GB and a single monero -# translation unit needs ~3.2 GB. See roles/xmrblocks/README.md. -- name: Deploy the Monero blockchain explorer - hosts: monero - become: yes - vars: - healthcheck_push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/probe_xmrblocks/external" - healthcheck_push_token: "{{ gatus_push_tokens[inventory_hostname] }}" - roles: - - xmrblocks - -- name: Publish the explorer from the edge host - hosts: edge - become: yes - tasks: - # No basic auth: this is a public block explorer over public chain data, - # which is what a block explorer is for. Caddy terminates TLS and proxies. - # Note the explorer has NO authentication of its own, so everything it can - # do is world-reachable - including the tx pusher, which relays a raw - # transaction through our node. That is deliberate; see the role README. - - name: Expose the Monero explorer over HTTPS - ansible.builtin.include_role: - name: caddy_site - vars: - caddy_site_name: monero-explorer - caddy_site_domain: "{{ subdomains.monero_explorer }}.{{ root_domain }}" - caddy_site_upstream: "{{ hostvars['monerod_box_local'].ansible_host }}:{{ hostvars['monerod_box_local'].xmrblocks_port }}" - caddy_site_resolvers: "100.100.100.100" - - # The per-site validate cannot catch a conflict between sites. - - name: Validate the whole Caddyfile - ansible.builtin.command: caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile - changed_when: false diff --git a/ansible/services_config.yml b/ansible/services_config.yml new file mode 100644 index 0000000..56cc570 --- /dev/null +++ b/ansible/services_config.yml @@ -0,0 +1,36 @@ +# Centralized Services Configuration +# Subdomains and Caddy settings for all services + +# Edit these subdomains to match your preferences +subdomains: + # Monitoring Services (on watchtower) + ntfy: ntfy + uptime_kuma: uptime + + # VPN Infrastructure (on spacey) + headscale: headscale + + # Core Services (on vipy) + vaultwarden: vault + forgejo: forgejo + lnbits: wallet + + # Secondary Services (on vipy) + ntfy_emergency_app: avisame + personal_blog: pablohere + + # Memos (on memos-box) + memos: memos + + # Mempool Block Explorer (on mempool_box, proxied via vipy) + mempool: mempool + +# Caddy configuration +caddy_sites_dir: /etc/caddy/sites-enabled + +# Service-specific settings shared across playbooks +service_settings: + ntfy: + topic: alerts + headscale: + namespace: counter-net diff --git a/ansible/site.yml b/ansible/site.yml deleted file mode 100644 index 7867397..0000000 --- a/ansible/site.yml +++ /dev/null @@ -1,95 +0,0 @@ ---- -# Everything, in the order it has to happen. -# -# This file is a TABLE OF CONTENTS, not a second source of truth. It says what -# runs and in what order. It does NOT say which hosts get what — that stays on -# the `hosts:` line inside each playbook, exactly where it is today. Nothing -# moves; this file only makes the set readable in one place. -# -# What runs on a host? ansible-playbook site.yml --limit --list-hosts -# Who gets thing Y? the `hosts:` line in Y's own playbook -# What is a host? ansible-inventory --graph -# -# Run a slice with --limit, or run one playbook directly as before. Nothing here -# changes how any individual playbook behaves. - -# ── Baseline: every managed machine ───────────────────────────────────────── -- import_playbook: infra/01_user_and_access_setup_playbook.yml -- import_playbook: infra/02_firewall_and_fail2ban_playbook.yml -- import_playbook: infra/900_install_rsync.yml -- import_playbook: infra/920_join_headscale_mesh.yml -# Idempotent and kept permanently: guarantees a rebuilt or restored host cannot -# quietly bring the Uptime-Kuma-era monitoring back. -- import_playbook: infra/409_remove_legacy_monitoring.yml - -# ── Monitoring ────────────────────────────────────────────────────────────── -# Gatus first: the three plays below register endpoints with it, and registering -# against a host that is not serving yet would simply fail. -- import_playbook: services/gatus/deploy_gatus_playbook.yml -# The Signal transport for Gatus alerts. Shares a docker network with Gatus and -# publishes no ports - the API has no authentication. Needs a one-time manual -# device link; see roles/signal_api/README.md. -- import_playbook: services/signal-api/deploy_signal_api_playbook.yml -- import_playbook: infra/400_host_monitoring.yml -- import_playbook: infra/401_service_monitoring.yml -- import_playbook: infra/402_public_monitoring.yml -# Registers where the per-service probes report. The probes themselves are -# deployed by each service's own playbook further down; the endpoints must exist -# before the first push arrives. -- import_playbook: infra/403_service_probe_registration.yml -# 910_docker says `hosts: managed`, but only 5 of 11 managed hosts have or need -# Docker. Left out until it has a [docker] group — see the note in PLAN_7. - -# ── The hypervisor ────────────────────────────────────────────────────────── -- import_playbook: infra/nodito/31_proxmox_community_repos_playbook.yml -- import_playbook: infra/nodito/32_zfs_pool_setup_playbook.yml -- import_playbook: infra/nodito/34_nut_ups_setup_playbook.yml - -# ── Reverse proxy, before anything that registers a vhost ─────────────────── -- import_playbook: services/caddy_playbook.yml - -# ── Services ──────────────────────────────────────────────────────────────── -- import_playbook: services/bitcoin-knots/deploy_bitcoin_knots_playbook.yml -- import_playbook: services/fulcrum/deploy_fulcrum_playbook.yml -- import_playbook: services/datum-gateway/deploy_datum_gateway_playbook.yml -- import_playbook: services/mempool/deploy_mempool_playbook.yml -- import_playbook: services/monero/deploy_monero_playbook.yml -- import_playbook: services/xmrblocks/deploy_xmrblocks_playbook.yml -- import_playbook: services/memos/deploy_memos_playbook.yml -- import_playbook: services/navidrome/deploy_navidrome_playbook.yml -- import_playbook: services/jar-lid-archive/deploy_jar_lid_archive_playbook.yml -- import_playbook: services/forgejo-runner/deploy_forgejo_runner_playbook.yml -- import_playbook: services/phoenixd/deploy_phoenixd_playbook.yml -- import_playbook: services/headscale/deploy_headscale_playbook.yml -- import_playbook: services/vaultwarden/deploy_vaultwarden_playbook.yml -- import_playbook: services/forgejo/deploy_forgejo_playbook.yml -- import_playbook: services/lnbits/deploy_lnbits_playbook.yml -- import_playbook: services/personal-blog/deploy_personal_blog_playbook.yml -- import_playbook: services/arbret/deploy_arbret_edge_playbook.yml - -# ── Backups: each source dumps itself, the box pulls ──────────────────────── -- import_playbook: services/headscale/setup_backup_headscale.yml -- import_playbook: services/vaultwarden/setup_backup_vaultwarden.yml -- import_playbook: services/forgejo/setup_backup_forgejo.yml -- import_playbook: services/lnbits/setup_backup_lnbits.yml -- import_playbook: services/memos/setup_backup_memos.yml -- import_playbook: services/jar-lid-archive/setup_backup_jar_lid_archive.yml -- import_playbook: playbooks/backups.yml - -# Deliberately not here. Every playbook in the repo is either imported above or -# listed below, so this file accounts for all of them: -# -# infra/910_docker_playbook.yml says `hosts: managed`, but Docker is on 5 -# of 11 managed hosts and those 5 are exactly -# the ones that need it. Running it would -# install Docker on the Bitcoin node and the -# hypervisor. Needs a [docker] group first. -# -# infra/nodito/30_proxmox_bootstrap one-shot: bare-metal bootstrap, run once -# infra/nodito/33_..._cloud_template one-shot: builds the VM template -# -# -# services/vaultwarden/disable_ deliberate manual actions, not convergence -# vaultwarden_sign_ups_playbook.yml -# services/personal-blog/setup_ -# deploy_alias_lapy.yml diff --git a/arbret_handoff.md b/arbret_handoff.md deleted file mode 100644 index 5503a50..0000000 --- a/arbret_handoff.md +++ /dev/null @@ -1,456 +0,0 @@ -# arbret on prd-vipy: monitoring and backups - -Handoff for the arbret owner. It covers what changed when prd-arbret was lost, and what -your Ansible repo needs to do so that arbret is **monitored by Gatus** and **backed up to -small-backups-box** from its new home. - -Written 2026-10-17, during the rebuild. - ---- - -## 0. What changed, in one screen - -| | Before | Now | -|---|---|---| -| Host | `prd-arbret`, a droplet of its own | **`prd-vipy`**, shared with the personal_infra services | -| Monitoring | Uptime Kuma (push URLs on `uptime.contrapeso.xyz`) | **Gatus** at `https://status.contrapeso.xyz`, running on `prd-spacey`. Uptime Kuma is gone for good. | -| Backup destination | small-backups-box pulls `arbret@prd-arbret:/opt/arbret/backups/` | small-backups-box pulls **`backup-pull@prd-vipy:/opt/backups/arbret/`** | -| TLS / front door | your own, on prd-arbret | **our Caddy on prd-vipy** (it owns 80/443) | - -Who does what: - -| Area | **You** (arbret repo) | **Us** (personal_infra repo) | -|---|---|---| -| App, workers, Postgres, analytics | deploy and run them | — | -| Public HTTPS for `arbret.com` | tell us your local port | Caddy vhost, certificates | -| Gatus endpoint definitions | — | all of them: Gatus config lives on prd-spacey, which you don't touch | -| Pulled checks (HTTPS, `/api/health`) | keep `/api/health` honest | define and run them | -| systemd unit checks (`arbret-server`, `arbret-worker`, `postgresql`) | keep the unit names stable | add them to vipy's monitored units | -| Host liveness and disk | — | already covered for prd-vipy | -| Analytics timer result | **push it** (§2.4) | register the endpoint | -| Backup dump | **produce it to the contract** (§3) | pull it, retain it 90 days, check freshness | -| Backup dump result | **push it** (§3.5) | register the endpoint | - -Your work comes down to three things: - -1. Live on a shared host without stepping on it (§1). -2. Push two results to Gatus: analytics and backup. -3. Write the dump to a new place, in a fixed shape. - ---- - -## 1. Living on a shared host - -prd-vipy also runs Vaultwarden, Forgejo, LNbits, phoenixd and a reverse proxy for a -dozen other services. Our Ansible converges it regularly. Anything below that your -playbooks change will be reverted by ours, or will break ours. - -### Owned by personal_infra: do not manage these - -- **Ports 80 and 443 and the Caddy install** (`/etc/caddy/**`). Don't install nginx, - certbot or a second Caddy, and don't bind 80/443. Give us a port and we write the vhost - (see below). -- **UFW and fail2ban.** The default policy is deny-incoming, with only SSH and 80/443 - open. Don't disable UFW or reset its rules. If you need a port reachable from outside, - ask us. -- **Users `counterweight` and `backup-pull`**, and their `authorized_keys`. In - particular **don't create, modify or authorise keys for `backup-pull`**: we create it - and it is shared by every backup on this host (§3). -- **`/opt/backups/`** (the parent directory, `root:root 0755`). You own exactly - `/opt/backups/arbret/` and nothing beside it. -- **Tailscale/Headscale membership** of the host, and the tailnet ACLs. -- **Docker.** It's installed for our services. You don't need it (arbret is native - systemd); if you start using it, don't touch `/etc/docker/daemon.json`. -- systemd unit names we use: `caddy`, `forgejo`, `lnbits`, `phoenixd`, - `*-backup.{service,timer}` for vaultwarden/lnbits/forgejo, `liveness-healthcheck`, - `disk-usage-healthcheck`, `service-health-healthcheck`. Prefix all of yours with - `arbret-` and there's no risk of collision. - -### Free for you - -- PostgreSQL. We run no Postgres on this host, so the system `postgresql` package and - cluster are yours. -- `/opt/arbret/`, `/etc/arbret/`, the `arbret` system user, `arbret-*` units and timers. - -### Local ports already taken - -`22, 80, 443, 3000, 7657, 8222, 8333, 8765, 9740, 23334, 50002`. Bind your app to -`127.0.0.1:` on anything else and tell us the port. Our Caddy then serves -`arbret.com` with a Let's Encrypt certificate and reverse-proxies to it, setting -`X-Forwarded-For`, `X-Forwarded-Proto` and `X-Forwarded-Host`. Configure the app to -trust proxy headers from `127.0.0.1` only. - -Port `4117` used to be reachable from one admin laptop over the tailnet. If that's still -needed, say so and we'll add the ACL for prd-vipy. Don't open it in UFW yourself. - -### The host itself - -Debian 13, clock in **UTC**, tailnet name `prd-vipy`. - ---- - -## 2. Monitoring with Gatus - -### 2.1 How it works here - -Gatus runs on prd-spacey and answers at `https://status.contrapeso.xyz`. Alerts go to -Signal. There are two kinds of check: - -- **Pulled.** Gatus makes the request itself (HTTP, TCP, DNS) every N seconds and - evaluates conditions. Nothing is installed on your side. This is the right kind for - anything with a reachable URL. -- **Pushed ("external endpoints").** Something on the host `POST`s a result to Gatus, - and Gatus alarms if a result is `false` **or if nothing arrives within the heartbeat - window**. This is the right kind for things with no URL: timers, jobs, internal state. - The heartbeat is what makes a dead timer visible: silence is a failure. - -Everything about an endpoint (name, group, token, heartbeat, alert rules) is defined in -**our** repo, because Gatus's config files live on prd-spacey. Your side of a push check -is just an HTTP call. - -### 2.2 The push API - -``` -POST https://status.contrapeso.xyz/api/v1/endpoints/{KEY}/external?success={true|false}&error={message} -Authorization: Bearer {TOKEN} -``` - -- **`KEY`** is `sanitize(group) + "_" + sanitize(name)`, where sanitize lowercases and - turns any of `/ _ . , space # + &` into `-`. Use the exact keys in the table below. -- **`success`**: `true` or `false`. -- **`error`**: free text, URL-encoded. It's shown on the dashboard and in the Signal - alert, so on success it's fine to send something useful like `ok 31M`. -- **`TOKEN`**: a bearer token we give you (§4). It's dedicated to arbret: it can only - write arbret's endpoints, and nobody else's token can write yours. - -Responses: - -| HTTP | Meaning | -|---|---| -| `200` | recorded | -| `401` | wrong token for that key | -| `404` | key not registered (yet). We register before you deploy. | - -Test it by hand once you have the token: - -```bash -curl -i -X POST -H "Authorization: Bearer $TOKEN" \ - "https://status.contrapeso.xyz/api/v1/endpoints/probe_arbret-analytics/external?success=true&error=manual%20test" -``` - -### 2.3 Your old Uptime Kuma monitors, mapped - -| Old monitor (prod) | New check | Gatus key | Kind | Who | -|---|---|---|---|---| -| `https`: front door returns 200 | `GET https://arbret.com/` → 200 | `public_arbret` | pulled, 5 min | us | -| `health`: `/api/health` returns ok | `GET https://arbret.com/api/health` → 200 and `[BODY].status == ok` | `public_arbret-health` | pulled, 5 min | us | -| `arbret-server`: unit active | systemd unit check on prd-vipy | `services_vipy-arbret-server` | push, every 5 min, by our host script | us | -| `arbret-worker`: unit active | same | `services_vipy-arbret-worker` | same | us | -| `postgresql`: unit active | same | `services_vipy-postgresql` | same | us | -| `arbret-analytics`: timer ran OK | your analytics run reports its result | `probe_arbret-analytics` | **push, by you**, hourly; heartbeat 130m | **you** (§2.4) | -| (new) beancount: daily job ran OK | your beancount run reports its result | `probe_arbret-beancount` | **push, by you**, daily 03:00; heartbeat 26h | **you** (§2.4, same hook) | -| `arbret-backup`: timer ran OK | your backup run reports its result | `backup-dump_arbret` | **push, by you** | **you** (§3.5) | -| prd-arbret liveness / disk | covered by prd-vipy's own host checks | `liveness_vipy`, `disk_vipy` | push | us | - -We also check: - -- `backup-store_arbret`: the backup **arrived** on small-backups-box, is fresh (<26h, - read from the filename timestamp), non-empty, and not suspiciously smaller than the - previous one. -- `arbret.com` resolves to prd-vipy, checked every 6 hours. -- `arbret.com` registration expiry, alarming 14 days ahead. This already existed. - -**Delete, don't port, every Uptime Kuma push script, timer and URL in your repo.** The -endpoint no longer exists. A leftover timer will just fail silently every two minutes -forever. - -The pulled health check relies on `/api/health` returning HTTP 200 with a JSON body that -has `"status": "ok"` when healthy, and **a non-200 when it isn't**. If the body differs, -tell us the exact shape. A health endpoint that checks the database is worth more than -one that only proves the process answers. - -### 2.4 What you implement: the analytics push - -The cleanest hook is `ExecStopPost=` on the oneshot service the timer runs. systemd runs -it after every run, success or failure, and sets `$SERVICE_RESULT` and `$EXIT_STATUS` -for it. So no wrapper script is needed, and the run itself stays untouched. - -`/etc/arbret/gatus.env` (`root:root 0600`): - -``` -GATUS_PUSH_BASE=https://status.contrapeso.xyz/api/v1/endpoints -GATUS_PUSH_TOKEN= -``` - -`/usr/local/bin/arbret-gatus-push` (`root:root 0755`): - -```bash -#!/usr/bin/env bash -# Usage: arbret-gatus-push -# Meant for ExecStopPost=, where systemd provides SERVICE_RESULT / EXIT_STATUS. -# Can also be called by hand: SERVICE_RESULT=success arbret-gatus-push -set -uo pipefail -KEY="$1" -: "${GATUS_PUSH_BASE:?}" "${GATUS_PUSH_TOKEN:?}" - -if [ "${SERVICE_RESULT:-}" = "success" ]; then - success=true; msg="ok" -else - success=false; msg="${SERVICE_RESULT:-unknown} exit=${EXIT_STATUS:-?}" -fi -enc=$(printf '%s' "$msg" | sed 's/%/%25/g; s/ /%20/g; s/&/%26/g; s/+/%2B/g; s/#/%23/g; s/=/%3D/g') - -# Never let a monitoring hiccup fail the job itself. -curl -s -o /dev/null --max-time 15 --retry 2 --retry-delay 3 -X POST \ - -H "Authorization: Bearer ${GATUS_PUSH_TOKEN}" \ - "${GATUS_PUSH_BASE}/${KEY}/external?success=${success}&error=${enc}" || true -exit 0 -``` - -In the analytics service unit: - -```ini -[Service] -Type=oneshot -EnvironmentFile=/etc/arbret/gatus.env -ExecStart=...your existing command... -ExecStopPost=/usr/local/bin/arbret-gatus-push probe_arbret-analytics -``` - -Two things to get right: - -- `EnvironmentFile=` puts the token in the unit's environment, **not** in the unit file, - so the unit file can stay world-readable while the env file is `0600`. -- **Tell us the timer's schedule.** We set the heartbeat to the interval plus slack: a - 5-minute job gets an 11-minute window, a daily job gets 26 hours. If the schedule - changes later, tell us, or Gatus will either alarm on a healthy job or sleep through a - dead one. - -As Ansible, this is two templated files, an `EnvironmentFile=` line and an -`ExecStopPost=` line. It's small enough to live in the role that deploys the analytics -timer, rather than becoming a role of its own. - -### 2.5 Staging (arbret-staging-box) - -arbret-staging-box is a VM in our inventory, so its liveness and disk checks already -exist. The same mapping applies with these keys: - -| Check | Key | Who | -|---|---|---| -| front door / health | `public_arbret-staging`, `public_arbret-staging-health` | us | -| units | `services_arbret-staging-local-arbret-server` (and `-worker`, `-postgresql`) | us | -| analytics | `probe_arbret-staging-analytics` | you, with the **staging** token | -| backup dump | `backup-dump_arbret-staging` | you, with the staging token. Only if staging should be backed up; tell us. | - -The staging vhost (`arbretstaging.contrapeso.xyz`) lived on the old prd-vipy and was not -in either repo. We're recreating it on our side. - ---- - -## 3. Backups - -### 3.1 The pipeline - -``` - 02:00 UTC prd-vipy your timer: pg_dump | age -r -> /opt/backups/arbret/arbret_.dump.age - your script: POST backup-dump_arbret (success/failure) - 04:00 UTC small-backups-box rsync backup-pull@prd-vipy:/opt/backups/arbret/ -> ~/backups/arbret/ (keeps 90 days) - 05:30 UTC small-backups-box check-backups.sh -> POST backup-store_arbret (then every 6 h) -``` - -- **The box only pulls.** Nothing on prd-vipy can reach into the backup box. The pull - logs in as `backup-pull`, an unprivileged account with a `restrict`ed key that can - read the backup directories and nothing else. -- **The source encrypts.** Only the age **public** key is on prd-vipy, so the host can't - decrypt its own backups. The private key lives offline with the infra owner. Plaintext - never touches disk or the network. -- **Retention is two-tier.** Keep about 7 days locally (the disk is shared). The box - keeps 90. - -Your existing dump role already produces the right **artefact**: a Postgres custom-format -dump, age-encrypted, named `arbret_YYYYmmdd_HHMMSS.dump.age`. The last pre-incident one -is `arbret_20261016_020001.dump.age`. Only its **location, ownership and reporting** -change. - -### 3.2 What changes from prd-arbret - -| | prd-arbret | prd-vipy | -|---|---|---| -| Directory | `/opt/arbret/backups/` | **`/opt/backups/arbret/`** | -| Account the box pulls as | `arbret` (your user, with our key authorised) | **`backup-pull`** (ours, already exists) | -| SSH key for the pull | you authorised it | **nothing to do.** Remove that task, and don't authorise anything for `arbret` or `backup-pull`. | -| Result reporting | Uptime Kuma push | Gatus push to `backup-dump_arbret` | - -### 3.3 The contract - -The checks on the box enforce these, and some failures are silent, so read every line. - -1. **Directory:** `/opt/backups/arbret/`, owner `root`, group `backup-pull`, mode `0750`. - Don't change `/opt/backups/` itself (`root:root 0755`, ours). The `backup-pull` group - exists before your playbook runs; if it doesn't, fail loudly rather than creating it. -2. **Filename:** `arbret_YYYYmmdd_HHMMSS.dump.age`, timestamp in the host's local time - (UTC). The prefix must be exactly `arbret_`, because the box's prune and freshness - checks glob on `arbret_*`, and it reads the **dump time from the filename**. -3. **Encryption:** `age -r ` in the pipe (§4). Nothing unencrypted is ever - written to disk. -4. **Atomicity:** write `.partial`, set ownership and mode, then `mv` it to the - final name. The pull skips `*.partial`. A half-written file under the final name - would be pulled and kept for 90 days as if it were a backup. -5. **File ownership:** `root:backup-pull`, mode `0640`, on both the `.partial` and the - final file. If the pull account can't read a file, **rsync fails for the whole - directory**, so one bad file blocks every later night. -6. **`set -euo pipefail`.** Without `pipefail`, a failing `pg_dump` piped into a - succeeding `age` exits 0 and produces a tiny, valid-looking, useless artefact. -7. **Timing:** finished before 04:00 UTC. 02:00 is good. -8. **Local prune:** delete `arbret_*.dump.age` older than ~7 days, plus stale - `arbret_*.partial` files at the start of each run. Put nothing else in the directory: - no subdirectories, logs or checksums. -9. **Report** the outcome to Gatus on every exit path (§3.5). -10. **No service stop is needed.** `pg_dump` takes a consistent snapshot online. Don't - stop `arbret-server` for it. - -### 3.4 Reference script - -This is the same shape as the script every other service on the host uses (our -`backup_source` role), adapted to `pg_dump`. Adjust the database name and the -`pg_dump` invocation to match your current role. - -`/usr/local/bin/arbret-backup.sh` (`root:root 0750`): - -```bash -#!/usr/bin/env bash -set -euo pipefail -umask 077 - -BACKUP_DIR=/opt/backups/arbret -RECIPIENT="age192wwdaseqej2ggwyp884gtm05c396anp7chr0vr8m47g50fahpyqr9fsza" -RETENTION_DAYS=7 -NAME=arbret -DB=arbret # <- your database name -ARTIFACT="${BACKUP_DIR}/${NAME}_$(date +%Y%m%d_%H%M%S).dump.age" - -PUSH_URL="${GATUS_PUSH_BASE:-}/backup-dump_arbret/external" -PUSH_TOKEN="${GATUS_PUSH_TOKEN:-}" - -report() { # $1=true|false $2=message - [ -n "${GATUS_PUSH_BASE:-}" ] || return 0 - local enc; enc=$(printf '%s' "$2" | sed 's/%/%25/g; s/ /%20/g; s/&/%26/g; s/+/%2B/g; s/#/%23/g') - curl -s -o /dev/null --max-time 15 --retry 2 --retry-delay 3 -X POST \ - -H "Authorization: Bearer ${PUSH_TOKEN}" "${PUSH_URL}?success=$1&error=${enc}" || true -} -on_exit() { # the ONE exit trap: bash keeps only one, a second replaces it - local rc=$? - if [ "$rc" -ne 0 ]; then report false "arbret dump exited ${rc}" - elif [ ! -s "$ARTIFACT" ]; then report false "arbret produced no artefact" - else report true "arbret $(du -h "$ARTIFACT" | cut -f1)"; fi -} -trap on_exit EXIT - -command -v age >/dev/null || { echo "age not installed" >&2; exit 1; } -getent group backup-pull >/dev/null || { echo "group backup-pull missing - personal_infra has not provisioned this host" >&2; exit 1; } - -install -d -o root -g backup-pull -m 0750 "$BACKUP_DIR" -rm -f "${BACKUP_DIR}/${NAME}_"*.partial - -runuser -u postgres -- pg_dump -Fc "$DB" | age -r "$RECIPIENT" -o "${ARTIFACT}.partial" -chown root:backup-pull "${ARTIFACT}.partial"; chmod 0640 "${ARTIFACT}.partial" -mv "${ARTIFACT}.partial" "$ARTIFACT" - -find "$BACKUP_DIR" -maxdepth 1 -type f -name "${NAME}_*.dump.age" -mtime +"$RETENTION_DAYS" -delete -echo "wrote $ARTIFACT" -``` - -Units: - -```ini -# /etc/systemd/system/arbret-backup.service (0644; the token is in the env file, not here) -[Unit] -Description=arbret database backup -After=postgresql.service - -[Service] -Type=oneshot -EnvironmentFile=/etc/arbret/gatus.env -ExecStart=/usr/local/bin/arbret-backup.sh - -# /etc/systemd/system/arbret-backup.timer -[Unit] -Description=arbret database backup, nightly - -[Timer] -OnCalendar=*-*-* 02:00:00 -Persistent=true - -[Install] -WantedBy=timers.target -``` - -`age` and `curl` come from Debian (`apt install age curl`). Installing them from your -playbook is fine, since both repos asking for the same package doesn't conflict. - -### 3.5 Reporting the backup - -The script above already reports, on every exit path, to `backup-dump_arbret`, using the -same `/etc/arbret/gatus.env` as the analytics push. Its heartbeat is 26 hours: one -missed night alarms. The box-side check (`backup-store_arbret`) independently notices -within 6 hours of the next pull if the artefact stops arriving or stops being fresh. -So a broken dump and a broken pull show up as **different** red lights. - -### 3.6 Verify after your first deploy - -```bash -sudo systemctl start arbret-backup.service -sudo journalctl -u arbret-backup.service -n 20 --no-pager -ls -l /opt/backups/arbret/ # root backup-pull, -rw-r----- -sudo -u backup-pull head -c 30 /opt/backups/arbret/arbret_*.dump.age # must print "age-encryption.org/v1" -systemctl list-timers arbret-backup.timer -``` - -Then check that `backup-dump_arbret` is green on the status page, and tell us. We'll -trigger a pull rather than wait for 04:00, and confirm `backup-store_arbret`. - -### 3.7 Restoring - -Only the infra owner can decrypt: the age identity is offline. They'll hand you a -decrypted dump over a secure channel. The latest pre-incident one is -`arbret_20261016_020001.dump.age` (31.9 MB, sha256 `2bb771957513654cb907e5d63153e7714d54c7d5695d7caf09a05f948dcf515d`). -Anything written to arbret after 2026-10-16 02:00 UTC is not in any backup. - -For a custom-format dump: - -```bash -sudo -u postgres createdb -O arbret arbret # or let --create do it -sudo -u postgres pg_restore -d arbret --no-owner --role=arbret -j 4 arbret_20261016_020001.dump -``` - -Create the roles the app expects first. `pg_dump -Fc` doesn't include them. Then shred -the plaintext dump: `shred -u arbret_20261016_020001.dump`. - ---- - -## 4. What you'll receive from us - -| Item | Secret | Used for | -|---|---|---| -| Gatus push token, **prod** | yes | `/etc/arbret/gatus.env` on prd-vipy. Keep it in your vault. | -| Gatus push token, **staging** | yes | the same file on arbret-staging-box, if you do the staging pushes | -| age recipient `age192wwdaseqej2ggwyp884gtm05c396anp7chr0vr8m47g50fahpyqr9fsza` | **no**, it's a public key | the backup script. Safe to commit in plain text. | -| SSH access to prd-vipy for your playbooks | access | being arranged separately | -| The decrypted 2026-10-16 dump | yes, it's the full prod DB | restore, handed over once | - -You need **no** SSH key for the backup pull, and nothing from the old Uptime Kuma setup. - -## 5. What we need from you - -1. The local port arbret listens on (for the `arbret.com` vhost), and whether - `www.arbret.com` should be served or redirected. -2. The exact `/api/health` response when healthy, and confirmation that it returns a - non-200 when the DB is down. -3. The analytics timer's schedule (sets the heartbeat). -4. Whether port 4117 over the tailnet is still needed. -5. The final unit names, if not `arbret-server`, `arbret-worker`, `postgresql`. -6. Whether staging should be backed up, and whether staging pushes are in scope. -7. Who besides the infra owner should get Signal alerts for arbret. - -Once we have 1 to 3, we register the endpoints, the vhost and the pull source. Then you -can deploy and see green on the first run. diff --git a/requirements.txt b/requirements.txt index 653523e..972dc4f 100644 --- a/requirements.txt +++ b/requirements.txt @@ -8,7 +8,4 @@ packaging==25.0 pycparser==2.22 PyYAML==6.0.2 resolvelib==1.0.1 -# Only needed by the deprecated Uptime Kuma monitoring blocks, which are kept for -# reference but never run (uptime_kuma_enabled: false). Drop this once they are -# rewired to a replacement. See archive/uptime_kuma/. uptime-kuma-api>=1.2.1 diff --git a/scripts/add-music.sh b/scripts/add-music.sh deleted file mode 100755 index 5bb7b89..0000000 --- a/scripts/add-music.sh +++ /dev/null @@ -1,189 +0,0 @@ -#!/usr/bin/env bash -# -# Add music to music-box: push it, tag it with beets, let Navidrome see it. -# -# WHY THIS SCRIPT EXISTS -# ---------------------- -# The three steps are simple individually and easy to get subtly wrong together: -# -# 1. rsync to /srv/incoming, NOT to /srv/music. Navidrome reads /srv/music and -# beets is the only thing that should ever write there. Dropping files -# straight into the library skips tagging entirely, which is the whole -# point of having beets. -# 2. `beet import` is INTERACTIVE, so the ssh needs a TTY (-t). Without it the -# prompt never renders and the import appears to hang. -# 3. beets must run as counterweight. Under sudo it finds no config (the -# config lives in ~counterweight), silently falls back to its defaults and -# writes to a SECOND library database in root's home - so you end up with -# files in the wrong place and two competing indexes. -# -# WHAT IT DOES NOT DO -# It does not touch your local files. rsync copies; the `move: yes` in the -# beets config only moves things around ON THE BOX, from /srv/incoming into -# /srv/music. Your originals on this machine are never modified or deleted. -# -# Usage: scripts/add-music.sh [path] (prompts for the path if omitted) -# Needs: rsync and ssh here, tailnet up (music-box resolves via MagicDNS) -set -euo pipefail - -MUSIC_HOST="${MUSIC_HOST:-music-box}" -REMOTE_USER="${MUSIC_USER:-counterweight}" -INCOMING="/srv/incoming" -# Pre-filled into the prompt, editable in place. $HOME rather than a literal -# /home/counterweight so the script still behaves if it is ever run as anyone -# else; for the usual user it resolves to exactly /home/counterweight/music. -DEFAULT_SRC="${MUSIC_LOCAL_DIR:-$HOME/music}" -SSH_KEY="${MUSIC_SSH_KEY:-$HOME/.ssh/counterganzua}" -TARGET="${REMOTE_USER}@${MUSIC_HOST}" - -# Colours, but only when we are actually on a terminal. -if [ -t 1 ]; then - B=$'\033[1m'; DIM=$'\033[2m'; R=$'\033[31m'; G=$'\033[32m'; Y=$'\033[33m'; N=$'\033[0m' -else - B=""; DIM=""; R=""; G=""; Y=""; N="" -fi - -say() { printf '%s==>%s %s\n' "$B" "$N" "$*"; } -warn() { printf '%s==>%s %s\n' "$Y" "$N" "$*" >&2; } -die() { printf '%sERROR:%s %s\n' "$R" "$N" "$*" >&2; exit 1; } - -# -n is load-bearing: without it ssh reads OUR stdin and swallows the answers -# meant for the read prompts below, so "Continue? [y/N]" gets EOF instead of -# your keypress. Under `set -e` that aborts the script with no message at all. -ssh_run() { ssh -n -i "$SSH_KEY" -o ConnectTimeout=15 "$TARGET" "$@"; } -# -t for anything that talks to a human (beets prompts, sudo password). No -n -# here - these deliberately DO want your terminal. -ssh_tty() { ssh -t -i "$SSH_KEY" -o ConnectTimeout=15 "$TARGET" "$@"; } - -# --- 0. Preflight ------------------------------------------------------------ -command -v rsync >/dev/null || die "rsync is not installed here." -[ -f "$SSH_KEY" ] || die "SSH key not found: $SSH_KEY (override with MUSIC_SSH_KEY=)" - -say "Checking ${MUSIC_HOST} is reachable..." -if ! ssh_run -o BatchMode=yes true 2>/dev/null; then - die "Cannot reach ${TARGET}. - Is the tailnet up? tailscale status | grep ${MUSIC_HOST} - MagicDNS resolving? getent hosts ${MUSIC_HOST}" -fi - -# --- 1. Where is the music? -------------------------------------------------- -SRC="${1:-}" -if [ -z "$SRC" ]; then - printf '\n%sPath to the album or folder to add%s %s(edit or tab-complete)%s\n' "$B" "$N" "$DIM" "$N" - # -e gives readline (tab completion, arrow keys); -i pre-fills the buffer - # with the default so it is already typed and editable in place. - # -i is ignored when stdin is not a terminal, hence the ${SRC:-} fallback: - # piping into this script, or just pressing Enter, still yields the default. - read -e -i "$DEFAULT_SRC" -r -p "> " SRC || true - SRC="${SRC:-$DEFAULT_SRC}" -fi - -# People paste paths with surrounding quotes (drag-and-drop does this) and with -# a trailing slash. Both change rsync's meaning, so normalise before using it. -SRC="${SRC%\"}"; SRC="${SRC#\"}" -SRC="${SRC%\'}"; SRC="${SRC#\'}" -SRC="${SRC/#\~/$HOME}" -while [ "${SRC}" != "/" ] && [ "${SRC: -1}" = "/" ]; do SRC="${SRC%/}"; done - -[ -n "$SRC" ] || die "No path given." -[ "$SRC" != "/" ] && [ "$SRC" != "$HOME" ] || die "Refusing to sync '$SRC' - that is almost certainly not what you meant." -[ -e "$SRC" ] || die "Not found: $SRC" -[ -r "$SRC" ] || die "Not readable: $SRC" - -BASENAME="$(basename -- "$SRC")" -if [ -d "$SRC" ]; then - COUNT=$(find "$SRC" -type f | wc -l | tr -d ' ') - SIZE=$(du -sh "$SRC" 2>/dev/null | cut -f1) - [ "$COUNT" -gt 0 ] || die "$SRC contains no files." -else - COUNT=1 - SIZE=$(du -h "$SRC" 2>/dev/null | cut -f1) -fi - -# --- 2. Does it fit? --------------------------------------------------------- -# /srv/incoming and /srv/music share one filesystem, so the beets move is a -# rename and costs nothing extra - but the transfer itself still needs room. -AVAIL=$(ssh_run "df -Pk ${INCOMING} | awk 'NR==2 {print \$4}'" 2>/dev/null || echo 0) -NEED=$(du -sk "$SRC" 2>/dev/null | cut -f1) -if [ "${AVAIL:-0}" -gt 0 ] && [ "${NEED:-0}" -gt "${AVAIL}" ]; then - die "Not enough space on ${MUSIC_HOST}: need $((NEED/1024)) MB, have $((AVAIL/1024)) MB. - The library shares the 50 GB root disk - see roles/navidrome/README.md." -fi - -printf '\n' -say "About to add:" -printf ' source %s\n' "$SRC" -printf ' files %s (%s)\n' "$COUNT" "${SIZE:-?}" -printf ' lands in %s:%s/%s\n' "$MUSIC_HOST" "$INCOMING" "$BASENAME" -printf ' free %s MB on the box\n\n' "$((${AVAIL:-0}/1024))" -printf '%sYour local files are only read, never moved or deleted.%s\n\n' "$DIM" "$N" - -read -r -p "Continue? [y/N] " REPLY -case "$REPLY" in [yY]*) ;; *) say "Nothing done."; exit 0 ;; esac - -# --- 3. Push ----------------------------------------------------------------- -printf '\n' -say "Pushing to ${MUSIC_HOST}:${INCOMING}/ ..." -# No trailing slash on "$SRC": that copies the DIRECTORY, not its contents, so -# the album arrives as its own folder and beets sees one album per directory. -# --chmod is not cosmetic. `rsync -a` preserves the SOURCE mode, and beets -# moves files into the library with rename(2), which keeps their group and -# mode. A track that leaves this machine mode 0600 therefore arrives in -# /srv/music mode 0600 owned by the curator's group - and Navidrome cannot read -# it. It does not error; the track is simply absent from the library. The beets -# `permissions` plugin also normalises this on the box; this is the same fix -# applied at the door so /srv/incoming is sane too. -rsync -av --info=progress2 --human-readable --chmod=D755,F644 \ - -e "ssh -i $SSH_KEY -o ConnectTimeout=15" \ - "$SRC" "${TARGET}:${INCOMING}/" - -# --- 4. Tag ------------------------------------------------------------------ -printf '\n' -say "Running beets. This is interactive - see the key guide below." -printf '%s A apply M more candidates E enter search I paste MB id%s\n' "$DIM" "$N" -printf '%s U use tags as-is S skip T as tracks G group albums%s\n\n' "$DIM" "$N" - -# As counterweight (NOT sudo) so it finds ~/.config/beets/config.yaml. -# `|| true` because beets exits non-zero when you skip or abort an album, and -# that is a normal outcome, not a script failure. -ssh_tty "beet import ${INCOMING}/$(printf '%q' "$BASENAME")" || true - -# --- 5. What is left behind? ------------------------------------------------- -printf '\n' -LEFT=$(ssh_run "find ${INCOMING} -type f 2>/dev/null | wc -l" | tr -d ' \r') -if [ "${LEFT:-0}" -gt 0 ]; then - warn "${LEFT} file(s) still in ${INCOMING} - skipped, or beets could not match them." - warn "Re-run later with: ssh ${MUSIC_HOST} 'beet import ${INCOMING}'" - warn "Details: ssh ${MUSIC_HOST} 'tail /var/lib/beets/import.log'" -else - printf '%s==>%s %sEverything imported; %s is empty.%s\n' "$B" "$N" "$G" "$INCOMING" "$N" -fi - -# --- 6. Optional post-processing --------------------------------------------- -# replaygain and lyrics are auto:no in the beets config on purpose - on a single -# core they turn a 2 minute import into a 20 minute one. Offered here so they do -# not get forgotten entirely, but off by default. -printf '\n' -read -r -p "Run replaygain now? (slow on 1 core) [y/N] " REPLY -case "$REPLY" in [yY]*) say "Running beet replaygain..."; ssh_tty "beet replaygain" || warn "replaygain reported a problem." ;; esac - -# --- 7. Let Navidrome see it ------------------------------------------------- -printf '\n' -say "Restarting Navidrome so it picks up the new files..." -ssh_tty "sudo systemctl restart navidrome" - -say "Waiting for it to come back..." -for i in $(seq 1 20); do - if ssh_run "curl -sf -m 5 -o /dev/null http://127.0.0.1:4533/app/" 2>/dev/null; then - printf '%s==>%s %sNavidrome is up.%s\n' "$B" "$N" "$G" "$N" - break - fi - [ "$i" -eq 20 ] && warn "Navidrome did not answer within ~60s. Check: ssh ${MUSIC_HOST} 'systemctl status navidrome'" - sleep 3 -done - -# --- 8. Summary -------------------------------------------------------------- -printf '\n' -say "Library now holds:" -ssh_run "beet stats" 2>/dev/null | sed 's/^/ /' || warn "could not read beet stats" -printf '\n%s==>%s %sDone.%s https://music.contrapeso.xyz\n' "$B" "$N" "$G" "$N" diff --git a/scripts/build-xmrblocks.sh b/scripts/build-xmrblocks.sh deleted file mode 100755 index 44e3ede..0000000 --- a/scripts/build-xmrblocks.sh +++ /dev/null @@ -1,180 +0,0 @@ -#!/usr/bin/env bash -# -# Build xmrblocks (the Onion Monero Blockchain Explorer) reproducibly. -# -# WHY THIS SCRIPT EXISTS -# ---------------------- -# xmrblocks is not a client of monerod. It links 21 static libraries out of a -# compiled monero source tree and opens the LMDB directly, so building it means -# first compiling monero itself. Measured on Debian 13: a single cc1plus peaks -# at ~3.1 GB and a -j4 build peaks at ~5.5 GB. monerod-box has 2 GB, so it -# cannot build this at any level of parallelism - the binary has to be built -# elsewhere and shipped. -# -# Shipping a binary that only one machine can produce is how estates end up -# with artefacts nobody can rebuild. This script is the answer to that: it runs -# the whole build inside a debian:13 container, so any machine with Docker and -# ~6 GB of RAM produces the same binary. Nothing is installed on the host. -# -# WHAT IT GUARANTEES -# - the monero source is the GPG-signed release tarball, verified against -# binaryFate's key (the same chain roles/monero uses for the daemon) -# - the explorer is pinned to an exact commit -# - the one upstream patch is applied from an anchor, and the script FAILS -# if the anchor is missing rather than silently skipping it -# -# WHAT IT IS NOT -# Not bit-for-bit reproducible. Debian's toolchain, timestamps and build -# paths all leak in. It is repeatable, not deterministic; the MANIFEST -# records what went in so two builds can be compared. -# -# Usage: scripts/build-xmrblocks.sh [output-dir] -# Needs: docker, ~6 GB RAM available to it, ~2 GB disk -set -euo pipefail - -# --- Pins. Change these together and rebuild. -------------------------------- -MONERO_VERSION="0.18.5.1" -MONERO_SRC_TARBALL="monero-source-v${MONERO_VERSION}.tar.bz2" -MONERO_SRC_URL="https://downloads.getmonero.org/cli/${MONERO_SRC_TARBALL}" -# From the GPG-signed hashes.txt. Verified by the container, not trusted from here. -MONERO_SRC_SHA256="9ec6ed0fd37db9d81cf7738a5f0536cf9aec6ed8ef8fd48649a59a6aaf20de3d" -BINARYFATE_FPR="81AC591FE9C4B65C5806AFC3F0AF4D462A0BDF92" - -EXPLORER_REPO="https://github.com/moneroexamples/onion-monero-blockchain-explorer.git" -EXPLORER_COMMIT="29593b709edeb23daba035009ccd134a5862543a" - -BUILD_IMAGE="debian:13" # must match the TARGET host's Debian release -JOBS="${JOBS:-4}" - -REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" -SIGNING_KEY="${REPO_ROOT}/ansible/roles/monero/files/binaryfate.asc" -OUT_DIR="${1:-$HOME/personal_infra_artifacts/xmrblocks}" - -command -v docker >/dev/null || { echo "docker is required" >&2; exit 1; } -[ -f "$SIGNING_KEY" ] || { echo "missing signing key: $SIGNING_KEY" >&2; exit 1; } - -WORK="$(mktemp -d)" -trap 'rm -rf "$WORK"' EXIT -cp "$SIGNING_KEY" "$WORK/binaryfate.asc" -mkdir -p "$OUT_DIR" - -echo "==> building xmrblocks" -echo " monero ${MONERO_VERSION} (signed source tarball)" -echo " explorer ${EXPLORER_COMMIT}" -echo " image ${BUILD_IMAGE}, -j${JOBS}" -echo " output ${OUT_DIR}" - -cat > "$WORK/build.sh" </dev/null - -cd /build - -echo "--> verifying the monero source tarball" -export GNUPGHOME=/build/gnupg; mkdir -p \$GNUPGHOME; chmod 700 \$GNUPGHOME -gpg --batch --import /build/binaryfate.asc 2>/dev/null -gpg --batch --with-colons --fingerprint | awk -F: '/^fpr:/{print \$10}' \\ - | grep -qx "${BINARYFATE_FPR}" || { echo "signing key is not binaryFate's"; exit 1; } -curl -sSL -o hashes.txt https://www.getmonero.org/downloads/hashes.txt -gpg --batch --status-fd 2 --output hashes-verified.txt --decrypt hashes.txt 2>gpg.err -grep -q "VALIDSIG.*${BINARYFATE_FPR}" gpg.err || { echo "hashes.txt did not verify"; exit 1; } -curl -sSL -o "${MONERO_SRC_TARBALL}" "${MONERO_SRC_URL}" -ACTUAL=\$(sha256sum "${MONERO_SRC_TARBALL}" | cut -d' ' -f1) -[ "\$ACTUAL" = "${MONERO_SRC_SHA256}" ] || { echo "tarball sha mismatch"; exit 1; } -grep -Fxq "\$ACTUAL ${MONERO_SRC_TARBALL}" hashes-verified.txt \\ - || { echo "tarball is not the one binaryFate signed"; exit 1; } -echo " source verified against the signed hashes" - -tar xjf "${MONERO_SRC_TARBALL}" -SRC=/build/monero-source-v${MONERO_VERSION} - -echo "--> building the monero libraries xmrblocks links against" -# Only the libraries. Upstream's Dockerfile runs a full \`make\`, which also -# builds monerod, both wallets and the test suite - none of which are linked -# into xmrblocks. USE_DEVICE_TREZOR=OFF drops the generated protobuf units too; -# an explorer has no use for hardware wallets. -mkdir -p \$SRC/build/release && cd \$SRC/build/release -cmake -D CMAKE_BUILD_TYPE=Release -D BUILD_TESTS=OFF -D USE_DEVICE_TREZOR=OFF ../.. >/dev/null -cmake --build . -j${JOBS} --target \\ - common blocks cryptonote_basic cryptonote_core multisig cryptonote_protocol \\ - daemonizer mnemonics epee lmdb device wallet-crypto blockchain_db ringct \\ - wallet cncrypto easylogging version checkpoints randomx hardforks >/dev/null - -echo "--> fetching the explorer at ${EXPLORER_COMMIT}" -cd /build -git clone -q "${EXPLORER_REPO}" explorer -cd explorer && git checkout -q "${EXPLORER_COMMIT}" - -# monero v0.18.5.0 changed get_pruned_transaction_hash from returning the hash -# to returning bool with an out-param. The explorer still calls the old form, -# so it does not compile against the daemon we actually run. Anchored on the -# exact line: if upstream fixes this, the anchor vanishes and we stop loudly -# rather than building something unexpected. -OLD=' txd.hash = get_pruned_transaction_hash(tx, tx.prunable_hash);' -NEW=' get_pruned_transaction_hash(tx, tx.prunable_hash, txd.hash);' -if grep -qF "\$OLD" src/page.h; then - sed -i "s|\$OLD|\$NEW|" src/page.h - echo " applied get_pruned_transaction_hash patch" -elif grep -qF "\$NEW" src/page.h; then - echo " upstream already uses the new signature - patch not needed" -else - echo "PATCH ANCHOR NOT FOUND in src/page.h - upstream changed; review before building" >&2 - exit 1 -fi - -echo "--> building xmrblocks" -mkdir -p build && cd build -# Xmr_MINIUPNPC_LIBRARY: FindMonero.cmake searches for miniupnpc *inside* the -# monero build tree with NO_DEFAULT_PATH, but monero no longer vendors it. -# Point it at the system library instead of patching their cmake. -cmake -D MONERO_DIR=\$SRC \\ - -D Xmr_MINIUPNPC_LIBRARY=/usr/lib/x86_64-linux-gnu/libminiupnpc.a .. >/dev/null -cmake --build . -j${JOBS} >/dev/null -[ -x xmrblocks ] || { echo "no xmrblocks produced"; exit 1; } -./xmrblocks --help >/dev/null 2>&1 || true - -echo "--> collecting" -install -m 0755 xmrblocks /out/xmrblocks -rm -rf /out/templates && cp -r templates /out/templates -chmod -R a+rX /out/templates - -{ - echo "artifact: xmrblocks" - echo "built: \$(date -u +%Y-%m-%dT%H:%M:%SZ)" - echo "build-image: ${BUILD_IMAGE}" - echo "monero-version: ${MONERO_VERSION}" - echo "monero-src-sha: ${MONERO_SRC_SHA256}" - echo "explorer-commit: ${EXPLORER_COMMIT}" - echo "cmake-flags: USE_DEVICE_TREZOR=OFF BUILD_TESTS=OFF CMAKE_BUILD_TYPE=Release" - echo "patches: get_pruned_transaction_hash -> out-param form (monero >= 0.18.5.0)" - echo "xmrblocks-sha256: \$(sha256sum /out/xmrblocks | cut -d' ' -f1)" - echo "templates-sha256: \$(find /out/templates -type f | sort | xargs sha256sum | sha256sum | cut -d' ' -f1)" - echo "needed-libs:" - objdump -p /out/xmrblocks 2>/dev/null | awk '/NEEDED/{print " " \$2}' | sort -} > /out/MANIFEST -INNER -chmod +x "$WORK/build.sh" - -docker run --rm \ - -v "$WORK":/build \ - -v "$OUT_DIR":/out \ - "$BUILD_IMAGE" bash /build/build.sh - -echo -echo "==> done" -sed -n '1,12p' "$OUT_DIR/MANIFEST" -echo -echo " pin xmrblocks_sha256 in ansible/roles/xmrblocks/defaults/main.yml to:" -grep '^xmrblocks-sha256:' "$OUT_DIR/MANIFEST" | awk '{print " " $2}' diff --git a/tofu/nodito/.terraform.lock.hcl b/tofu/nodito/.terraform.lock.hcl deleted file mode 100644 index fadff38..0000000 --- a/tofu/nodito/.terraform.lock.hcl +++ /dev/null @@ -1,24 +0,0 @@ -# This file is maintained automatically by "tofu init". -# Manual edits may be lost in future updates. - -provider "registry.opentofu.org/telmate/proxmox" { - version = "3.0.2-rc05" - constraints = "3.0.2-rc05" - hashes = [ - "h1:QfHovHn8h9uJXdJ+urOuiD7R46OXmdZQiRcCBaV6AD4=", - "zh:042d748367f33aaf440698644be4f2a2875f9db31915c1ef84616f176fc6174f", - "zh:1488781da1920d60d933c8ce926c34b5e989ffae58e3fbe437973d2b1d2faafc", - "zh:283dd6f74627f1d1d75d616b31f8ced3f97fd5277a07c9535e85cfa765d7a321", - "zh:378f1c2da21aeea083ac2e632db274a02c7a01e2486a40d3c813d05a21142db3", - "zh:38d63d0961f8c32273392caaace30f50cff8ab06e5dda17f67a8827ebffeba98", - "zh:52159782df101ec98f20faff81e8f2d9d92cb4ec903314fcddcc57ec16cdaacb", - "zh:6ca47b90c66b1d2706cb3cbb05da8b3f90a202c4865010202b2962e2b64d217e", - "zh:6e7b85cb2380e4dc0be694dd0e4a24927f7f66df41960eca3cfe907443d4f0b9", - "zh:758775f733673ab5c196db6a33648458037746f94d4bef7ce148cb01474efe2d", - "zh:7c31a3ca6d52db39da2bdd60be37af71d59d808fc206de50fe661535ea436da3", - "zh:af16984350a2f4d77c21f66a479007801e2527543310567c99cd82eb421e249e", - "zh:c1f965d3f96cf3f87af2c12ab9d4bde42f8ef660f8dc34ba3cfc9b20435a7269", - "zh:c2b9022a31103919a5ffbac6ee8d7feb6c4f5f580c1766f769569c2e8e4ce7f1", - "zh:e90162c42f1237323291e3d0de0c62701b3f89350fae18246da06702f41a6123", - ] -} diff --git a/tofu/nodito/README.md b/tofu/nodito/README.md index bff6585..3a0b18f 100644 --- a/tofu/nodito/README.md +++ b/tofu/nodito/README.md @@ -27,17 +27,16 @@ This directory lets you declare VMs on the `nodito` Proxmox node and apply with - The Ansible template exists: `debian-13-cloud-init` (VMID 9001 by default). ### Provider Auth -Credentials are the only thing not in git. Copy `terraform.tfvars.example` to -`terraform.tfvars` (gitignored) and set: +Create a `terraform.tfvars` (copy from `terraform.tfvars.example`) and set: - `proxmox_api_url` (e.g. `https://nodito:8006/api2/json`) - `proxmox_api_token_id` (e.g. `root@pam!tofu`) - `proxmox_api_token_secret` +- `ssh_authorized_keys` (your public key content) -Alternatively, export them as `TF_VAR_proxmox_api_token_secret` etc. +Alternatively, you can export env vars and reference them in a tfvars file. ### Declare VMs -VMs are declared in `vms.auto.tfvars`, which is committed. `*.auto.tfvars` is -loaded automatically, so it needs no `-var-file`. Example entry: +Edit `terraform.tfvars` and fill the `vms` map. Example entry: ``` vms = { web1 = { @@ -55,24 +54,15 @@ All VM disks are created on `zfs_storage_name` (defaults to `proxmox-tank-1`). N ### Usage ``` tofu init -tofu plan -tofu apply +tofu plan -var-file=terraform.tfvars +tofu apply -var-file=terraform.tfvars ``` -`terraform.tfvars` and `vms.auto.tfvars` are both auto-loaded. > VMs are created once and then protected: the module sets `lifecycle.prevent_destroy = true` and ignores subsequent config changes. After the initial apply, manage day‑2 changes directly in Proxmox (or remove the lifecycle block if you need OpenTofu to own ongoing updates). ### Notes - Clones are full clones by default (`full_clone = true`). - Cloud-init injects `cloud_init_user` and `ssh_authorized_keys`. -- `.terraform.lock.hcl` is committed: it pins the provider hashes, which matters - because `versions.tf` tracks a release candidate (`3.0.2-rc05`). -- State is local (`terraform.tfstate`, gitignored) and has no remote backend, so - it exists only on the machine that last ran `tofu apply`. -- The map is not a complete inventory of nodito: state also manages - `bastion-box` (1100) and `nonkeiwaisi-box` (3300), which are not declared in - `vms.auto.tfvars`. `tofu plan` is clean today, but relaxing the `lifecycle` - block without first declaring them would put them up for destruction. - Disks use `scsi0` on ZFS with `discard` enabled. diff --git a/tofu/nodito/terraform.tfvars.example b/tofu/nodito/terraform.tfvars.example index 37b6d7a..cc88b3f 100644 --- a/tofu/nodito/terraform.tfvars.example +++ b/tofu/nodito/terraform.tfvars.example @@ -1,7 +1,35 @@ -# Provider credentials. Copy to terraform.tfvars (gitignored) and fill in. -# Everything else — the node, storage, template, SSH key and the vms map — -# is committed in vms.auto.tfvars and needs no local copy. +proxmox_api_url = "https://nodito:8006/api2/json" +proxmox_api_token_id = "root@pam!tofu" +proxmox_api_token_secret = "REPLACE_ME" + +proxmox_node = "nodito" +zfs_storage_name = "proxmox-tank-1" +template_name = "debian-13-cloud-init" +cloud_init_user = "counterweight" + +# paste your ~/.ssh/id_ed25519.pub or similar +ssh_authorized_keys = <