prd-arbret was folded into vipy. arbret itself (app, worker, Postgres, timers,
its backup dump) is deployed by its owner's own repo; this repo owns the
shared host's side of it. arbret_handoff.md documents the split, the Gatus
push API and the backup contract.
- services/arbret: arbret.com -> localhost:3100, www redirects to the apex,
port 4117 allowed on tailscale0 only, /opt/backups/arbret pre-created to
the contract so the pull job is green before the first dump.
- headscale ACL: framie may reach prd-vipy:4117.
- vipy: arbret-server, arbret-worker and postgresql in the unit checks.
- 402: DNS and HTTP checks for arbret.com, www, and /api/health with
[BODY].status == ok. Entries may now name an fqdn outside root_domain and
add conditions.
- 403: probes arbret-analytics (hourly, 130m window) and arbret-beancount
(daily, 26h), pushing with the dedicated arbret token.
- backups: pull from backup-pull@prd-vipy:/opt/backups/arbret/, and a
backup-dump_arbret endpoint.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Caddy proxied /admin* to Headplane behind basic auth, but nothing in this repo
ever installed Headplane, so the route led nowhere. The vhost is now a plain
reverse proxy: the headscale API authenticates every request itself.
The headscale_ui_* vault entries are now unused and can be removed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
All four droplets were deleted by mistake on 2026-10-17 and rebuilt as two:
Gatus and signal-api now share spacey with headscale, and arbret moves onto
vipy. The old IPs were recycled to strangers within a day.
- inventory: [observability] is spacey. Left as it was, the bare name
`monitoring` resolved through ~/.ssh/config to an IP that now belongs to
someone else. [arbret]/prd-arbret is gone.
- inventory: lan_ip fallbacks corrected from the Proxmox guest agent -
memos-box is .130 (was .145) and arbret-staging-box .145 (was .147).
- spacey: gatus_port 8081. headscale already holds 0.0.0.0:8080, and the two
never shared a host before.
- 402: status.<domain> is expected to resolve to spacey.
- headscale ACL: prd-arbret, prd-monitoring and stalwart-box removed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Jar Lid Archive (Django + Postgres 18 + SeaweedFS) on jar-lid-archive-staging-box
(nodito VM 3300, 1 core / 1 GB / 10 GB), public at jars-staging.contrapeso.xyz
through vipy's Caddy. New role roles/jar_lid_archive; per-host values in
host_vars, so a production host is a new [jar_lid_archive] entry, not a new
playbook.
── Containers, following :master ──────────────────────────────────────────
The same three services as the app repo's compose.yaml, so staging rehearses
what dev and CI run. Every playbook run pulls :master and `up -d --wait`s;
compose recreates web only when the digest moved. --wait gates on web's
/health/, which checks the database and the bucket, so a green run means the
app serves. Upgrading is re-running the playbook.
── web is on the host network ─────────────────────────────────────────────
Docker's published ports are DNATed before UFW sees them, so `8000:8000` would
answer on the LAN whatever UFW says, and binding to the tailnet IP races
tailscaled at boot. On the host network gunicorn's port is an ordinary one: UFW
default-deny covers the LAN, tailscaled's ts-input accepts tailscale0 ahead of
UFW (checked on mempool-box), and the headscale ACL grants tag:prd-vipy exactly
:8000. db and storage publish on 127.0.0.1 only. Verified: LAN times out, vipy
gets 200.
That matters for HTTPS: the app trusts X-Forwarded-Proto
(DJANGO_BEHIND_TLS_PROXY=1), which is only safe while Caddy is the sole client.
Login POST through the public URL passes CSRF; cookies are Secure.
── Secrets ────────────────────────────────────────────────────────────────
Generated straight into host_vars/jar_lid_staging_local/vault.yml, never
printed: Django key, Postgres and S3 secrets, and seed passwords for the
fixture's alice/bob/admin usernames - never the fixture's published passwords,
since the site is public. The group vault gains one line, this host's Gatus push
token; decrypted, it differs from before by exactly that line.
── Backups, and the outage the first one caused ───────────────────────────
One uncompressed tar (pg_dump -Fc + the SeaweedFS data dir), age-encrypted on
the box at 02:20, pulled by small-backups-box; 3 days at both ends. Verified end
to end, including decrypting a pulled artefact on lapy.
The first test run stopped web AND storage and left the app down 26 minutes:
SeaweedFS outlasted its 150s healthcheck budget on this disk, compose marked it
unhealthy, and web's depends_on never started it. Now only web is stopped - it
is the only writer to both stores, so they are still captured at one instant -
and the healthchecks have start_periods sized for the disk.
Also inherited from the app's compose.yaml: storage writes /tmp/s3.json and
chowns it to seaweed, so on a RESTART of the same container root cannot
re-create it (sticky /tmp + protected_regular): "Permission denied". Fixed here
with rm -f first; the app repo needs the same line.
── The disk is the problem ────────────────────────────────────────────────
The VM disk is on proxmox-tank-1, the HDD mirror, which knots-box's random reads
(~6 MB/s) keep near saturation. Inside this VM: ~96% "full" I/O pressure while
idle, ~5 min to start web, sudo occasionally over 12s. Moving it to local-zfs
(NVMe) is the fix; noted in the README, not done.
── Also ───────────────────────────────────────────────────────────────────
- lan_ip read from the Proxmox guest agent; host key checked against the one
the agent reads off the disk before it went into known_hosts.
- Monitoring: liveness + disk (400), probe_jar-lid-archive-staging (403), public
/health/ + DNS (402), backup-dump/backup-store.
- tofu: the VM definition.
- Deployed v2026.10.03.1.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Navidrome 0.64.0 on music-box (nodito VM 8200, 1 core / 1 GB / 50 GB), public
at music.contrapeso.xyz. A single Go binary with a SQLite database; the only
external dependency is ffmpeg for transcoding. Plus beets, the CLI that fixes
tags before Navidrome ever indexes them.
── Exposed without basic auth, on purpose ─────────────────────────────────
Navidrome speaks two protocols on one port: its web UI, and the Subsonic API
(/rest/*) that every mobile client actually uses. A Subsonic client
authenticates with ?u=&t=md5(password+salt)&s= in the query string and has
nowhere to carry a second credential, so Basic Auth in front of the site is
invisible to most of them. The choice is a proxy credential or a working phone.
Same call this estate already made for monerod's restricted RPC, for the same
reason: proxied bare so Feather can reach it, with the service's own auth doing
the authenticating.
The cost is stated rather than glossed: the login page and the whole /rest/*
surface are reachable by anyone who finds the hostname, and Navidrome has no
rate limiting and no 2FA.
A fail2ban jail on vipy reads a per-vhost Caddy access log and catches WEB UI
brute force, where a bad login returns 401. It very likely does NOT catch
Subsonic brute force, because that protocol answers a bad password with HTTP
200 and puts the failure in the body - which Caddy never sees. Nor can it be
fixed on the music box: the request arrives from vipy over the tailnet, so the
only address fail2ban could ban there is vipy's, which would take down every
service behind the edge. Banning has to happen at the edge; the edge cannot see
the failure. So /rest/* is protected by password strength alone, and the README
carries a one-line curl to re-check that after every upgrade.
── Passwords are stored reversibly, and cannot not be ─────────────────────
The Subsonic API requires the server to compute md5(password+salt) per request,
so Navidrome cannot hash passwords one-way. It encrypts them at rest with
PasswordEncryptionKey instead. Two real consequences, both in the README: root
on that box means every Navidrome password in plaintext, so the password must
be unique to this service; and changing the key invalidates every stored
password, so it is set before the first account and left alone.
── beets is not a service, deliberately ───────────────────────────────────
`beet import` is interactive and it MOVES AND RENAMES FILES. A timer that
reorganises a music library unattended is a way to lose one. Installed and
configured, run by hand.
/srv/music is counterweight:navidrome mode 2750 and the setgid bit is
load-bearing: without it beets' files land in the curator's own group,
Navidrome cannot read them, and the symptom is an EMPTY LIBRARY rather than a
permission error. ReplayGain and lyrics are auto: no - on one core they turn a
two-minute import into a twenty-minute one.
── Monitoring ─────────────────────────────────────────────────────────────
Host liveness and disk (400), the systemd unit (401), a probe that fetches
/app/ and greps it (403), and public reachability plus cert expiry (402).
/app/ was chosen by testing all three candidates against 0.64.0: /ping returns
a bare "." and proves only that something listens; /rest/ping.view identifies
the server but logs two warnings per call, ~576 a day; /app/ returns
<title>Navidrome</title> and logs nothing. 402 asserts 200 on /app/ and not on
"/", because "/" answers 302 - checked live rather than assumed.
── Also ───────────────────────────────────────────────────────────────────
The headscale ACL grants tag:prd-vipy exactly tag:music-box:4533. Without a
rule the node is invisible rather than unreachable, which surfaces as a Caddy
502 "no such host".
inventory lan_ip was read off nodito's ARP table for the VM's own MAC, not
guessed - and .149 had belonged to another machine, whose stale host key made
SSH refuse the connection. That is the designed behaviour, not a fault.
50 GB is the LIBRARY ceiling, not just the OS: /srv/music is on the root disk,
so roughly 300-450 albums in FLAC. 1 GB is fine for serving and tight for the
initial scan, which is the memory peak.
No backups, deliberately. The music files would swamp the age-encrypted nightly
path, and navidrome.db is a rebuildable index. What is NOT rebuildable is
playlists, play counts, ratings and accounts - there are none today, which is
what makes the trade fair, and the README names this as the decision to revisit
if that changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Onion Monero Blockchain Explorer on monerod-box, published at
exmrplore.contrapeso.xyz. Ten Gatus endpoints green across the node and the
explorer.
── It needs a compiled monero source tree, not a running monerod ───────────
xmrblocks is not an RPC client. cmake/FindMonero.cmake imports 21 STATIC
libraries out of a monero build tree and includes its headers, because it opens
the LMDB directly. The release tarball we install for the daemon ships only
executables - no .a files, no headers - so there is nothing on the box to link
against, and building it means compiling monero first.
Measured on Debian 13, building only the libraries it links:
peak RSS, one cc1plus ~3.2 GB
peak RSS at -j4 ~5.5 GB
wall time at -j4 167 s
source + build tree 195 MB
toolchain 1.6 GB
That rules the box out at ANY parallelism: 768 MiB and 1800 MiB were both
OOM-killed at -j1 and -j2, because a single translation unit over 3 GB is not
something lower -j works around. My first estimate of "~10 GB of disk" was
wrong by two orders of magnitude - that figure describes upstream's full `make`,
which also builds monerod, both wallets and the test suite, none of it linked
here. Disk was never the constraint.
So scripts/build-xmrblocks.sh does the build in a debian:13 container. An
artefact only one machine can produce is one nobody can audit; this way any host
with Docker and ~6 GB reproduces it. It verifies the monero source against
binaryFate's signature exactly as roles/monero does for the daemon, and the role
asserts the resulting sha256 ON THE CONTROL NODE before copying a single byte.
── Three upstream problems, none documented anywhere ───────────────────────
Found by building, not by reading:
1. libasio-dev is absent from upstream's dependency list. That list is
Ubuntu-based; Crow wants standalone asio.hpp, not boost/asio.hpp, and on
Debian 13 the build dies outright.
2. The explorer does not compile against monero v0.18.5.1. v0.18.5.0 changed
get_pruned_transaction_hash from returning the hash to returning bool with
an out-param, and src/page.h still calls the old form. Upstream's
Dockerfile pins v0.18.4.0, which is why nobody has hit it. Patched from an
exact-line anchor, so if upstream fixes it the build STOPS rather than
quietly doing something else. We patch instead of building against 0.18.4.0
because the signed hashes.txt only covers the current release - using the
older source would mean dropping the provenance chain. LMDB is VERSION 5 in
both, so either reads the other's database.
3. FindMonero.cmake hunts for miniupnpc inside the monero build tree with
NO_DEFAULT_PATH, but monero stopped vendoring miniupnp as a submodule.
Fixed with a cache override rather than patching their cmake.
USE_DEVICE_TREZOR=OFF as well - an explorer has no use for hardware wallets and
it drops the generated protobuf units. Verified that xmrblocks still links.
── Two things in the unit that look wrong and are not ──────────────────────
WorkingDirectory is load-bearing: page.h hardcodes TMPL_DIR as "./templates",
resolved relative to cwd at REQUEST time. Get it wrong and the service starts,
binds 8081, and 500s every page. That is also why the health check fetches a
real page and greps it instead of probing the port.
ReadWritePaths must include the blockchain directory even though the database is
opened read-only: monero sets mdb_flags = MDB_RDONLY WITHOUT MDB_NOLOCK, so
LMDB still registers this reader in lock.mdb. With ProtectSystem=strict and
nothing writable the open fails. "It only reads, so grant it nothing" is wrong.
── Timeouts are measured, not guessed ──────────────────────────────────────
proxmox-tank-1 is a spinning-disk mirror and xmrblocks says so on startup.
First start took 98s to open the LMDB before binding (wait_for now allows 300s),
and a COLD index page took 68 SECONDS where the next two took 0.87s and 0.012s
(the check now allows 90s, its unit 180s). The first pass used 90s and 25s and
produced exactly the false failures you would expect.
A stat() bug was caught the same way: `stat` computes a sha1 by default, so
checking that data.mdb exists was hashing the entire 40 GB blockchain. --check
found it before a real run did.
── Exposure ────────────────────────────────────────────────────────────────
Caddy on vipy terminates TLS; the upstream is reached over the tailnet, which
needed a headscale ACL entry for tag:monerod-box:8081 - without it the node is
invisible, not merely unreachable, and the symptom is a 502 "no such host".
No authentication, deliberately: it is a block explorer over public chain data.
That includes --enable-pusher, so a visitor can relay a raw transaction through
our node. A choice, not an oversight; flip xmrblocks_enable_pusher to close it.
── Also ────────────────────────────────────────────────────────────────────
monerod-box raised 2048 -> 6144 MiB. Not for the explorer: monerod alone had
outgrown 2 GB past ~39% of the chain and was OOM-killed twice in a morning at
1.74 GB anon-rss. Recorded in vms.auto.tfvars because ignore_changes = all means
tofu will not reconcile it. The box now idles at ~1.2 GB of 5.9 GB with both
services up.
No backups, deliberately - binary, templates and chain are all reproducible.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
monerod v0.18.5.1 on monerod-box (VMID 2400), Debian 13, systemd, no Docker.
Syncing at 24% as of this commit, 0 restarts, 0 OOM kills.
── The build-from-source requirement could not be met ──────────────────────
The intent was to build it ourselves and check the result against the published
checksums. That is not achievable for this release, and the role's README says
so at length so nobody re-derives it:
contrib/guix - the reproducible build that yields a byte-identical tarball -
exists ONLY on monero master. At tag v0.18.5.1 contrib/ contains `gitian`,
whose README wants an Ubuntu 18.04 host (EOL 2023) and python-cheetah.
A plain `make release-static` is not reproducible at all, so it produces a
binary whose hash matches nothing - failing both halves of the ask.
So we install the published binary and prove its provenance instead:
1. binaryFate's key is VENDORED at files/binaryfate.asc. A trust root fetched
over the network at the moment you need it is not a trust root. Its
fingerprint is asserted after import regardless.
2. hashes.txt is verified, and the assertion is on gpg's `VALIDSIG <fpr>`
status line - NOT on the string "Good signature", which is what
bitcoin_knots greps for and which passes for any key in the keyring.
3. The hash is looked up in the output of `gpg --decrypt`, not in the raw
clearsigned file, so nothing unsigned is ever searched.
4. The lookup requires hash AND filename on one line.
Tested with a positive and a negative control before deploying: the real
tarball matches, a corrupted hash correctly fails. monero_sha256 in defaults/
is a second in-git assertion so a version bump cannot happen without the
checksum moving in the same diff.
When v0.19 ships with guix on the release branch this gets easier, and the
README records why: the published monero-linux-x64 tarball unpacks to
monero-x86_64-linux-gnu-v0.18.5.1/, i.e. it IS the guix DISTNAME renamed, so
sha256(guix output) will equal the hashes.txt entry exactly.
── The 500 GB disk is provisioned in code ──────────────────────────────────
knots-box mounts /mnt/knots_data (~875 GB of chain) entirely by hand; that
mount appears nowhere in Ansible and a rebuild of that host would come up with
the blockchain silently missing. tasks/storage.yml does not repeat it.
Whole device, no partition table. Everything after mkfs goes by UUID, because
/dev/sdX is assigned by the kernel at boot and this estate has been bitten
eight times by an identifier written down once whose thing then moved.
monero_data_device is used ONLY to create the filesystem. Guards refuse to
touch the root disk or to overwrite an unrecognised filesystem, and a final
assert confirms the data directory is not still on /.
fstab gets `nofail` so a missing disk cannot drop a headless VM into an
emergency shell - and the unit gets `RequiresMountsFor=`, without which monerod
would then start, find an empty directory, and write a 300 GB chain onto a
10 GB root disk. With it the unit refuses to start and infra/401 says so within
five minutes.
The mount-point task and the ownership task deliberately share owner and mode.
Writing the obvious root:root 0755 on the first made them disagree and every
run flipped the directory back and forth; caught by re-running --check.
── Exposure: one HTTPS vhost, zero new open ports ──────────────────────────
Two RPC servers, not one: unrestricted on 127.0.0.1:18081 for administration
and the health check, restricted on :18089 for everything else. That is
`rpc-restricted-bind-port` and deliberately NOT `restricted-rpc=1`, which would
restrict the loopback server too. Verified live: the restricted port reports
restricted=true and blanks version and start_time.
--rpc-login is HTTP Digest (contrib/epee/src/http_auth.cpp) and
core_rpc_server::init hands the same login to both servers. Caddy on vipy does
TLS and routing only - no caddy_site_basic_auth, because a Basic challenge
would collide with the Digest challenge Feather answers. Verified end to end
from the WAN: 401 unauthenticated, get_block_count over /json_rpc when
authenticated, 401 again with a wrong password.
No UFW rule, and that is correct: tailnet traffic bypasses UFW entirely because
`-A INPUT -j ts-input` sits ahead of every ufw-* chain. Confirmed on mempool-box
before relying on it here.
P2P is outbound only. Nothing forwards 18080, so the node dials out, syncs and
serves us without relaying for strangers; no-igd follows from that.
── Monitoring: infra, systemd, public. No backups ──────────────────────────
Six Gatus endpoints, all green: liveness, disk, services/monerod-box-local/
monerod, probe/monerod, dns/monero.contrapeso.xyz, public/monero.
The probe does NOT assert `synchronized`. A full sync takes days and a check
requiring it would report DOWN for every one of them - an alarm with the
batteries taken out. Height and sync state go in the message instead.
The timer carries no `Requires=monerod.service`. Fulcrum's did, and sat
active+enabled for seven months without firing once. public/monero asserts 401,
not 200, so it goes red if rpc-login ever stops being enforced.
No backups, deliberately, following the phoenixd precedent: the chain is
re-syncable public data and there is no wallet on this node.
── Known constraint ────────────────────────────────────────────────────────
The VM has 2 GB and no swap; monerod's RSS is ~1.3 GB mid-sync. It is holding
(0 OOM kills, ~1 GB available, kernel reclaiming cache) but 8 GB is the
recommendation once the initial sync is done. out-peers/in-peers are held at
16/8 partly for this reason. Bumping it is a Proxmox resize plus memory_mb
here, since the vm resource sets ignore_changes = all.
The 500 GB data disk is likewise invisible to tofu - it was attached in the UI
and the vm resource ignores it - so vms.auto.tfvars now carries a comment
saying where it is and what mounts it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`Create ACL policies file` wrote this, verbatim, to /etc/headscale/acl.json:
content: |
{}
An empty policy. The real rules - who may reach which host on which port -
were maintained by hand on spacey and existed nowhere else. So the repo and
the box disagreed about the single file that governs every machine's access
to every other machine, and the repo's version was "nothing is allowed".
It had already fired once. /etc/headscale/acl.json.937464.2026-09-12@14:03:08~
is Ansible's own backup of a 1596-byte policy it replaced with {} on 12 Sep;
the live file is dated 17 Sep, so somebody rebuilt it by hand afterwards
without the playbook ever being corrected. Every subsequent `site.yml` run was
one command away from flattening the whole tailnet's access rules again.
This commit changes nothing on spacey. The policy now lives in
templates/acl.json.j2, captured verbatim from the live file, and the task is a
`template:` instead of an inline `{}`. Rendering it produces sha256
1ed14e7247d957db5e9f8d4f4c53846dc646131ad900054c868483a9a84ed8fd, byte for byte
the file that was already there - verified before applying, and again after.
Three deliberate choices:
- A template file, not the inline `content:` used elsewhere in this playbook.
The whole failure was three invisible characters passing review; a 2KB
policy needs to show up as a diff.
- `validate: python3 -m json.tool %s`. headscale refuses to load a malformed
policy, and a policy it will not load is the same outage as an empty one.
- `"counter-net@"` is now `"{{ headscale_namespace }}@"`, which is already a
fact in group_vars/all/main.yml rather than a fourteenth hand-typed copy.
The header comment records why a node with no ACL rule is INVISIBLE rather than
merely unreachable - it never appears in `tailscale status`, MagicDNS does not
resolve it, and the symptom is a downstream "no such host" that looks like a
DNS problem. That cost an hour today.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nothing in the repo pushes to, authenticates against, or is gated by Uptime
Kuma any more.
── The sixth instance of the banner bug ────────────────────────────────────
memos had `Restart memos` guarded by `uptime_kuma_enabled`, because the
deprecation banner was placed immediately above it and swept it in. It is a
HANDLER, so every memos config change since 2026-09-11 applied to disk and
silently never restarted the service. Ungated.
That is the same failure found in forgejo-runner's self-assert, phoenixd's timer
enable, mempool's three timer enables, fulcrum's restart handler and bitcoind's
restart handler. Every guard was read and asked "monitoring or deployment?"
before being deleted, which is the only reason this was caught.
── What was removed ────────────────────────────────────────────────────────
30 uptime_kuma_enabled guards across 7 unconverted service playbooks, and
the 29 Kuma monitor-creation tasks they gated (embedded Python that drove
the Kuma API, temp credential files, cleanup)
7 dead uptime_kuma_api_url definitions
7 stale DEPRECATED banners
uptime_kuma_enabled and subdomains.uptime_kuma from group_vars/all
healthcheck_push_urls from the vault - 30 push tokens
services/ntfy/setup_ntfy_uptime_kuma_notification.yml -> archive/
The explanatory comments in the six converted roles are KEPT on purpose. They
record why a handler is ungated, and deleting the explanation invites someone
to helpfully re-add the guard.
── The probes moved rather than died ───────────────────────────────────────
Eight per-service health checks were still pushing to Kuma. They are not
superseded by infra/401: that answers "is the unit running", these answer "does
the service actually respond" - an RPC call to bitcoind, a TCP connect to
Fulcrum's Electrum port, an HTTP fetch from Mempool's backend. A process can be
perfectly `active` and useless.
So they were repointed, not deleted. Gatus external endpoints take a POST with
a bearer token and success=true|false where Kuma took a GET with ?status=up, so
report() now maps up/down to true/false internally and no call site changed.
Registered by infra/403 as the `probe` group, one token per host.
Two bugs fixed while in there:
* forgejo-runner's check only ever reported SUCCESS - it exited before pushing
when the runner was down, so a failure was invisible until the heartbeat
window expired. Reporting the failure is the entire point of a check.
* All six healthcheck .service units were mode 0644 and now carry a bearer
token. They are 0600.
Verified: 91 endpoints, 91 UP, 0 DOWN. Every probe triggered by hand and
confirmed arriving. Zero Kuma URLs left in the vault, zero live references in
any playbook or role.
Still standing, deliberately: the Kuma container on watchtower, its Caddy vhost,
and the uptime.contrapeso.xyz DNS record. Turning the service off is a separate
decision from removing the code that talked to it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Twelve endpoints in two groups, because they answer different questions and
fail for different reasons:
backup-dump_<svc> pushed by the SOURCE right after its dump runs
backup-store_<svc> pushed by the BOX at 05:30, per source
backup-store_pull-job pushed by the BOX, about the box itself
The store alone could catch almost everything, because the artefact filename
carries the source's dump timestamp - a source whose timer died still pulls
"ok" forever, but the timestamp gives it away. What the source side adds is
LATENCY and DIAGNOSIS: the store only learns at the next 04:00 pull, and it
cannot tell you whether the dump broke or the pull did.
pull-job is separate from the per-source checks because it is a LEADING
indicator where those are lagging ones. A disabled pull timer, a failed pull
job, or a filling disk are all visible immediately, while the per-source checks
only fire once an artefact is >26h stale - about a day later. Disable the timer
at 10:00 and every source stays green until tomorrow; pull-job goes red this
morning and names the cause instead of showing six stale sources.
Frequencies: dumps 02:00-02:30 staggered, pull 04:00, verification 05:30, all
daily and Persistent. 26h staleness decides red; a 30h Gatus heartbeat catches
the verification itself having stopped, so a dead check-backups.timer cannot
hide a stale backup.
arbret has no dump endpoint: prd-arbret is in [arbret], which `managed`
deliberately excludes, so nothing of ours runs there. Store-checked only.
check-backups.sh was manual-only; it now runs on a timer and reports per source
rather than only printing. The human-readable report is unchanged.
A SERIOUS bug introduced and fixed in this change, recorded because the shape
is easy to repeat: the reporting hook was added to backup.sh as a second
`trap ... EXIT`. Bash REPLACES the EXIT handler rather than adding to it, so
that silently deleted the trap which restarts the stopped service - the one the
script's own comment calls "the point", and the bug the role was written to
eliminate. Every backup then stopped its service and left it stopped. It took
forgejo, lnbits, headscale and memos down for several minutes each, and nothing
caught it: the dumps exit 0, the artefacts are correct, the deploy reports
failed=0, and liveness only proves the HOST is up. There is now ONE EXIT
handler doing both jobs, armed BEFORE the stop so a failure during the stop
still restarts. Verified by rendering both variants, asserting exactly one EXIT
trap in each, and simulating a mid-way failure to confirm the restart fires.
Verified: all 12 endpoints UP; six sources pulled cleanly (arbret 31M,
headscale 198K, memos 7.8M, vaultwarden 2.1M, lnbits 30M, forgejo 2.6G), store
16% full, "RESULT: all checks passed".
Known gaps, deliberately not closed here:
* `yell` warnings - disk 75-90%, an artefact under half the previous size,
retention not pruning - never reach Gatus, because a push is binary.
* Nothing verifies a backed-up service came back UP. That is the gap that let
the trap bug run unnoticed, and it is what the next change addresses.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The four ports were the only entries in services_config.yml with a real
justification: each is read twice, by the role that deploys the service on its
own box AND by a socket-proxy or Caddy play that runs on the EDGE host and
publishes it. A role default is invisible to that second play.
But the shape was wrong in two ways. The file had to be named in vars_files: by
30 plays - opt-in configuration that someone will eventually forget - and five
role defaults silently interpolated service_settings.*, so bitcoin_knots,
fulcrum, datum_gateway and mempool were not self-contained: using any of them
without that one vars_file entry broke it.
Each port now lives in host_vars/<owning box>/main.yml:
host_vars/knots_box_local/main.yml bitcoin_p2p_port, datum_gateway_api_port,
datum_gateway_stratum_port
host_vars/fulcrum_box_local/main.yml fulcrum_ssl_port
host_vars/mempool_box_local/main.yml mempool_frontend_port
host_vars auto-loads and outranks role defaults, so the owning role picks the
value up with no vars_files at all, and the edge play reads the same single
definition as hostvars['<host>'].<name>. The role defaults keep the protocol
standard (8333, 50002, ...) so each role still works standalone, with the live
deployment's value in host_vars winning.
Also fixed a fourth copy of an inventory identity: the mempool Caddy play had
"mempool-box:{{ ... }}" hardcoded in the upstream. It now derives the host from
hostvars['mempool_box_local'].ansible_host, so inventory is the only place any
box's name is written down.
services_config.yml is deleted, with 25 more vars_files entries across 19
playbooks. Between this and the previous commit, 87 vars_files entries are gone
and every variable in the repo now comes from group_vars/all, host_vars,
inventory, a role default, or that service's own *_vars.yml.
Verification: an edge-host probe resolves all eight ports and hostnames to
byte-identical values to the ones services_config.yml used to supply. Each
owning host resolves its own port through host_vars. All 37 playbooks'
--list-tasks output is unchanged. The four edge plays that consume these values
all check-diff changed=0 - the socket-proxy and Caddy units on vipy are
byte-identical, which is the direct proof the rewiring landed on the same
values. fulcrum and datum-gateway check-diff exactly as before (ok=28/changed=1
and ok=15/changed=1, both the known timer re-arm).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three files existed only as second copies of things group_vars/all already
auto-loads, and 34 playbooks named them in vars_files: - which outranks
group_vars, so the copies won. The day someone edited one and not the other,
those plays would silently keep the stale value. infra_vars.yml was already
drifting: group_vars/all/main.yml had grown age_backup_recipient and
backup_pull_public_key that it lacked.
infra_vars.yml - a strict subset of group_vars/all/main.yml
infra_secrets.yml - decrypts byte-identical to group_vars/all/vault.yml
infra_secrets.yml.example - documented Uptime Kuma credentials as the reason
the file exists, which stopped being true
Deleted, along with 62 vars_files entries across 34 playbooks (12 of which
named ../../group_vars/all/main.yml directly - same defect, a vars_files entry
duplicating an auto-loaded file at higher precedence than the file itself).
Checked before touching anything: infra_secrets.yml was listed LAST in 10 plays,
after services_config.yml, so removal would flip precedence if the two shared a
key. They share none, and neither does services_config.yml with
group_vars/all/main.yml, so the removal is provably inert.
services_config.yml was the last one standing. It held four unrelated things:
caddy_sites_dir - an identical copy of roles/caddy_site/defaults/.
Deleted; the role default is now the only one.
*.tailscale_hostname (x3) - a THIRD copy of each box's identity, which
inventory.ini already holds as ansible_host.
Deleted. Edge plays now read
hostvars['<host>'].ansible_host - verified an
edge play resolves that with nothing loaded and
the other host in no play. Three copies of one
name is how bitcoin_rpc_host ended up labelled
"knots_box" while pointing at fulcrum-box.
subdomains, ntfy topic, - genuinely global: their readers span managed,
headscale namespace monitoring, vpn_control and edge, so no single
group covers them. Moved to group_vars/all/main.yml
where they auto-load. The ntfy_topic and
headscale_namespace indirection through
service_settings collapses to the global name.
the four cross-host ports - the only entries with a real justification.
Left in place; they move in the next commit.
Also dead, all Uptime Kuma residue or duplication:
phoenixd_monitor_name, forgejo_runner healthcheck_timeout_seconds/retries,
fulcrum_tailscale_hostname, and bitcoin_knots_version - the last being a
v-prefixed copy of bitcoin_knots_version_short that nothing read, two
hand-maintained copies of one version string.
Corrected a false comment: services_config.yml claimed the uptime_kuma subdomain
"no longer resolves to anything". It resolves to 164.92.239.72 and answers HTTP
302, and 11 playbooks still template it. Same wrong premise as PLAN_3.
Verification: all 37 playbooks' --list-tasks output is byte-identical before and
after. A probe resolving all 22 values services_config.yml used to supply returns
21 identical and one intended deletion (caddy_sites_dir, now role-only - confirmed
the role still resolves it: "Ensure Caddy sites-enabled directory exists" comes
back ok against the real path). memos check-diff identical before and after.
Syntax passes on every playbook.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Completes Stage 3. No hand-rolled Caddy plumbing remains anywhere:
`grep sites-enabled` outside roles/ returns nothing, and so does
`grep "systemctl reload caddy"`.
ntfy uses caddy_site_body for its plain-HTTP listener and @httpget redirect.
Verified ok/unchanged against watchtower; the one other changed task is a
pre-existing "Update APT cache".
datum-gateway keeps a whole-Caddyfile validate after the role call. The role
validates its own fragment, but only a whole-file validate catches a conflict
between two sites, and this playbook was the only one that ever had it. Its
two debug tasks that echoed command output are gone with the commands.
headscale is the one that mattered. Its playbook wrote
`reverse_proxy localhost:8080`, but spacey is actually running a /admin*
route in front of Headplane behind Caddy basic auth. Running that playbook
would have deleted the admin route and its auth - a hazard that predates this
work. It now renders the config that is really there, verified ok/unchanged
via --start-at-task (the play cannot reach Caddy in check mode: "Install
headscale package" fails because the .deb is not really downloaded, before
and after this edit alike).
Supporting changes for headscale:
- headscale_ui_password_hash added to infra_secrets.yml and the identical
group_vars/all/vault.yml, read from the live config on spacey. The vault
already had headscale_ui_username (= counterweight, confirmed) and
headscale_ui_password; I did not verify the password is the plaintext of
this hash.
- headplane_port added to headscale_vars.yml.
- The role's handler now sets become: true. Handlers do not inherit become
from the task that notified them, and this play runs become: no.
- The include uses `apply: become: yes`; `become:` on an include_role is
rejected outright.
All 14 site files on all 3 hosts still byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Removing the credentials would otherwise break these playbooks mid-deploy: they
template uptime_kuma_password with no assert to stop them first. 100 tasks are
now guarded by uptime_kuma_enabled (false), so deployments run normally and the
monitoring sections skip. A further 28 tasks were already self-guarding on
monitor_setup/push_url being defined; verified that a skipped task's registered
variable makes those skip cleanly rather than error.
The blocks are kept on purpose — the health-check logic is the durable part and
should be rewired to whatever replaces Uptime Kuma.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
