personal_infra/arbret_handoff.md

arbret on prd-vipy: monitoring and backups

Handoff for the arbret owner. It covers what changed when prd-arbret was lost, and what your Ansible repo needs to do so that arbret is monitored by Gatus and backed up to small-backups-box from its new home.

Written 2026-10-17, during the rebuild.

---

0. What changed, in one screen

	Before	Now
Host	prd-arbret, a droplet of its own	prd-vipy, shared with the personal_infra services
Monitoring	Uptime Kuma (push URLs on uptime.contrapeso.xyz)	Gatus at https://status.contrapeso.xyz, running on prd-spacey. Uptime Kuma is gone for good.
Backup destination	small-backups-box pulls arbret@prd-arbret:/opt/arbret/backups/	small-backups-box pulls backup-pull@prd-vipy:/opt/backups/arbret/
TLS / front door	your own, on prd-arbret	our Caddy on prd-vipy (it owns 80/443)

Who does what:

Area	You (arbret repo)	Us (personal_infra repo)
App, workers, Postgres, analytics	deploy and run them	—
Public HTTPS for arbret.com	tell us your local port	Caddy vhost, certificates
Gatus endpoint definitions	—	all of them: Gatus config lives on prd-spacey, which you don't touch
Pulled checks (HTTPS, /api/health)	keep /api/health honest	define and run them
systemd unit checks (arbret-server, arbret-worker, postgresql)	keep the unit names stable	add them to vipy's monitored units
Host liveness and disk	—	already covered for prd-vipy
Analytics timer result	push it (§2.4)	register the endpoint
Backup dump	produce it to the contract (§3)	pull it, retain it 90 days, check freshness
Backup dump result	push it (§3.5)	register the endpoint

Your work comes down to three things:

1. Live on a shared host without stepping on it (§1).
2. Push two results to Gatus: analytics and backup.
3. Write the dump to a new place, in a fixed shape.

---

1. Living on a shared host

prd-vipy also runs Vaultwarden, Forgejo, LNbits, phoenixd and a reverse proxy for a dozen other services. Our Ansible converges it regularly. Anything below that your playbooks change will be reverted by ours, or will break ours.

Owned by personal_infra: do not manage these

• Ports 80 and 443 and the Caddy install (/etc/caddy/**). Don't install nginx, certbot or a second Caddy, and don't bind 80/443. Give us a port and we write the vhost (see below).
• UFW and fail2ban. The default policy is deny-incoming, with only SSH and 80/443 open. Don't disable UFW or reset its rules. If you need a port reachable from outside, ask us.
• Users counterweight and backup-pull, and their authorized_keys. In particular don't create, modify or authorise keys for backup-pull: we create it and it is shared by every backup on this host (§3).
• /opt/backups/ (the parent directory, root:root 0755). You own exactly /opt/backups/arbret/ and nothing beside it.
• Tailscale/Headscale membership of the host, and the tailnet ACLs.
• Docker. It's installed for our services. You don't need it (arbret is native systemd); if you start using it, don't touch /etc/docker/daemon.json.
• systemd unit names we use: caddy, forgejo, lnbits, phoenixd, *-backup.{service,timer} for vaultwarden/lnbits/forgejo, liveness-healthcheck, disk-usage-healthcheck, service-health-healthcheck. Prefix all of yours with arbret- and there's no risk of collision.

Free for you

• PostgreSQL. We run no Postgres on this host, so the system postgresql package and cluster are yours.
• /opt/arbret/, /etc/arbret/, the arbret system user, arbret-* units and timers.

Local ports already taken

22, 80, 443, 3000, 7657, 8222, 8333, 8765, 9740, 23334, 50002. Bind your app to 127.0.0.1:<port> on anything else and tell us the port. Our Caddy then serves arbret.com with a Let's Encrypt certificate and reverse-proxies to it, setting X-Forwarded-For, X-Forwarded-Proto and X-Forwarded-Host. Configure the app to trust proxy headers from 127.0.0.1 only.

Port 4117 used to be reachable from one admin laptop over the tailnet. If that's still needed, say so and we'll add the ACL for prd-vipy. Don't open it in UFW yourself.

The host itself

Debian 13, clock in UTC, tailnet name prd-vipy.

---

2. Monitoring with Gatus

2.1 How it works here

Gatus runs on prd-spacey and answers at https://status.contrapeso.xyz. Alerts go to Signal. There are two kinds of check:

• Pulled. Gatus makes the request itself (HTTP, TCP, DNS) every N seconds and evaluates conditions. Nothing is installed on your side. This is the right kind for anything with a reachable URL.
• Pushed ("external endpoints"). Something on the host POSTs a result to Gatus, and Gatus alarms if a result is false or if nothing arrives within the heartbeat window. This is the right kind for things with no URL: timers, jobs, internal state. The heartbeat is what makes a dead timer visible: silence is a failure.

Everything about an endpoint (name, group, token, heartbeat, alert rules) is defined in our repo, because Gatus's config files live on prd-spacey. Your side of a push check is just an HTTP call.

2.2 The push API

POST https://status.contrapeso.xyz/api/v1/endpoints/{KEY}/external?success={true|false}&error={message}
Authorization: Bearer {TOKEN}

• KEY is sanitize(group) + "_" + sanitize(name), where sanitize lowercases and turns any of / _ . , space # + & into -. Use the exact keys in the table below.
• success: true or false.
• error: free text, URL-encoded. It's shown on the dashboard and in the Signal alert, so on success it's fine to send something useful like ok 31M.
• TOKEN: a bearer token we give you (§4). It's dedicated to arbret: it can only write arbret's endpoints, and nobody else's token can write yours.

Responses:

HTTP	Meaning
200	recorded
401	wrong token for that key
404	key not registered (yet). We register before you deploy.

Test it by hand once you have the token:

curl -i -X POST -H "Authorization: Bearer $TOKEN" \
  "https://status.contrapeso.xyz/api/v1/endpoints/probe_arbret-analytics/external?success=true&error=manual%20test"

2.3 Your old Uptime Kuma monitors, mapped

Old monitor (prod)	New check	Gatus key	Kind	Who
https: front door returns 200	GET https://arbret.com/ → 200	public_arbret	pulled, 5 min	us
health: /api/health returns ok	GET https://arbret.com/api/health → 200 and [BODY].status == ok	public_arbret-health	pulled, 5 min	us
arbret-server: unit active	systemd unit check on prd-vipy	services_vipy-arbret-server	push, every 5 min, by our host script	us
arbret-worker: unit active	same	services_vipy-arbret-worker	same	us
postgresql: unit active	same	services_vipy-postgresql	same	us
arbret-analytics: timer ran OK	your analytics run reports its result	probe_arbret-analytics	push, by you, hourly; heartbeat 130m	you (§2.4)
(new) beancount: daily job ran OK	your beancount run reports its result	probe_arbret-beancount	push, by you, daily 03:00; heartbeat 26h	you (§2.4, same hook)
arbret-backup: timer ran OK	your backup run reports its result	backup-dump_arbret	push, by you	you (§3.5)
prd-arbret liveness / disk	covered by prd-vipy's own host checks	liveness_vipy, disk_vipy	push	us

We also check:

• backup-store_arbret: the backup arrived on small-backups-box, is fresh (<26h, read from the filename timestamp), non-empty, and not suspiciously smaller than the previous one.
• arbret.com resolves to prd-vipy, checked every 6 hours.
• arbret.com registration expiry, alarming 14 days ahead. This already existed.

Delete, don't port, every Uptime Kuma push script, timer and URL in your repo. The endpoint no longer exists. A leftover timer will just fail silently every two minutes forever.

The pulled health check relies on /api/health returning HTTP 200 with a JSON body that has "status": "ok" when healthy, and a non-200 when it isn't. If the body differs, tell us the exact shape. A health endpoint that checks the database is worth more than one that only proves the process answers.

2.4 What you implement: the analytics push

The cleanest hook is ExecStopPost= on the oneshot service the timer runs. systemd runs it after every run, success or failure, and sets $SERVICE_RESULT and $EXIT_STATUS for it. So no wrapper script is needed, and the run itself stays untouched.

/etc/arbret/gatus.env (root:root 0600):

GATUS_PUSH_BASE=https://status.contrapeso.xyz/api/v1/endpoints
GATUS_PUSH_TOKEN=<token from §4>

/usr/local/bin/arbret-gatus-push (root:root 0755):

#!/usr/bin/env bash
# Usage: arbret-gatus-push <gatus-key>
# Meant for ExecStopPost=, where systemd provides SERVICE_RESULT / EXIT_STATUS.
# Can also be called by hand: SERVICE_RESULT=success arbret-gatus-push <key>
set -uo pipefail
KEY="$1"
: "${GATUS_PUSH_BASE:?}" "${GATUS_PUSH_TOKEN:?}"

if [ "${SERVICE_RESULT:-}" = "success" ]; then
  success=true;  msg="ok"
else
  success=false; msg="${SERVICE_RESULT:-unknown} exit=${EXIT_STATUS:-?}"
fi
enc=$(printf '%s' "$msg" | sed 's/%/%25/g; s/ /%20/g; s/&/%26/g; s/+/%2B/g; s/#/%23/g; s/=/%3D/g')

# Never let a monitoring hiccup fail the job itself.
curl -s -o /dev/null --max-time 15 --retry 2 --retry-delay 3 -X POST \
  -H "Authorization: Bearer ${GATUS_PUSH_TOKEN}" \
  "${GATUS_PUSH_BASE}/${KEY}/external?success=${success}&error=${enc}" || true
exit 0

In the analytics service unit:

[Service]
Type=oneshot
EnvironmentFile=/etc/arbret/gatus.env
ExecStart=...your existing command...
ExecStopPost=/usr/local/bin/arbret-gatus-push probe_arbret-analytics

Two things to get right:

• EnvironmentFile= puts the token in the unit's environment, not in the unit file, so the unit file can stay world-readable while the env file is 0600.
• Tell us the timer's schedule. We set the heartbeat to the interval plus slack: a 5-minute job gets an 11-minute window, a daily job gets 26 hours. If the schedule changes later, tell us, or Gatus will either alarm on a healthy job or sleep through a dead one.

As Ansible, this is two templated files, an EnvironmentFile= line and an ExecStopPost= line. It's small enough to live in the role that deploys the analytics timer, rather than becoming a role of its own.

2.5 Staging (arbret-staging-box)

arbret-staging-box is a VM in our inventory, so its liveness and disk checks already exist. The same mapping applies with these keys:

Check	Key	Who
front door / health	public_arbret-staging, public_arbret-staging-health	us
units	services_arbret-staging-local-arbret-server (and -worker, -postgresql)	us
analytics	probe_arbret-staging-analytics	you, with the staging token
backup dump	backup-dump_arbret-staging	you, with the staging token. Only if staging should be backed up; tell us.

The staging vhost (arbretstaging.contrapeso.xyz) lived on the old prd-vipy and was not in either repo. We're recreating it on our side.

---

3. Backups

3.1 The pipeline

 02:00 UTC  prd-vipy   your timer:  pg_dump | age -r <recipient>  ->  /opt/backups/arbret/arbret_<ts>.dump.age
                       your script: POST backup-dump_arbret (success/failure)
 04:00 UTC  small-backups-box  rsync  backup-pull@prd-vipy:/opt/backups/arbret/  ->  ~/backups/arbret/  (keeps 90 days)
 05:30 UTC  small-backups-box  check-backups.sh  ->  POST backup-store_arbret  (then every 6 h)

• The box only pulls. Nothing on prd-vipy can reach into the backup box. The pull logs in as backup-pull, an unprivileged account with a restricted key that can read the backup directories and nothing else.
• The source encrypts. Only the age public key is on prd-vipy, so the host can't decrypt its own backups. The private key lives offline with the infra owner. Plaintext never touches disk or the network.
• Retention is two-tier. Keep about 7 days locally (the disk is shared). The box keeps 90.

Your existing dump role already produces the right artefact: a Postgres custom-format dump, age-encrypted, named arbret_YYYYmmdd_HHMMSS.dump.age. The last pre-incident one is arbret_20261016_020001.dump.age. Only its location, ownership and reporting change.

3.2 What changes from prd-arbret

	prd-arbret	prd-vipy
Directory	/opt/arbret/backups/	/opt/backups/arbret/
Account the box pulls as	arbret (your user, with our key authorised)	backup-pull (ours, already exists)
SSH key for the pull	you authorised it	nothing to do. Remove that task, and don't authorise anything for arbret or backup-pull.
Result reporting	Uptime Kuma push	Gatus push to backup-dump_arbret

3.3 The contract

The checks on the box enforce these, and some failures are silent, so read every line.

1. Directory: /opt/backups/arbret/, owner root, group backup-pull, mode 0750. Don't change /opt/backups/ itself (root:root 0755, ours). The backup-pull group exists before your playbook runs; if it doesn't, fail loudly rather than creating it.
2. Filename: arbret_YYYYmmdd_HHMMSS.dump.age, timestamp in the host's local time (UTC). The prefix must be exactly arbret_, because the box's prune and freshness checks glob on arbret_*, and it reads the dump time from the filename.
3. Encryption: age -r <recipient> in the pipe (§4). Nothing unencrypted is ever written to disk.
4. Atomicity: write <file>.partial, set ownership and mode, then mv it to the final name. The pull skips *.partial. A half-written file under the final name would be pulled and kept for 90 days as if it were a backup.
5. File ownership: root:backup-pull, mode 0640, on both the .partial and the final file. If the pull account can't read a file, rsync fails for the whole directory, so one bad file blocks every later night.
6. set -euo pipefail. Without pipefail, a failing pg_dump piped into a succeeding age exits 0 and produces a tiny, valid-looking, useless artefact.
7. Timing: finished before 04:00 UTC. 02:00 is good.
8. Local prune: delete arbret_*.dump.age older than ~7 days, plus stale arbret_*.partial files at the start of each run. Put nothing else in the directory: no subdirectories, logs or checksums.
9. Report the outcome to Gatus on every exit path (§3.5).
10. No service stop is needed. pg_dump takes a consistent snapshot online. Don't stop arbret-server for it.

3.4 Reference script

This is the same shape as the script every other service on the host uses (our backup_source role), adapted to pg_dump. Adjust the database name and the pg_dump invocation to match your current role.

/usr/local/bin/arbret-backup.sh (root:root 0750):

#!/usr/bin/env bash
set -euo pipefail
umask 077

BACKUP_DIR=/opt/backups/arbret
RECIPIENT="age192wwdaseqej2ggwyp884gtm05c396anp7chr0vr8m47g50fahpyqr9fsza"
RETENTION_DAYS=7
NAME=arbret
DB=arbret                                   # <- your database name
ARTIFACT="${BACKUP_DIR}/${NAME}_$(date +%Y%m%d_%H%M%S).dump.age"

PUSH_URL="${GATUS_PUSH_BASE:-}/backup-dump_arbret/external"
PUSH_TOKEN="${GATUS_PUSH_TOKEN:-}"

report() {   # $1=true|false  $2=message
  [ -n "${GATUS_PUSH_BASE:-}" ] || return 0
  local enc; enc=$(printf '%s' "$2" | sed 's/%/%25/g; s/ /%20/g; s/&/%26/g; s/+/%2B/g; s/#/%23/g')
  curl -s -o /dev/null --max-time 15 --retry 2 --retry-delay 3 -X POST \
    -H "Authorization: Bearer ${PUSH_TOKEN}" "${PUSH_URL}?success=$1&error=${enc}" || true
}
on_exit() {   # the ONE exit trap: bash keeps only one, a second replaces it
  local rc=$?
  if   [ "$rc" -ne 0 ];       then report false "arbret dump exited ${rc}"
  elif [ ! -s "$ARTIFACT" ];  then report false "arbret produced no artefact"
  else                             report true  "arbret $(du -h "$ARTIFACT" | cut -f1)"; fi
}
trap on_exit EXIT

command -v age >/dev/null || { echo "age not installed" >&2; exit 1; }
getent group backup-pull >/dev/null || { echo "group backup-pull missing - personal_infra has not provisioned this host" >&2; exit 1; }

install -d -o root -g backup-pull -m 0750 "$BACKUP_DIR"
rm -f "${BACKUP_DIR}/${NAME}_"*.partial

runuser -u postgres -- pg_dump -Fc "$DB" | age -r "$RECIPIENT" -o "${ARTIFACT}.partial"
chown root:backup-pull "${ARTIFACT}.partial"; chmod 0640 "${ARTIFACT}.partial"
mv "${ARTIFACT}.partial" "$ARTIFACT"

find "$BACKUP_DIR" -maxdepth 1 -type f -name "${NAME}_*.dump.age" -mtime +"$RETENTION_DAYS" -delete
echo "wrote $ARTIFACT"

Units:

# /etc/systemd/system/arbret-backup.service   (0644; the token is in the env file, not here)
[Unit]
Description=arbret database backup
After=postgresql.service

[Service]
Type=oneshot
EnvironmentFile=/etc/arbret/gatus.env
ExecStart=/usr/local/bin/arbret-backup.sh

# /etc/systemd/system/arbret-backup.timer
[Unit]
Description=arbret database backup, nightly

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true

[Install]
WantedBy=timers.target

age and curl come from Debian (apt install age curl). Installing them from your playbook is fine, since both repos asking for the same package doesn't conflict.

3.5 Reporting the backup

The script above already reports, on every exit path, to backup-dump_arbret, using the same /etc/arbret/gatus.env as the analytics push. Its heartbeat is 26 hours: one missed night alarms. The box-side check (backup-store_arbret) independently notices within 6 hours of the next pull if the artefact stops arriving or stops being fresh. So a broken dump and a broken pull show up as different red lights.

3.6 Verify after your first deploy

sudo systemctl start arbret-backup.service
sudo journalctl -u arbret-backup.service -n 20 --no-pager
ls -l /opt/backups/arbret/                     # root backup-pull, -rw-r-----
sudo -u backup-pull head -c 30 /opt/backups/arbret/arbret_*.dump.age   # must print "age-encryption.org/v1"
systemctl list-timers arbret-backup.timer

Then check that backup-dump_arbret is green on the status page, and tell us. We'll trigger a pull rather than wait for 04:00, and confirm backup-store_arbret.

3.7 Restoring

Only the infra owner can decrypt: the age identity is offline. They'll hand you a decrypted dump over a secure channel. The latest pre-incident one is arbret_20261016_020001.dump.age (31.9 MB, sha256 2bb771957513654cb907e5d63153e7714d54c7d5695d7caf09a05f948dcf515d). Anything written to arbret after 2026-10-16 02:00 UTC is not in any backup.

For a custom-format dump:

sudo -u postgres createdb -O arbret arbret          # or let --create do it
sudo -u postgres pg_restore -d arbret --no-owner --role=arbret -j 4 arbret_20261016_020001.dump

Create the roles the app expects first. pg_dump -Fc doesn't include them. Then shred the plaintext dump: shred -u arbret_20261016_020001.dump.

---

4. What you'll receive from us

Item	Secret	Used for
Gatus push token, prod	yes	/etc/arbret/gatus.env on prd-vipy. Keep it in your vault.
Gatus push token, staging	yes	the same file on arbret-staging-box, if you do the staging pushes
age recipient age192wwdaseqej2ggwyp884gtm05c396anp7chr0vr8m47g50fahpyqr9fsza	no, it's a public key	the backup script. Safe to commit in plain text.
SSH access to prd-vipy for your playbooks	access	being arranged separately
The decrypted 2026-10-16 dump	yes, it's the full prod DB	restore, handed over once

You need no SSH key for the backup pull, and nothing from the old Uptime Kuma setup.

5. What we need from you

1. The local port arbret listens on (for the arbret.com vhost), and whether www.arbret.com should be served or redirected.
2. The exact /api/health response when healthy, and confirmation that it returns a non-200 when the DB is down.
3. The analytics timer's schedule (sets the heartbeat).
4. Whether port 4117 over the tailnet is still needed.
5. The final unit names, if not arbret-server, arbret-worker, postgresql.
6. Whether staging should be backed up, and whether staging pushes are in scope.
7. Who besides the infra owner should get Signal alerts for arbret.

Once we have 1 to 3, we register the endpoints, the vhost and the pull source. Then you can deploy and see green on the first run.