backup stuff

ansible/roles/backup_source/README.md

@@ -0,0 +1,76 @@
+# `backup_source`
+
+Makes a host back **itself** up: dump to stdout, encrypt with `age`, write to a
+local directory, prune, on a systemd timer. `small-backups-box` pulls the
+directory later (see `backup_store`).
+
+Modelled on `prd-arbret`, which has been doing exactly this correctly since
+before the rest of the estate was migrated.
+
+## Usage
+
+```yaml
+- ansible.builtin.include_role:
+    name: backup_source
+  vars:
+    backup_source_name: headscale
+    backup_source_description: "Headscale"
+    backup_source_dump_command: "tar -czf - -C / var/lib/headscale etc/headscale"
+    backup_source_stop_service: headscale
+    backup_source_retention_days: 7
+```
+
+Produces `/opt/backups/headscale/headscale_<YYYYmmdd_HHMMSS>.tar.gz.age`,
+`headscale-backup.{service,timer}`, and `/usr/local/bin/headscale-backup.sh`.
+
+## Why the source encrypts, not the destination
+
+`age -r <recipient>` is asymmetric and the host holds only the **public** key, so
+a compromised host cannot read its own backups — or anyone else's. The scripts
+this replaces encrypted with GPG *on the laptop, after the data had already
+crossed the network*, which protects the artefact at rest but not in transit.
+
+The matching identity lives only on lapy and is escrowed. **Lose it and every
+artefact everywhere becomes noise**, including arbret's.
+
+## `backup_source_dump_command` writes to STDOUT
+
+The role pipes it into `age`, so plaintext never touches the disk. Use `-C /`
+with relative paths in `tar` rather than absolute ones: it avoids tar's "removing
+leading /" and makes the restore target explicit.
+
+## The trap is the reason this role exists
+
+When `backup_source_stop_service` is set, the script stops the unit and installs
+an EXIT trap that starts it again. Without it, a failed dump leaves the service
+down until the next timer fires — **every hand-written script this replaced had
+that bug**, and it was only ever masked because their `systemctl stop` failed
+first, before anything was stopped.
+
+Verified on spacey: with the dump forced to fail, the log shows
+`Stopping → Writing → Restarting`, the script exits 1 (so systemd marks the unit
+failed rather than hiding it), and headscale is `active` afterwards.
+
+If `systemctl stop` itself fails, `set -e` exits *before* the trap is installed —
+which is correct, because nothing was stopped.
+
+## `.partial`
+
+The dump writes `<artifact>.partial` and only `mv`s it into place on success, so
+a truncated file is never mistaken for a backup. A failure inside the pipeline
+does leave one behind, and the prune glob cannot match it (it ends `.partial`,
+not `.tar.gz.age`), so the script clears stale partials at the **start** of each
+run. Tested by failing mid-pipeline: 1 partial left, 0 after the next run.
+
+## `backup_source_stop_service` may be a bare name
+
+`headscale` and `headscale.service` both work. The unit template normalises it,
+because systemd rejects a bare name in `After=` with
+`Failed to add dependency ... Invalid argument` — which it logs and then ignores,
+so the unit appears to work while carrying no ordering at all.
+
+## Retention is two-tier
+
+`backup_source_retention_days` is **local** and short — these hosts are
+disk-constrained. The long tail lives on `small-backups-box`, which decides its
+own retention per source. Losing the local copy is expected and fine.