personal_infra/ansible/roles/backup_source

backup_source

Makes a host back itself up: dump to stdout, encrypt with age, write to a local directory, prune, on a systemd timer. small-backups-box pulls the directory later (see backup_store).

Modelled on prd-arbret, which has been doing exactly this correctly since before the rest of the estate was migrated.

Usage

- ansible.builtin.include_role:
    name: backup_source
  vars:
    backup_source_name: headscale
    backup_source_description: "Headscale"
    backup_source_dump_command: "tar -czf - -C / var/lib/headscale etc/headscale"
    backup_source_stop_service: headscale
    backup_source_retention_days: 7

Produces /opt/backups/headscale/headscale_<YYYYmmdd_HHMMSS>.tar.gz.age, headscale-backup.{service,timer}, and /usr/local/bin/headscale-backup.sh.

Why the source encrypts, not the destination

age -r <recipient> is asymmetric and the host holds only the public key, so a compromised host cannot read its own backups — or anyone else's. The scripts this replaces encrypted with GPG on the laptop, after the data had already crossed the network, which protects the artefact at rest but not in transit.

The matching identity lives only on lapy and is escrowed. Lose it and every artefact everywhere becomes noise, including arbret's.

backup_source_dump_command writes to STDOUT

The role pipes it into age, so plaintext never touches the disk. Use -C / with relative paths in tar rather than absolute ones: it avoids tar's "removing leading /" and makes the restore target explicit.

The trap is the reason this role exists

When backup_source_stop_service is set, the script stops the unit and installs an EXIT trap that starts it again. Without it, a failed dump leaves the service down until the next timer fires — every hand-written script this replaced had that bug, and it was only ever masked because their systemctl stop failed first, before anything was stopped.

Verified on spacey: with the dump forced to fail, the log shows Stopping → Writing → Restarting, the script exits 1 (so systemd marks the unit failed rather than hiding it), and headscale is active afterwards.

If systemctl stop itself fails, set -e exits before the trap is installed — which is correct, because nothing was stopped.

.partial

The dump writes <artifact>.partial and only mvs it into place on success, so a truncated file is never mistaken for a backup. A failure inside the pipeline does leave one behind, and the prune glob cannot match it (it ends .partial, not .tar.gz.age), so the script clears stale partials at the start of each run. Tested by failing mid-pipeline: 1 partial left, 0 after the next run.

backup_source_stop_service may be a bare name

headscale and headscale.service both work. The unit template normalises it, because systemd rejects a bare name in After= with Failed to add dependency ... Invalid argument — which it logs and then ignores, so the unit appears to work while carrying no ordering at all.

Retention is two-tier

backup_source_retention_days is local and short — these hosts are disk-constrained. The long tail lives on small-backups-box, which decides its own retention per source. Losing the local copy is expected and fine.