backup stuff

ansible/roles/backup_source/README.md

@@ -0,0 +1,76 @@
+# `backup_source`
+
+Makes a host back **itself** up: dump to stdout, encrypt with `age`, write to a
+local directory, prune, on a systemd timer. `small-backups-box` pulls the
+directory later (see `backup_store`).
+
+Modelled on `prd-arbret`, which has been doing exactly this correctly since
+before the rest of the estate was migrated.
+
+## Usage
+
+```yaml
+- ansible.builtin.include_role:
+    name: backup_source
+  vars:
+    backup_source_name: headscale
+    backup_source_description: "Headscale"
+    backup_source_dump_command: "tar -czf - -C / var/lib/headscale etc/headscale"
+    backup_source_stop_service: headscale
+    backup_source_retention_days: 7
+```
+
+Produces `/opt/backups/headscale/headscale_<YYYYmmdd_HHMMSS>.tar.gz.age`,
+`headscale-backup.{service,timer}`, and `/usr/local/bin/headscale-backup.sh`.
+
+## Why the source encrypts, not the destination
+
+`age -r <recipient>` is asymmetric and the host holds only the **public** key, so
+a compromised host cannot read its own backups — or anyone else's. The scripts
+this replaces encrypted with GPG *on the laptop, after the data had already
+crossed the network*, which protects the artefact at rest but not in transit.
+
+The matching identity lives only on lapy and is escrowed. **Lose it and every
+artefact everywhere becomes noise**, including arbret's.
+
+## `backup_source_dump_command` writes to STDOUT
+
+The role pipes it into `age`, so plaintext never touches the disk. Use `-C /`
+with relative paths in `tar` rather than absolute ones: it avoids tar's "removing
+leading /" and makes the restore target explicit.
+
+## The trap is the reason this role exists
+
+When `backup_source_stop_service` is set, the script stops the unit and installs
+an EXIT trap that starts it again. Without it, a failed dump leaves the service
+down until the next timer fires — **every hand-written script this replaced had
+that bug**, and it was only ever masked because their `systemctl stop` failed
+first, before anything was stopped.
+
+Verified on spacey: with the dump forced to fail, the log shows
+`Stopping → Writing → Restarting`, the script exits 1 (so systemd marks the unit
+failed rather than hiding it), and headscale is `active` afterwards.
+
+If `systemctl stop` itself fails, `set -e` exits *before* the trap is installed —
+which is correct, because nothing was stopped.
+
+## `.partial`
+
+The dump writes `<artifact>.partial` and only `mv`s it into place on success, so
+a truncated file is never mistaken for a backup. A failure inside the pipeline
+does leave one behind, and the prune glob cannot match it (it ends `.partial`,
+not `.tar.gz.age`), so the script clears stale partials at the **start** of each
+run. Tested by failing mid-pipeline: 1 partial left, 0 after the next run.
+
+## `backup_source_stop_service` may be a bare name
+
+`headscale` and `headscale.service` both work. The unit template normalises it,
+because systemd rejects a bare name in `After=` with
+`Failed to add dependency ... Invalid argument` — which it logs and then ignores,
+so the unit appears to work while carrying no ordering at all.
+
+## Retention is two-tier
+
+`backup_source_retention_days` is **local** and short — these hosts are
+disk-constrained. The long tail lives on `small-backups-box`, which decides its
+own retention per source. Losing the local copy is expected and fine.

ansible/roles/backup_source/defaults/main.yml

@@ -0,0 +1,27 @@
+---
+# Required
+backup_source_name: ""                  # "headscale" -> headscale_<ts>.tar.gz.age
+backup_source_description: ""           # "Headscale"
+backup_source_dump_command: ""          # must write the payload to STDOUT
+
+# Placement
+backup_source_dir: "/opt/backups/{{ backup_source_name }}"
+backup_source_artifact_suffix: "tar.gz.age"
+
+# Encryption. Asymmetric: the host holds only the public key and cannot decrypt
+# what it produces.
+backup_source_recipient: "{{ age_backup_recipient }}"
+
+# The unprivileged account small-backups-box pulls as. It owns the dump
+# directory and nothing else; it deliberately has no sudo.
+backup_source_pull_user: backup-pull
+backup_source_pull_key: "{{ backup_pull_public_key }}"
+
+# Safety
+backup_source_stop_service: ""          # local unit stopped for the dump, restored by a trap
+
+# Retention here is LOCAL and short; small-backups-box keeps the long tail.
+backup_source_retention_days: 7
+
+# Schedule. The box pulls at 04:00, so dumps must land before that.
+backup_source_on_calendar: "*-*-* 02:00:00"

ansible/roles/backup_source/handlers/main.yml

@@ -0,0 +1,4 @@
+---
+- name: Reload systemd for backup units
+  ansible.builtin.systemd:
+    daemon_reload: yes

ansible/roles/backup_source/tasks/main.yml

@@ -0,0 +1,89 @@
+---
+- name: Assert backup_source parameters are sane
+  ansible.builtin.assert:
+    that:
+      - backup_source_name | length > 0
+      - backup_source_description | length > 0
+      - backup_source_dump_command | length > 0
+      - backup_source_recipient | length > 0
+      - backup_source_recipient is match('^age1[0-9a-z]{58}$')
+    fail_msg: >-
+      backup_source: '{{ backup_source_name | default("<unnamed>") }}' needs a name,
+      description, dump command and a valid age recipient (age1... 62 chars).
+    quiet: true
+
+# Declared here rather than assumed. Stage 1 installed it by hand; this is what
+# makes a rebuilt host get it too.
+- name: Ensure age is installed
+  ansible.builtin.apt:
+    name: age
+    state: present
+    update_cache: yes
+    cache_valid_time: 3600
+
+# The pull account: unprivileged, no sudo, exists only so small-backups-box can
+# read the dump directory. Trust points one way — the box can read backups, and
+# can do nothing else on this host.
+- name: "Ensure the {{ backup_source_pull_user }} account exists"
+  ansible.builtin.user:
+    name: "{{ backup_source_pull_user }}"
+    system: yes
+    shell: /bin/sh          # rsync-over-ssh needs a shell; nologin breaks it
+    home: "/var/lib/{{ backup_source_pull_user }}"
+    create_home: yes
+    password: '!'           # no password login, ever
+  when: backup_source_pull_user | length > 0
+
+- name: "Authorise the backup box's key for {{ backup_source_pull_user }}"
+  ansible.posix.authorized_key:
+    user: "{{ backup_source_pull_user }}"
+    key: "{{ backup_source_pull_key }}"
+    key_options: "restrict"   # no pty, no forwarding, no user rc
+    exclusive: yes
+    state: present
+  when: backup_source_pull_user | length > 0
+
+# The shared container above the per-service directories. It must be traversable
+# or the pull account cannot reach its own directory. The script's `mkdir -p`
+# runs under `umask 077` and would otherwise create this 0700.
+- name: "Ensure {{ backup_source_dir | dirname }} is traversable"
+  ansible.builtin.file:
+    path: "{{ backup_source_dir | dirname }}"
+    state: directory
+    owner: root
+    group: root
+    mode: '0755'
+
+- name: "Ensure {{ backup_source_dir }} exists"
+  ansible.builtin.file:
+    path: "{{ backup_source_dir }}"
+    state: directory
+    owner: root
+    group: "{{ backup_source_pull_user | default('root', true) }}"
+    mode: '0750'
+
+- name: "Install the {{ backup_source_name }} backup script"
+  ansible.builtin.template:
+    src: backup.sh.j2
+    dest: "/usr/local/bin/{{ backup_source_name }}-backup.sh"
+    owner: root
+    group: root
+    mode: '0750'
+    validate: "bash -n %s"
+
+- name: "Install the {{ backup_source_name }}-backup systemd units"
+  ansible.builtin.template:
+    src: "backup.{{ item }}.j2"
+    dest: "/etc/systemd/system/{{ backup_source_name }}-backup.{{ item }}"
+    owner: root
+    group: root
+    mode: '0644'
+  loop: [service, timer]
+  notify: Reload systemd for backup units
+
+- name: "Enable the {{ backup_source_name }}-backup timer"
+  ansible.builtin.systemd:
+    name: "{{ backup_source_name }}-backup.timer"
+    enabled: yes
+    state: started
+    daemon_reload: yes

ansible/roles/backup_source/templates/backup.service.j2

@@ -0,0 +1,14 @@
+[Unit]
+Description={{ backup_source_description }} backup
+{% if backup_source_stop_service %}
+{# systemd rejects a bare name here ("Failed to add dependency ... Invalid
+   argument"), so normalise to a full unit name. #}
+After={{ backup_source_stop_service if '.' in backup_source_stop_service else backup_source_stop_service ~ '.service' }}
+{% endif %}
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/bin/{{ backup_source_name }}-backup.sh
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier={{ backup_source_name }}-backup

ansible/roles/backup_source/templates/backup.sh.j2

@@ -0,0 +1,69 @@
+#!/usr/bin/env bash
+# {{ backup_source_description }} backup — managed by Ansible (roles/backup_source)
+#
+# Dumps to stdout, encrypts with age, writes {{ backup_source_dir }}.
+# The host holds only the age PUBLIC key, so it cannot read its own backups.
+set -euo pipefail
+umask 077
+
+BACKUP_DIR="{{ backup_source_dir }}"
+RETENTION_DAYS={{ backup_source_retention_days }}
+RECIPIENT="{{ backup_source_recipient }}"
+SUFFIX="{{ backup_source_artifact_suffix }}"
+NAME="{{ backup_source_name }}"
+{% if backup_source_stop_service %}
+SERVICE="{{ backup_source_stop_service }}"
+{% endif %}
+
+TIMESTAMP=$(date +%Y%m%d_%H%M%S)
+ARTIFACT="${BACKUP_DIR}/${NAME}_${TIMESTAMP}.${SUFFIX}"
+
+die() { echo "FATAL: $*" >&2; exit 1; }
+log() { echo "$(date '+%Y-%m-%d %H:%M:%S') $*"; }
+
+# --- Pre-flight ---
+[[ -n "$RECIPIENT" ]] || die "no age recipient configured"
+command -v age >/dev/null || die "age is not installed"
+
+# Mode must agree with what the role sets, or each undoes the other every run.
+mkdir -p "$BACKUP_DIR"
+{% if backup_source_pull_user %}
+chown root:{{ backup_source_pull_user }} "$BACKUP_DIR"
+chmod 750 "$BACKUP_DIR"
+{% else %}
+chmod 700 "$BACKUP_DIR"
+{% endif %}
+
+# A run that died mid-dump leaves a .partial. It is not a backup, and the prune
+# glob below cannot match it (it ends .partial, not .${SUFFIX}), so clear them
+# here or they accumulate forever.
+rm -f "${BACKUP_DIR}/${NAME}_"*.partial
+
+{% if backup_source_stop_service %}
+# --- Stop the service, and guarantee it comes back ---
+# The trap is the point: without it a failed dump leaves the service down until
+# the next timer fires. Every hand-written script this replaced had that bug.
+log "Stopping ${SERVICE}..."
+systemctl stop "$SERVICE"
+trap 'log "Restarting ${SERVICE}..."; systemctl start "${SERVICE}" || true' EXIT
+{% endif %}
+
+# --- Dump straight into age; plaintext never touches the disk ---
+log "Writing ${ARTIFACT}..."
+{{ backup_source_dump_command }} | age -r "$RECIPIENT" -o "${ARTIFACT}.partial"
+mv "${ARTIFACT}.partial" "$ARTIFACT"
+{% if backup_source_pull_user %}
+# Readable by the pull account and nobody else. The contents are age-encrypted
+# regardless, so this is depth rather than the actual protection.
+chown root:{{ backup_source_pull_user }} "$ARTIFACT"
+chmod 640 "$ARTIFACT"
+{% else %}
+chmod 600 "$ARTIFACT"
+{% endif %}
+log "Wrote ${ARTIFACT} ($(du -h "$ARTIFACT" | cut -f1))"
+
+# --- Prune ---
+log "Pruning local artefacts older than ${RETENTION_DAYS} days..."
+find "$BACKUP_DIR" -maxdepth 1 -type f -name "${NAME}_*.${SUFFIX}" -mtime +"${RETENTION_DAYS}" -delete
+
+log "Done."

ansible/roles/backup_source/templates/backup.timer.j2

@@ -0,0 +1,11 @@
+[Unit]
+Description={{ backup_source_description }} backup
+
+[Timer]
+OnCalendar={{ backup_source_on_calendar }}
+# Persistent: a window missed while the host was down runs on next boot. cron on
+# a laptop had no equivalent, which is how two backups went unnoticed for months.
+Persistent=true
+
+[Install]
+WantedBy=timers.target