age backups everywhere

ansible/roles/backup_source/README.md

@@ -39,6 +39,56 @@ The role pipes it into `age`, so plaintext never touches the disk. Use `-C /`
 with relative paths in `tar` rather than absolute ones: it avoids tar's "removing
 leading /" and makes the restore target explicit.
 
+## Services that are not systemd
+
+`backup_source_stop_service` runs `systemctl stop/start`. For anything else,
+give the pair explicitly — vaultwarden is a docker compose stack, so
+`systemctl stop vaultwarden` silently does nothing:
+
+```yaml
+backup_source_stop_command:  "docker compose -f /opt/vaultwarden/docker-compose.yml stop"
+backup_source_start_command: "docker compose -f /opt/vaultwarden/docker-compose.yml start"
+```
+
+The same EXIT trap wraps both forms. The assert refuses a stop command without a
+matching start command, because that combination fails in the one way you would
+not notice: the service stops and never comes back.
+
+## More than one thing to back up
+
+`tar` takes several paths, so multiple files or directories are normally **one**
+artefact — headscale captures `/var/lib/headscale` and `/etc/headscale` together,
+lnbits captures its data directory and its `.env`.
+
+Prefer one artefact. A backup should be a consistent snapshot, and two artefacts
+written by two runs can drift — you can end up restoring an `.env` that does not
+match the database it configures. Pulling a single file back out needs no
+unpacking:
+
+```bash
+age -d -i <identity> <artefact> | tar -xzO opt/lnbits/lnbits/.env
+```
+
+If you genuinely need separate artefacts, call the role twice with different
+`backup_source_name`s rather than extending it — but only one call may set
+`backup_source_stop_service`, or the service is stopped twice per night.
+
+The case this shape cannot express is a **database dump plus a file tree**
+(`pg_dump` and a media directory, say): you cannot merge those into one stream
+without staging plaintext on disk, which is exactly what this design avoids.
+None of the current services need it — all are file trees, all stopped for the
+dump. A future one that does should use two role calls.
+
+## Everything here is sqlite, so everything stops
+
+All five services are sqlite-backed, several in WAL mode (`-wal`/`-shm` files
+present). A live copy of a WAL-mode database can be torn or stale, so each is
+stopped for the duration. Measured downtime: under a second for headscale and
+memos, ~6 s vaultwarden, ~11 s lnbits, and **2m36s for forgejo** — 2.7 G of repos
+and database. That last one is the real cost of a consistent snapshot; if it
+becomes unacceptable the answer is `sqlite3 .backup` plus an online repo copy,
+not skipping the stop.
+
 ## The trap is the reason this role exists
 
 When `backup_source_stop_service` is set, the script stops the unit and installs

ansible/roles/backup_source/defaults/main.yml

@@ -17,8 +17,12 @@ backup_source_recipient: "{{ age_backup_recipient }}"
 backup_source_pull_user: backup-pull
 backup_source_pull_key: "{{ backup_pull_public_key }}"
 
-# Safety
-backup_source_stop_service: ""          # local unit stopped for the dump, restored by a trap
+# Safety. Give either a systemd unit, or an explicit pair of commands for
+# services that are not systemd-managed (vaultwarden is a docker compose stack).
+# Whichever is used, a trap guarantees the restart.
+backup_source_stop_service: ""          # systemd unit stopped for the dump
+backup_source_stop_command: ""          # overrides stop_service when set
+backup_source_start_command: ""         # required alongside stop_command
 
 # Retention here is LOCAL and short; small-backups-box keeps the long tail.
 backup_source_retention_days: 7

ansible/roles/backup_source/tasks/main.yml

@@ -7,9 +7,11 @@
       - backup_source_dump_command | length > 0
       - backup_source_recipient | length > 0
       - backup_source_recipient is match('^age1[0-9a-z]{58}$')
+      - not (backup_source_stop_command | length > 0 and backup_source_start_command | length == 0)
     fail_msg: >-
       backup_source: '{{ backup_source_name | default("<unnamed>") }}' needs a name,
       description, dump command and a valid age recipient (age1... 62 chars).
+      backup_source_stop_command must be paired with backup_source_start_command.
     quiet: true
 
 # Declared here rather than assumed. Stage 1 installed it by hand; this is what

ansible/roles/backup_source/templates/backup.sh.j2

@@ -11,8 +11,10 @@ RETENTION_DAYS={{ backup_source_retention_days }}
 RECIPIENT="{{ backup_source_recipient }}"
 SUFFIX="{{ backup_source_artifact_suffix }}"
 NAME="{{ backup_source_name }}"
-{% if backup_source_stop_service %}
-SERVICE="{{ backup_source_stop_service }}"
+{% if backup_source_stop_service or backup_source_stop_command %}
+STOP_CMD={{ (backup_source_stop_command or ('systemctl stop ' ~ backup_source_stop_service)) | quote }}
+START_CMD={{ (backup_source_start_command or ('systemctl start ' ~ backup_source_stop_service)) | quote }}
+SERVICE="{{ backup_source_stop_service or backup_source_description }}"   # label for the log only
 {% endif %}
 
 TIMESTAMP=$(date +%Y%m%d_%H%M%S)
@@ -39,13 +41,13 @@ chmod 700 "$BACKUP_DIR"
 # here or they accumulate forever.
 rm -f "${BACKUP_DIR}/${NAME}_"*.partial
 
-{% if backup_source_stop_service %}
+{% if backup_source_stop_service or backup_source_stop_command %}
 # --- Stop the service, and guarantee it comes back ---
 # The trap is the point: without it a failed dump leaves the service down until
 # the next timer fires. Every hand-written script this replaced had that bug.
 log "Stopping ${SERVICE}..."
-systemctl stop "$SERVICE"
-trap 'log "Restarting ${SERVICE}..."; systemctl start "${SERVICE}" || true' EXIT
+eval "$STOP_CMD"
+trap 'log "Restarting ${SERVICE}..."; eval "$START_CMD" || true' EXIT
 {% endif %}
 
 # --- Dump straight into age; plaintext never touches the disk ---