126 lines
5.6 KiB
Markdown
126 lines
5.6 KiB
Markdown
# `backup_source`
|
|
|
|
Makes a host back **itself** up: dump to stdout, encrypt with `age`, write to a
|
|
local directory, prune, on a systemd timer. `small-backups-box` pulls the
|
|
directory later (see `backup_store`).
|
|
|
|
Modelled on `prd-arbret`, which has been doing exactly this correctly since
|
|
before the rest of the estate was migrated.
|
|
|
|
## Usage
|
|
|
|
```yaml
|
|
- ansible.builtin.include_role:
|
|
name: backup_source
|
|
vars:
|
|
backup_source_name: headscale
|
|
backup_source_description: "Headscale"
|
|
backup_source_dump_command: "tar -czf - -C / var/lib/headscale etc/headscale"
|
|
backup_source_stop_service: headscale
|
|
backup_source_retention_days: 7
|
|
```
|
|
|
|
Produces `/opt/backups/headscale/headscale_<YYYYmmdd_HHMMSS>.tar.gz.age`,
|
|
`headscale-backup.{service,timer}`, and `/usr/local/bin/headscale-backup.sh`.
|
|
|
|
## Why the source encrypts, not the destination
|
|
|
|
`age -r <recipient>` is asymmetric and the host holds only the **public** key, so
|
|
a compromised host cannot read its own backups — or anyone else's. The scripts
|
|
this replaces encrypted with GPG *on the laptop, after the data had already
|
|
crossed the network*, which protects the artefact at rest but not in transit.
|
|
|
|
The matching identity lives only on lapy and is escrowed. **Lose it and every
|
|
artefact everywhere becomes noise**, including arbret's.
|
|
|
|
## `backup_source_dump_command` writes to STDOUT
|
|
|
|
The role pipes it into `age`, so plaintext never touches the disk. Use `-C /`
|
|
with relative paths in `tar` rather than absolute ones: it avoids tar's "removing
|
|
leading /" and makes the restore target explicit.
|
|
|
|
## Services that are not systemd
|
|
|
|
`backup_source_stop_service` runs `systemctl stop/start`. For anything else,
|
|
give the pair explicitly — vaultwarden is a docker compose stack, so
|
|
`systemctl stop vaultwarden` silently does nothing:
|
|
|
|
```yaml
|
|
backup_source_stop_command: "docker compose -f /opt/vaultwarden/docker-compose.yml stop"
|
|
backup_source_start_command: "docker compose -f /opt/vaultwarden/docker-compose.yml start"
|
|
```
|
|
|
|
The same EXIT trap wraps both forms. The assert refuses a stop command without a
|
|
matching start command, because that combination fails in the one way you would
|
|
not notice: the service stops and never comes back.
|
|
|
|
## More than one thing to back up
|
|
|
|
`tar` takes several paths, so multiple files or directories are normally **one**
|
|
artefact — headscale captures `/var/lib/headscale` and `/etc/headscale` together,
|
|
lnbits captures its data directory and its `.env`.
|
|
|
|
Prefer one artefact. A backup should be a consistent snapshot, and two artefacts
|
|
written by two runs can drift — you can end up restoring an `.env` that does not
|
|
match the database it configures. Pulling a single file back out needs no
|
|
unpacking:
|
|
|
|
```bash
|
|
age -d -i <identity> <artefact> | tar -xzO opt/lnbits/lnbits/.env
|
|
```
|
|
|
|
If you genuinely need separate artefacts, call the role twice with different
|
|
`backup_source_name`s rather than extending it — but only one call may set
|
|
`backup_source_stop_service`, or the service is stopped twice per night.
|
|
|
|
The case this shape cannot express is a **database dump plus a file tree**
|
|
(`pg_dump` and a media directory, say): you cannot merge those into one stream
|
|
without staging plaintext on disk, which is exactly what this design avoids.
|
|
None of the current services need it — all are file trees, all stopped for the
|
|
dump. A future one that does should use two role calls.
|
|
|
|
## Everything here is sqlite, so everything stops
|
|
|
|
All five services are sqlite-backed, several in WAL mode (`-wal`/`-shm` files
|
|
present). A live copy of a WAL-mode database can be torn or stale, so each is
|
|
stopped for the duration. Measured downtime: under a second for headscale and
|
|
memos, ~6 s vaultwarden, ~11 s lnbits, and **2m36s for forgejo** — 2.7 G of repos
|
|
and database. That last one is the real cost of a consistent snapshot; if it
|
|
becomes unacceptable the answer is `sqlite3 .backup` plus an online repo copy,
|
|
not skipping the stop.
|
|
|
|
## The trap is the reason this role exists
|
|
|
|
When `backup_source_stop_service` is set, the script stops the unit and installs
|
|
an EXIT trap that starts it again. Without it, a failed dump leaves the service
|
|
down until the next timer fires — **every hand-written script this replaced had
|
|
that bug**, and it was only ever masked because their `systemctl stop` failed
|
|
first, before anything was stopped.
|
|
|
|
Verified on spacey: with the dump forced to fail, the log shows
|
|
`Stopping → Writing → Restarting`, the script exits 1 (so systemd marks the unit
|
|
failed rather than hiding it), and headscale is `active` afterwards.
|
|
|
|
If `systemctl stop` itself fails, `set -e` exits *before* the trap is installed —
|
|
which is correct, because nothing was stopped.
|
|
|
|
## `.partial`
|
|
|
|
The dump writes `<artifact>.partial` and only `mv`s it into place on success, so
|
|
a truncated file is never mistaken for a backup. A failure inside the pipeline
|
|
does leave one behind, and the prune glob cannot match it (it ends `.partial`,
|
|
not `.tar.gz.age`), so the script clears stale partials at the **start** of each
|
|
run. Tested by failing mid-pipeline: 1 partial left, 0 after the next run.
|
|
|
|
## `backup_source_stop_service` may be a bare name
|
|
|
|
`headscale` and `headscale.service` both work. The unit template normalises it,
|
|
because systemd rejects a bare name in `After=` with
|
|
`Failed to add dependency ... Invalid argument` — which it logs and then ignores,
|
|
so the unit appears to work while carrying no ordering at all.
|
|
|
|
## Retention is two-tier
|
|
|
|
`backup_source_retention_days` is **local** and short — these hosts are
|
|
disk-constrained. The long tail lives on `small-backups-box`, which decides its
|
|
own retention per source. Losing the local copy is expected and fine.
|