backups: monitor both the dump and the pull, per source

Twelve endpoints in two groups, because they answer different questions and
fail for different reasons:

  backup-dump_<svc>       pushed by the SOURCE right after its dump runs
  backup-store_<svc>      pushed by the BOX at 05:30, per source
  backup-store_pull-job   pushed by the BOX, about the box itself

The store alone could catch almost everything, because the artefact filename
carries the source's dump timestamp - a source whose timer died still pulls
"ok" forever, but the timestamp gives it away. What the source side adds is
LATENCY and DIAGNOSIS: the store only learns at the next 04:00 pull, and it
cannot tell you whether the dump broke or the pull did.

pull-job is separate from the per-source checks because it is a LEADING
indicator where those are lagging ones. A disabled pull timer, a failed pull
job, or a filling disk are all visible immediately, while the per-source checks
only fire once an artefact is >26h stale - about a day later. Disable the timer
at 10:00 and every source stays green until tomorrow; pull-job goes red this
morning and names the cause instead of showing six stale sources.

Frequencies: dumps 02:00-02:30 staggered, pull 04:00, verification 05:30, all
daily and Persistent. 26h staleness decides red; a 30h Gatus heartbeat catches
the verification itself having stopped, so a dead check-backups.timer cannot
hide a stale backup.

arbret has no dump endpoint: prd-arbret is in [arbret], which `managed`
deliberately excludes, so nothing of ours runs there. Store-checked only.

check-backups.sh was manual-only; it now runs on a timer and reports per source
rather than only printing. The human-readable report is unchanged.

A SERIOUS bug introduced and fixed in this change, recorded because the shape
is easy to repeat: the reporting hook was added to backup.sh as a second
`trap ... EXIT`. Bash REPLACES the EXIT handler rather than adding to it, so
that silently deleted the trap which restarts the stopped service - the one the
script's own comment calls "the point", and the bug the role was written to
eliminate. Every backup then stopped its service and left it stopped. It took
forgejo, lnbits, headscale and memos down for several minutes each, and nothing
caught it: the dumps exit 0, the artefacts are correct, the deploy reports
failed=0, and liveness only proves the HOST is up. There is now ONE EXIT
handler doing both jobs, armed BEFORE the stop so a failure during the stop
still restarts. Verified by rendering both variants, asserting exactly one EXIT
trap in each, and simulating a mid-way failure to confirm the restart fires.

Verified: all 12 endpoints UP; six sources pulled cleanly (arbret 31M,
headscale 198K, memos 7.8M, vaultwarden 2.1M, lnbits 30M, forgejo 2.6G), store
16% full, "RESULT: all checks passed".

Known gaps, deliberately not closed here:
  * `yell` warnings - disk 75-90%, an artefact under half the previous size,
    retention not pruning - never reach Gatus, because a push is binary.
  * Nothing verifies a backed-up service came back UP. That is the gap that let
    the trap bug run unnoticed, and it is what the next change addresses.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ansible/roles/backup_source/defaults/main.yml

@@ -29,3 +29,16 @@ backup_source_retention_days: 7
 
 # Schedule. The box pulls at 04:00, so dumps must land before that.
 backup_source_on_calendar: "*-*-* 02:00:00"
+
+# ── Reporting ────────────────────────────────────────────────────────────────
+# Where to report that this dump ran and produced a plausible artefact.
+# Gatus external endpoint:
+#     POST {url}?success=true|false&error=...
+#     Authorization: Bearer {token}
+# Empty is valid and is not an error: the unit's exit code is still the answer,
+# and the STORE will independently notice a stale dump within ~26h because the
+# artefact filename carries this dump's timestamp. Reporting here only buys
+# earlier detection and tells you it was the DUMP that broke rather than the
+# pull.
+backup_source_push_url: ""
+backup_source_push_token: ""

ansible/roles/backup_source/tasks/main.yml

@@ -30,7 +30,12 @@
 
 - name: Ensure age is installed
   ansible.builtin.apt:
-    name: age
+    name:
+      - age
+      # curl is needed only when backup_source_push_url is set, but installing it
+      # unconditionally keeps the task idempotent and it is present on every
+      # Debian host here anyway.
+      - curl
     state: present
 
 # The pull account: unprivileged, no sudo, exists only so small-backups-box can
@@ -83,14 +88,18 @@
     mode: '0750'
     validate: "bash -n %s"
 
+# The .service carries the push token in an Environment= line, so it is 0600.
+# The .timer holds nothing secret and stays world-readable.
 - name: "Install the {{ backup_source_name }}-backup systemd units"
   ansible.builtin.template:
-    src: "backup.{{ item }}.j2"
-    dest: "/etc/systemd/system/{{ backup_source_name }}-backup.{{ item }}"
+    src: "backup.{{ item.unit }}.j2"
+    dest: "/etc/systemd/system/{{ backup_source_name }}-backup.{{ item.unit }}"
     owner: root
     group: root
-    mode: '0644'
-  loop: [service, timer]
+    mode: "{{ item.mode }}"
+  loop:
+    - {unit: service, mode: "0600"}
+    - {unit: timer,   mode: "0644"}
   notify: Reload systemd for backup units
 
 - name: "Enable the {{ backup_source_name }}-backup timer"

ansible/roles/backup_source/templates/backup.service.j2

@@ -9,6 +9,8 @@ After={{ backup_source_stop_service if '.' in backup_source_stop_service else ba
 [Service]
 Type=oneshot
 ExecStart=/usr/local/bin/{{ backup_source_name }}-backup.sh
+Environment=BACKUP_PUSH_URL={{ backup_source_push_url }}
+Environment=BACKUP_PUSH_TOKEN={{ backup_source_push_token }}
 StandardOutput=journal
 StandardError=journal
 SyslogIdentifier={{ backup_source_name }}-backup

ansible/roles/backup_source/templates/backup.sh.j2

@@ -41,13 +41,65 @@ chmod 700 "$BACKUP_DIR"
 # here or they accumulate forever.
 rm -f "${BACKUP_DIR}/${NAME}_"*.partial
 
+# --- Reporting -------------------------------------------------------------
+# A dump that exits non-zero, or that produces a zero-byte artefact, is a failed
+# backup even though the script "finished". Both are reported as failures.
+PUSH_URL="${BACKUP_PUSH_URL:-}"
+PUSH_TOKEN="${BACKUP_PUSH_TOKEN:-}"
+
+report() {
+    local success="$1" message="$2"
+    [ -n "$PUSH_URL" ] || return 0
+    local encoded
+    encoded=$(printf '%s' "$message" | sed 's/%/%25/g; s/ /%20/g; s/&/%26/g; s/+/%2B/g; s/#/%23/g')
+    curl -s -o /dev/null --max-time 15 --retry 2 --retry-delay 3 -X POST \
+        -H "Authorization: Bearer ${PUSH_TOKEN}" \
+        "${PUSH_URL}?success=${success}&error=${encoded}" 2>/dev/null || true
+}
+
+# Reports on ANY exit path, so a dump that dies halfway still reports rather
+# than going quiet. The size of the FINISHED artefact decides success, not
+# merely reaching the end of the script.
+#
+# This is called FROM the single EXIT trap below - it must never register an
+# EXIT trap of its own. `trap ... EXIT` REPLACES the existing handler rather
+# than adding to it, so a second trap here silently discards the one that
+# restarts the service, and a backup run leaves the service stopped. That is
+# precisely the failure the restart trap exists to prevent.
+report_outcome() {
+    local rc="$1"
+    if [ "$rc" -ne 0 ]; then
+        report "false" "${NAME} dump exited ${rc}"
+    elif [ ! -s "$ARTIFACT" ]; then
+        report "false" "${NAME} produced no artefact at ${ARTIFACT}"
+    else
+        report "true" "${NAME} $(du -h "$ARTIFACT" | cut -f1)"
+    fi
+}
+
+# --- One EXIT handler, doing both jobs -------------------------------------
+# bash keeps exactly ONE EXIT trap: `trap ... EXIT` REPLACES the previous
+# handler rather than adding to it. Registering a second one here would
+# silently discard the service restart and leave the service stopped after
+# every backup - which is the exact bug the restart exists to prevent, and it
+# is invisible until someone notices the service is down.
+on_exit() {
+    local rc=$?
 {% if backup_source_stop_service or backup_source_stop_command %}
-# --- Stop the service, and guarantee it comes back ---
+    log "Restarting ${SERVICE}..."
+    eval "$START_CMD" || true
+{% endif %}
+    report_outcome "$rc"
+}
+trap on_exit EXIT
+
+{% if backup_source_stop_service or backup_source_stop_command %}
+# --- Stop the service; the trap above guarantees it comes back -------------
 # The trap is the point: without it a failed dump leaves the service down until
 # the next timer fires. Every hand-written script this replaced had that bug.
+# It is armed BEFORE the stop, so even a failure during the stop restarts.
 log "Stopping ${SERVICE}..."
 eval "$STOP_CMD"
-trap 'log "Restarting ${SERVICE}..."; eval "$START_CMD" || true' EXIT
 {% endif %}
 
 # --- Dump straight into age; plaintext never touches the disk ---