personal_infra/ansible/playbooks/backups.yml

96 lines
4.2 KiB
YAML
Raw Normal View History

2026-09-12 16:02:00 +02:00
- name: Configure the offsite backup pull
hosts: backup_store
gather_facts: yes
tasks:
- name: Ensure the box pulls every source on a timer
ansible.builtin.include_role:
name: backup_store
vars:
backups: monitor both the dump and the pull, per source Twelve endpoints in two groups, because they answer different questions and fail for different reasons: backup-dump_<svc> pushed by the SOURCE right after its dump runs backup-store_<svc> pushed by the BOX at 05:30, per source backup-store_pull-job pushed by the BOX, about the box itself The store alone could catch almost everything, because the artefact filename carries the source's dump timestamp - a source whose timer died still pulls "ok" forever, but the timestamp gives it away. What the source side adds is LATENCY and DIAGNOSIS: the store only learns at the next 04:00 pull, and it cannot tell you whether the dump broke or the pull did. pull-job is separate from the per-source checks because it is a LEADING indicator where those are lagging ones. A disabled pull timer, a failed pull job, or a filling disk are all visible immediately, while the per-source checks only fire once an artefact is >26h stale - about a day later. Disable the timer at 10:00 and every source stays green until tomorrow; pull-job goes red this morning and names the cause instead of showing six stale sources. Frequencies: dumps 02:00-02:30 staggered, pull 04:00, verification 05:30, all daily and Persistent. 26h staleness decides red; a 30h Gatus heartbeat catches the verification itself having stopped, so a dead check-backups.timer cannot hide a stale backup. arbret has no dump endpoint: prd-arbret is in [arbret], which `managed` deliberately excludes, so nothing of ours runs there. Store-checked only. check-backups.sh was manual-only; it now runs on a timer and reports per source rather than only printing. The human-readable report is unchanged. A SERIOUS bug introduced and fixed in this change, recorded because the shape is easy to repeat: the reporting hook was added to backup.sh as a second `trap ... EXIT`. Bash REPLACES the EXIT handler rather than adding to it, so that silently deleted the trap which restarts the stopped service - the one the script's own comment calls "the point", and the bug the role was written to eliminate. Every backup then stopped its service and left it stopped. It took forgejo, lnbits, headscale and memos down for several minutes each, and nothing caught it: the dumps exit 0, the artefacts are correct, the deploy reports failed=0, and liveness only proves the HOST is up. There is now ONE EXIT handler doing both jobs, armed BEFORE the stop so a failure during the stop still restarts. Verified by rendering both variants, asserting exactly one EXIT trap in each, and simulating a mid-way failure to confirm the restart fires. Verified: all 12 endpoints UP; six sources pulled cleanly (arbret 31M, headscale 198K, memos 7.8M, vaultwarden 2.1M, lnbits 30M, forgejo 2.6G), store 16% full, "RESULT: all checks passed". Known gaps, deliberately not closed here: * `yell` warnings - disk 75-90%, an artefact under half the previous size, retention not pruning - never reach Gatus, because a push is binary. * Nothing verifies a backed-up service came back UP. That is the gap that let the trap bug run unnoticed, and it is what the next change addresses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-14 09:22:12 +02:00
# check-backups.sh reports one result per source plus one for the store
# itself, so it needs the collection URL and appends each key.
backup_store_check_push_base: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints"
backup_store_check_push_token: "{{ gatus_push_tokens[inventory_hostname] }}"
2026-09-12 16:02:00 +02:00
backup_store_sources:
- name: arbret
source: "arbret@prd-arbret:/opt/arbret/backups/"
retention_days: 90
- name: headscale
source: "backup-pull@headscale.contrapeso.xyz:/opt/backups/headscale/"
retention_days: 90
2026-09-12 16:20:42 +02:00
- name: memos
source: "backup-pull@memos-box:/opt/backups/memos/"
retention_days: 90
- name: vaultwarden
source: "backup-pull@prd-vipy:/opt/backups/vaultwarden/"
retention_days: 90
- name: lnbits
source: "backup-pull@prd-vipy:/opt/backups/lnbits/"
retention_days: 90
- name: forgejo
source: "backup-pull@prd-vipy:/opt/backups/forgejo/"
retention_days: 14
backups: monitor both the dump and the pull, per source Twelve endpoints in two groups, because they answer different questions and fail for different reasons: backup-dump_<svc> pushed by the SOURCE right after its dump runs backup-store_<svc> pushed by the BOX at 05:30, per source backup-store_pull-job pushed by the BOX, about the box itself The store alone could catch almost everything, because the artefact filename carries the source's dump timestamp - a source whose timer died still pulls "ok" forever, but the timestamp gives it away. What the source side adds is LATENCY and DIAGNOSIS: the store only learns at the next 04:00 pull, and it cannot tell you whether the dump broke or the pull did. pull-job is separate from the per-source checks because it is a LEADING indicator where those are lagging ones. A disabled pull timer, a failed pull job, or a filling disk are all visible immediately, while the per-source checks only fire once an artefact is >26h stale - about a day later. Disable the timer at 10:00 and every source stays green until tomorrow; pull-job goes red this morning and names the cause instead of showing six stale sources. Frequencies: dumps 02:00-02:30 staggered, pull 04:00, verification 05:30, all daily and Persistent. 26h staleness decides red; a 30h Gatus heartbeat catches the verification itself having stopped, so a dead check-backups.timer cannot hide a stale backup. arbret has no dump endpoint: prd-arbret is in [arbret], which `managed` deliberately excludes, so nothing of ours runs there. Store-checked only. check-backups.sh was manual-only; it now runs on a timer and reports per source rather than only printing. The human-readable report is unchanged. A SERIOUS bug introduced and fixed in this change, recorded because the shape is easy to repeat: the reporting hook was added to backup.sh as a second `trap ... EXIT`. Bash REPLACES the EXIT handler rather than adding to it, so that silently deleted the trap which restarts the stopped service - the one the script's own comment calls "the point", and the bug the role was written to eliminate. Every backup then stopped its service and left it stopped. It took forgejo, lnbits, headscale and memos down for several minutes each, and nothing caught it: the dumps exit 0, the artefacts are correct, the deploy reports failed=0, and liveness only proves the HOST is up. There is now ONE EXIT handler doing both jobs, armed BEFORE the stop so a failure during the stop still restarts. Verified by rendering both variants, asserting exactly one EXIT trap in each, and simulating a mid-way failure to confirm the restart fires. Verified: all 12 endpoints UP; six sources pulled cleanly (arbret 31M, headscale 198K, memos 7.8M, vaultwarden 2.1M, lnbits 30M, forgejo 2.6G), store 16% full, "RESULT: all checks passed". Known gaps, deliberately not closed here: * `yell` warnings - disk 75-90%, an artefact under half the previous size, retention not pruning - never reach Gatus, because a push is binary. * Nothing verifies a backed-up service came back UP. That is the gap that let the trap bug run unnoticed, and it is what the next change addresses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-14 09:22:12 +02:00
# ─────────────────────────────────────────────────────────────────────────────
# Register the backup checks with Gatus.
#
# Two groups on purpose, because they answer different questions and fail for
# different reasons:
#
# backup-dump did the SOURCE produce an artefact? Pushed by each dump right
# after it runs, so a broken dump is visible within minutes.
# backup-store did it ARRIVE, is it fresh, non-zero, plausibly sized, and is
# retention pruning? Pushed by check-backups.sh at 05:30.
#
# The store alone could catch almost everything, because the artefact filename
# carries the source's dump timestamp - a source whose timer died still pulls
# "ok" forever, but the timestamp gives it away. What the source side adds is
# LATENCY and DIAGNOSIS: the store only learns at the next 04:00 pull, and it
# cannot tell you whether the dump broke or the pull did.
#
# arbret has no dump endpoint: prd-arbret lives in [arbret], which `managed`
# deliberately excludes, so nothing of ours runs there. It is store-checked only.
# ─────────────────────────────────────────────────────────────────────────────
- name: Register the backup checks with Gatus
hosts: observability
become: yes
vars:
# Sources we deploy the dump for, and the host each one runs on.
dump_sources:
- {name: headscale, host: spacey}
- {name: memos, host: memos_box_local}
- {name: vaultwarden, host: vipy}
- {name: lnbits, host: vipy}
- {name: forgejo, host: vipy}
store_sources: [arbret, headscale, memos, vaultwarden, lnbits, forgejo]
tasks:
- name: Build the dump endpoint list
ansible.builtin.set_fact:
dump_endpoints: "{{ dump_endpoints | default([]) + [{
'name': item.name,
'group': 'backup-dump',
'token': gatus_push_tokens[item.host],
'heartbeat': '30h'}] }}"
loop: "{{ dump_sources }}"
- name: Build the store endpoint list
ansible.builtin.set_fact:
store_endpoints: "{{ store_endpoints | default([]) + [{
'name': item,
'group': 'backup-store',
'token': gatus_push_tokens['small_backups_local'],
'heartbeat': '30h'}] }}"
loop: "{{ store_sources }}"
- name: Register the backup endpoints
ansible.builtin.include_role:
name: gatus_endpoint
vars:
gatus_endpoint_name: backups
gatus_endpoint_external: "{{ dump_endpoints + store_endpoints + [{
'name': 'pull job',
'group': 'backup-store',
'token': gatus_push_tokens['small_backups_local'],
'heartbeat': '30h'}] }}"