counterweight/personal_infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

arbret: our side of running it on vipy

Browse source 
prd-arbret was folded into vipy. arbret itself (app, worker, Postgres, timers,
its backup dump) is deployed by its owner's own repo; this repo owns the
shared host's side of it. arbret_handoff.md documents the split, the Gatus
push API and the backup contract.

- services/arbret: arbret.com -> localhost:3100, www redirects to the apex,
  port 4117 allowed on tailscale0 only, /opt/backups/arbret pre-created to
  the contract so the pull job is green before the first dump.
- headscale ACL: framie may reach prd-vipy:4117.
- vipy: arbret-server, arbret-worker and postgresql in the unit checks.
- 402: DNS and HTTP checks for arbret.com, www, and /api/health with
  [BODY].status == ok. Entries may now name an fqdn outside root_domain and
  add conditions.
- 403: probes arbret-analytics (hourly, 130m window) and arbret-beancount
  (daily, 26h), pushing with the dedicated arbret token.
- backups: pull from backup-pull@prd-vipy:/opt/backups/arbret/, and a
  backup-dump_arbret endpoint.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
counterweight 2026-10-19 22:10:26 +02:00
parent 3be5e7934c
commit 5447e03d1f
Signed by: counterweight
GPG key ID: 883EDBAA726BD96C
8 changed files with 554 additions and 8 deletions
Download patch file Download diff file Expand all files Collapse all files

5
ansible/host_vars/vipy/main.yml
View file

@ -13,3 +13,8 @@ monitored_services:
- lnbits
- caddy
- phoenixd
# Deployed by the arbret repo, not this one (see arbret_handoff.md); listed
# here because "what runs on this host" is still a property of the host.
- arbret-server
- arbret-worker
- postgresql

21
ansible/infra/402_public_monitoring.yml
View file

@ -35,6 +35,11 @@
- {sub: "{{ subdomains.monero_explorer }}", host: vipy}
- {sub: "{{ subdomains.music }}", host: vipy}
- {sub: "{{ subdomains.jar_lid_archive_staging }}", host: vipy}
# Not under root_domain, so named in full. arbret.com's zone is NOT in
# Porkbun (mynymdns) - this is the check that would have caught it still
# pointing at prd-arbret's recycled IP after the 2026-10-17 rebuild.
- {fqdn: arbret.com, host: vipy}
- {fqdn: www.arbret.com, host: vipy}
# A public resolver on purpose: this must test what the internet sees, not
# what a local cache or the tailnet's MagicDNS happens to answer.
@ -68,6 +73,13 @@
# tailnet hop, gunicorn, and the database and bucket behind it (503 if
# either is down). "/" would only prove the login redirect works.
- {name: jars-staging, sub: "{{ subdomains.jar_lid_archive_staging }}", path: "/health/", status: 200}
# arbret (runs on vipy since 2026-10-17; deployed by the arbret repo, not
# this one). /api/health answers 200 + "status":"ok" when healthy and 503
# "degraded" otherwise, DB down included - per the arbret owner. The body
# condition guards against a 200 that is not actually healthy.
- {name: arbret, fqdn: arbret.com, path: "/", status: 200}
- {name: arbret-health, fqdn: arbret.com, path: "/api/health", status: 200,
conditions: ['[BODY].status == ok']}
# Ports published from the edge host by socket_proxy.
public_tcp:
@ -107,11 +119,11 @@
- name: Build the DNS endpoints
ansible.builtin.set_fact:
dns_endpoints: "{{ dns_endpoints | default([]) + [{
'name': item.sub ~ '.' ~ root_domain,
'name': (item.fqdn if item.fqdn is defined else item.sub ~ '.' ~ root_domain),
'group': 'dns',
'url': dns_resolver,
'interval': item.interval | default('6h'),
'dns': {'query-type': 'A', 'query-name': item.sub ~ '.' ~ root_domain},
'dns': {'query-type': 'A', 'query-name': (item.fqdn if item.fqdn is defined else item.sub ~ '.' ~ root_domain)},
'conditions': ['[DNS_RCODE] == NOERROR',
'[BODY] == ' ~ hostvars[item.host].ansible_host],
'alerts': [{'type': 'signal', 'failure-threshold': 1,
@ -128,10 +140,11 @@
http_endpoints: "{{ http_endpoints | default([]) + [{
'name': item.name,
'group': 'public',
'url': 'https://' ~ item.sub ~ '.' ~ root_domain ~ item.path,
'url': 'https://' ~ (item.fqdn if item.fqdn is defined else item.sub ~ '.' ~ root_domain) ~ item.path,
'interval': item.interval | default('5m'),
'conditions': ['[STATUS] == ' ~ item.status,
'[CERTIFICATE_EXPIRATION] > 168h'],
'[CERTIFICATE_EXPIRATION] > 168h']
+ item.conditions | default([]),
'alerts': [{'type': 'signal', 'failure-threshold': item.threshold | default(3),
'success-threshold': 2, 'send-on-resolved': true,
'minimum-reminder-interval': '6h'}]}] }}"

9
ansible/infra/403_service_probe_registration.yml
View file

@ -59,6 +59,15 @@
- {name: xmrblocks, host: monerod_box_local}
- {name: navidrome, host: music_box_local}
- {name: jar-lid-archive-staging, host: jar_lid_staging_local}
# arbret (on vipy, deployed by the arbret repo). `host` here is the KEY in
# gatus_push_tokens, and arbret has a dedicated `arbret` token rather than
# vipy's - so it cannot write vipy's own results, and vice versa.
# Windows are two runs plus slack, as for the 5m probes above:
# analytics hourly at :00 -> 130m
# beancount daily at 03:00 -> 26h
# Cadence lives in the arbret repo; tell us if it changes.
- {name: arbret-analytics, host: arbret, heartbeat: '130m'}
- {name: arbret-beancount, host: arbret, heartbeat: '26h'}
# The alert every probe gets. failure-threshold is 1 - the heartbeat window
# is the tolerance, see the note above - unless a probe sets

9
ansible/playbooks/backups.yml
View file

@ -12,8 +12,10 @@
backup_store_check_push_base: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints"
backup_store_check_push_token: "{{ gatus_push_tokens[inventory_hostname] }}"
backup_store_sources:
# On vipy since 2026-10-17 (prd-arbret is gone). The arbret repo writes
# the dump to the backup_source contract - see arbret_handoff.md.
- name: arbret
source: "arbret@prd-arbret:/opt/arbret/backups/"
source: "backup-pull@prd-vipy:/opt/backups/arbret/"
retention_days: 90
- name: headscale
source: "backup-pull@headscale.contrapeso.xyz:/opt/backups/headscale/"
@ -52,8 +54,8 @@
# LATENCY and DIAGNOSIS: the store only learns at the next 04:00 pull, and it
# cannot tell you whether the dump broke or the pull did.
#
# arbret has no dump endpoint: prd-arbret lives in [arbret], which `managed`
# deliberately excludes, so nothing of ours runs there. It is store-checked only.
# arbret's dump is written by the arbret repo, not by backup_source, and it
# pushes with its own `arbret` token (the `host` below is the token key).
# ─────────────────────────────────────────────────────────────────────────────
- name: Register the backup checks with Gatus
hosts: observability
@ -67,6 +69,7 @@
- {name: lnbits, host: vipy}
- {name: forgejo, host: vipy}
- {name: jar-lid-archive-staging, host: jar_lid_staging_local}
- {name: arbret, host: arbret}
store_sources: [arbret, headscale, memos, vaultwarden, lnbits, forgejo, jar-lid-archive-staging]
tasks:

59
ansible/services/arbret/deploy_arbret_edge_playbook.yml Normal file
View file

@ -0,0 +1,59 @@
---
# arbret's footprint on the edge host - and ONLY that.
#
# arbret itself (app, worker, Postgres, timers, its backup dump) is deployed by
# the arbret owner's own Ansible repo since prd-arbret was merged into vipy on
# 2026-10-17. This playbook owns the parts of the shared host that are ours:
# the public front door, the tailnet port, and the backup directory contract.
# See arbret_handoff.md at the repo root for the division of labour.
- name: Publish arbret on the edge host
hosts: edge
become: yes
vars:
# Local port the arbret app binds on 127.0.0.1, per the arbret owner.
arbret_port: 3100
arbret_domain: arbret.com
# Admin port reachable from the tailnet only (ACL in headscale's acl.json.j2).
arbret_tailnet_port: 4117
tasks:
- name: Publish arbret through Caddy
ansible.builtin.include_role:
name: caddy_site
vars:
caddy_site_name: arbret
caddy_site_domain: "{{ arbret_domain }}"
caddy_site_upstream: "localhost:{{ arbret_port }}"
- name: Redirect www to the apex
ansible.builtin.include_role:
name: caddy_site
vars:
caddy_site_name: arbret-www
caddy_site_domain: "www.{{ arbret_domain }}"
caddy_site_body: |
redir https://{{ arbret_domain }}{uri} permanent
# tailscale0 only. UFW's default-deny applies to the tailnet interface too,
# so the headscale ACL alone does not open it - and this must never be
# reachable on the public interface.
- name: Allow the arbret admin port from the tailnet
community.general.ufw:
rule: allow
interface: tailscale0
direction: in
port: "{{ arbret_tailnet_port | string }}"
proto: tcp
comment: "arbret admin, tailnet only"
# The backup contract directory (arbret_handoff.md section 3.3). The arbret
# repo creates it identically; creating it here too means small-backups-box
# can pull - an empty dir - before arbret's first dump, so one missing
# source does not turn the whole pull job red.
- name: Ensure the arbret backup directory exists
ansible.builtin.file:
path: /opt/backups/arbret
state: directory
owner: root
group: backup-pull
mode: "0750"

2
ansible/services/headscale/templates/acl.json.j2
View file

@ -26,7 +26,7 @@
"dst": [
"*:22",
"tag:arbret-staging-box:4117",
"tag:prd-arbret:4117",
"tag:prd-vipy:4117",
"tag:nodito:8006"
],
"src": [

1
ansible/site.yml
View file

@ -65,6 +65,7 @@
- import_playbook: services/forgejo/deploy_forgejo_playbook.yml
- import_playbook: services/lnbits/deploy_lnbits_playbook.yml
- import_playbook: services/personal-blog/deploy_personal_blog_playbook.yml
- import_playbook: services/arbret/deploy_arbret_edge_playbook.yml
# ── Backups: each source dumps itself, the box pulls ────────────────────────
- import_playbook: services/headscale/setup_backup_headscale.yml