mempool: convert to a role, de-Uptime-Kuma the health checks

745-line playbook becomes 37 lines (the role, plus the Caddy play for the edge
host) and a 408-line role with docker/deploy/healthcheck phases and six
templates. mempool_vars.yml is deleted; its content is the role's defaults.

Three health checks are kept, not collapsed: Mempool is three moving parts and
knowing which one is down is the point. Each has its own script, unit, timer and
push_url, driven by a mempool_healthchecks list. The Uptime Kuma specifics are
gone - the embedded Python creating monitors over the API, the /tmp credentials
file, the push-URL file read back and parsed, three Environment= rewrites - and
the three live push URLs are preserved from the vault, so reporting is unchanged.

`Enable and start health check timers` and `Display deployment status` were both
guarded by uptime_kuma_enabled despite being deployment tasks. Third service in
a row with that pattern: the deprecation banner was applied to contiguous blocks,
so anything sitting near the push plumbing was disabled with it. Ungated.

TWO OWNERSHIP PROBLEMS, different in kind:

- MINE: I wrote `owner: root` on docker-compose.yml where the original says
  `owner: "{{ ansible_user }}"`. A straight violation of extract-mechanically-
  change-nothing, caught only by reading the check-mode diff line by line.
  Reverted to match the original.

- PRE-EXISTING, and dangerous: the playbook declared
  `owner: "{{ ansible_user }}"` (1000) on the MariaDB data directory, which the
  container owns as uid 999. Confirmed against `git show HEAD:` before
  concluding it was not mine. It had drifted since the containers were created
  and went unnoticed because the playbook had not been run since.

  This was not academic. The first real run pulled a newer mariadb:10.11 and
  recreated mempool-db; with the chown still in place MariaDB would have come
  back to a data directory it could not write. The role now ensures the
  directory exists and leaves ownership to the container. Verified after the
  run: /opt/mempool/mysql is still 999:999 and all three containers are healthy.

  This is a deliberate behaviour change, not part of the extraction. It is in
  this commit rather than a follow-up because the faithful version was never
  safe to run, so there was no intermediate state worth recording as verified.

mempool_frontend_port moved to services_config.yml: two hosts need it (this role
deploys the frontend, the Caddy play proxies to it from the edge host) and a role
default is invisible to the second play. caddy_site's parameter assert caught
this loudly - "'mempool_frontend_port' is undefined" - rather than silently.

Verified: check-mode diff clean apart from unavoidable check-mode artifacts;
first run ok=24 changed=5, zero failures; second run changed=2 - the two bare
`command:` tasks (pull, compose up) that have no changed_when and always report
changed. That is the idempotent floor. All three health checks report
ExecMainStatus 0 with their push URLs intact.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ansible/roles/mempool/README.md

@@ -0,0 +1,69 @@
+# `mempool`
+
+Deploys the [Mempool](https://mempool.space) block explorer as a three-container
+Docker Compose stack — MariaDB, backend, frontend — on `mempool-box`, and keeps a
+health check on each.
+
+Converted from `deploy_mempool_playbook.yml` (745 lines) under Plan 6. The
+playbook is now 37 lines: this role, plus a second play that publishes the
+frontend through Caddy on the edge host.
+
+## Phases
+
+| | |
+|---|---|
+| `docker.yml` | Docker engine: repo, key, packages, service |
+| `deploy.yml` | directories, `docker-compose.yml`, pull, up, wait-for-healthy |
+| `healthcheck.yml` | three check scripts, three services, three timers |
+
+## Three health checks, not one
+
+Mempool is three moving parts and knowing *which* one is down is the point, so
+each gets its own check, unit and timer, driven by the `mempool_healthchecks`
+list:
+
+| | checks |
+|---|---|
+| `mariadb` | `docker inspect` health status of `mempool-db` |
+| `backend` | `GET /api/v1/backend-info` |
+| `frontend` | `GET /` |
+
+Each records its answer in its exit code, which systemd keeps:
+`systemctl is-failed mempool-backend-healthcheck.service`. Reporting elsewhere
+is one field per check, `push_url`, and is the plug-in point for whatever
+monitoring exists. Empty means check, exit honestly, report nowhere. The URLs
+are credentials, so callers pass them from the vault.
+
+Nothing here is specific to a monitoring product. The embedded Python that
+created monitors over the Uptime Kuma API, the `/tmp` credentials file, the
+push-URL file read back and parsed, and three systemd `Environment=` rewrites
+are gone.
+
+## MariaDB owns its own data directory
+
+`{{ mempool_mysql_dir }}` is bind-mounted into the container, which runs as uid
+**999** and must create files there. The playbook this replaced declared
+`owner: "{{ ansible_user }}"` (1000) on it, which had drifted from reality ever
+since the containers were created — unnoticed, because the playbook had not been
+run since.
+
+That was not academic. The first real run of this role pulled a newer
+`mariadb:10.11` and recreated `mempool-db`; had the chown still been in place,
+MariaDB would have come back to a directory it could not write. The role now
+ensures the directory exists and leaves ownership to the container.
+
+## `mempool_frontend_port` lives in `services_config.yml`
+
+Two hosts need it: this role deploys the frontend on `mempool-box`, and the Caddy
+play proxies to it from the edge host. A role default is invisible to the second
+play, so the value lives in `service_settings.mempool.frontend_port` and the role
+default derives from it.
+
+## Expect `changed=2` on a converged host
+
+`Pull Mempool images` and `Deploy Mempool containers with docker compose` are
+bare `command:` tasks with no `changed_when`, so they always report changed.
+That is the idempotent floor, not drift. Everything else reports `ok`.
+
+**`mariadb:10.11` is a moving tag**, so a run can pull a newer patch release and
+recreate the database container. Pin it if that is not what you want.

ansible/roles/mempool/defaults/main.yml

@@ -0,0 +1,55 @@
+# Mempool Configuration Variables
+
+# Version - Pinned to specific release
+mempool_version: "v3.2.1"
+
+# Directories
+mempool_dir: /opt/mempool
+mempool_data_dir: "{{ mempool_dir }}/data"
+mempool_mysql_dir: "{{ mempool_dir }}/mysql"
+
+# Network - Bitcoin Core/Knots connection (via Tailnet Magic DNS)
+bitcoin_host: "knots-box"
+bitcoin_rpc_port: 8332
+# Note: bitcoin_rpc_user and bitcoin_rpc_password are loaded from infra_secrets.yml
+
+# Network - Fulcrum Electrum server (via Tailnet Magic DNS)
+fulcrum_host: "fulcrum-box"
+fulcrum_port: 50001
+fulcrum_tls: "false"
+
+# Mempool network mode
+mempool_network: "mainnet"
+
+# Container ports (internal)
+# Sourced from services_config.yml: the Caddy play on the edge host needs this
+# too, and a role default is not visible outside this role.
+mempool_frontend_port: "{{ service_settings.mempool.frontend_port }}"
+mempool_backend_port: 8999
+
+# MariaDB settings
+mariadb_database: "mempool"
+mariadb_user: "mempool"
+# Note: mariadb_mempool_password is loaded from infra_secrets.yml
+
+
+
+# --- Health checks ----------------------------------------------------------
+# Three independent checks, because Mempool is three moving parts and knowing
+# WHICH one is down is the whole point. Each records its answer in its exit
+# code, which systemd keeps:
+#     systemctl is-failed mempool-backend-healthcheck.service
+#
+# push_url is where to report, and is the single plug-in point for whatever
+# monitoring exists. Empty means check, exit honestly, report nowhere.
+# The URLs are credentials, so callers pass them from the vault.
+mempool_healthchecks:
+  - name: mariadb
+    label: MariaDB
+    push_url: ""
+  - name: backend
+    label: Backend
+    push_url: ""
+  - name: frontend
+    label: Frontend
+    push_url: ""

ansible/roles/mempool/tasks/deploy.yml

@@ -0,0 +1,75 @@
+---
+- name: Create mempool directories
+  file:
+    path: "{{ item }}"
+    state: directory
+    owner: "{{ ansible_user }}"
+    group: "{{ ansible_user }}"
+    mode: '0755'
+  loop:
+    - "{{ mempool_dir }}"
+    - "{{ mempool_data_dir }}"
+
+# MariaDB owns its own data directory. The container runs as uid 999 and has to
+# create files in there; this playbook declared owner: {{ ansible_user }} (1000),
+# which had been drifting from reality ever since the containers were created and
+# would have broken MariaDB the first time it needed a new file. It went
+# unnoticed only because the playbook had not been run since.
+#
+# So: ensure the directory exists, and let the container own it. On a fresh
+# install the mariadb image's entrypoint sets ownership itself.
+- name: Ensure the MariaDB data directory exists
+  file:
+    path: "{{ mempool_mysql_dir }}"
+    state: directory
+
+- name: Create docker-compose.yml for Mempool
+  ansible.builtin.template:
+    src: docker-compose.yml.j2
+    dest: "{{ mempool_dir }}/docker-compose.yml"
+    owner: "{{ ansible_user }}"
+    group: "{{ ansible_user }}"
+    mode: '0644'
+- name: Pull Mempool images
+  command: docker compose pull
+  args:
+    chdir: "{{ mempool_dir }}"
+
+- name: Deploy Mempool containers with docker compose
+  command: docker compose up -d
+  args:
+    chdir: "{{ mempool_dir }}"
+
+- name: Wait for MariaDB to be healthy
+  command: docker inspect --format='{{ '{{' }}.State.Health.Status{{ '}}' }}' mempool-db
+  register: mariadb_health
+  until: mariadb_health.stdout == 'healthy'
+  retries: 30
+  delay: 10
+  changed_when: false
+
+- name: Wait for Mempool backend to start
+  uri:
+    url: "http://localhost:{{ mempool_backend_port }}/api/v1/backend-info"
+    method: GET
+    status_code: 200
+    timeout: 10
+  register: backend_check
+  until: backend_check.status == 200
+  retries: 30
+  delay: 10
+  ignore_errors: yes
+
+- name: Wait for Mempool frontend to be available
+  uri:
+    url: "http://localhost:{{ mempool_frontend_port }}"
+    method: GET
+    status_code: 200
+    timeout: 10
+  register: frontend_check
+  until: frontend_check.status == 200
+  retries: 20
+  delay: 5
+  ignore_errors: yes
+
+# ═════════════════════════════════════════════════════════════════════════

ansible/roles/mempool/tasks/docker.yml

@@ -0,0 +1,72 @@
+---
+- name: Remove old Docker-related packages
+  apt:
+    name:
+      - docker.io
+      - docker-doc
+      - docker-compose
+      - podman-docker
+      - containerd
+      - runc
+    state: absent
+    purge: yes
+    autoremove: yes
+
+- name: Update apt cache
+  apt:
+    update_cache: yes
+
+- name: Install prerequisites
+  apt:
+    name:
+      - ca-certificates
+      - curl
+    state: present
+
+- name: Create directory for Docker GPG key
+  file:
+    path: /etc/apt/keyrings
+    state: directory
+    mode: '0755'
+
+- name: Download Docker GPG key
+  get_url:
+    url: https://download.docker.com/linux/debian/gpg
+    dest: /etc/apt/keyrings/docker.asc
+    mode: '0644'
+
+- name: Get Debian architecture
+  command: dpkg --print-architecture
+  register: deb_arch
+  changed_when: false
+
+- name: Add Docker repository
+  apt_repository:
+    repo: "deb [arch={{ deb_arch.stdout }} signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian {{ ansible_distribution_release }} stable"
+    filename: docker
+    state: present
+    update_cache: yes
+
+- name: Install Docker packages
+  apt:
+    name:
+      - docker-ce
+      - docker-ce-cli
+      - containerd.io
+      - docker-buildx-plugin
+      - docker-compose-plugin
+    state: present
+    update_cache: yes
+
+- name: Ensure Docker is started and enabled
+  systemd:
+    name: docker
+    enabled: yes
+    state: started
+
+- name: Add user to docker group
+  user:
+    name: "{{ ansible_user }}"
+    groups: docker
+    append: yes
+

ansible/roles/mempool/tasks/healthcheck.yml

@@ -0,0 +1,58 @@
+---
+# Three checks, one per moving part. The Uptime Kuma specifics that used to
+# follow — an embedded Python script creating monitors over the API, a /tmp
+# credentials file, a push-URL file read back and parsed, and three systemd
+# Environment= rewrites — are gone. Where each reports is now hc.push_url.
+- name: Create Mempool health check scripts
+  ansible.builtin.template:
+    src: "healthcheck-{{ hc.name }}.sh.j2"
+    dest: "/usr/local/bin/mempool-{{ hc.name }}-healthcheck-push.sh"
+    owner: root
+    group: root
+    mode: '0755'
+    validate: "bash -n %s"
+  loop: "{{ mempool_healthchecks }}"
+  loop_control:
+    loop_var: hc
+    label: "{{ hc.name }}"
+
+- name: Create systemd services for health checks
+  ansible.builtin.template:
+    src: healthcheck.service.j2
+    dest: "/etc/systemd/system/mempool-{{ hc.name }}-healthcheck.service"
+    owner: root
+    group: root
+    mode: '0644'
+  loop: "{{ mempool_healthchecks }}"
+  loop_control:
+    loop_var: hc
+    label: "{{ hc.name }}"
+
+- name: Create systemd timers for health checks
+  ansible.builtin.template:
+    src: healthcheck.timer.j2
+    dest: "/etc/systemd/system/mempool-{{ hc.name }}-healthcheck.timer"
+    owner: root
+    group: root
+    mode: '0644'
+  loop: "{{ mempool_healthchecks }}"
+  loop_control:
+    loop_var: hc
+    label: "{{ hc.name }}"
+
+- name: Reload systemd daemon
+  systemd:
+    daemon_reload: yes
+
+# Ungated on purpose: enabling a timer is deployment, not monitoring. The
+# deprecation banner swept this up with the push plumbing, so Ansible stopped
+# managing three timers that are in fact running on the host.
+- name: Enable and start health check timers
+  systemd:
+    name: "mempool-{{ hc.name }}-healthcheck.timer"
+    enabled: yes
+    state: started
+  loop: "{{ mempool_healthchecks }}"
+  loop_control:
+    loop_var: hc
+    label: "{{ hc.name }}"

ansible/roles/mempool/tasks/main.yml

@@ -0,0 +1,6 @@
+---
+# import_tasks, not include_tasks: static imports stay visible to --list-tasks,
+# which is how this conversion was verified against the playbook it replaced.
+- ansible.builtin.import_tasks: docker.yml
+- ansible.builtin.import_tasks: deploy.yml
+- ansible.builtin.import_tasks: healthcheck.yml

ansible/roles/mempool/templates/docker-compose.yml.j2

@@ -0,0 +1,75 @@
+# All containers use host network for Tailscale MagicDNS resolution
+services:
+  mariadb:
+    image: mariadb:10.11
+    container_name: mempool-db
+    restart: unless-stopped
+    network_mode: host
+    environment:
+      MYSQL_DATABASE: "{{ mariadb_database }}"
+      MYSQL_USER: "{{ mariadb_user }}"
+      MYSQL_PASSWORD: "{{ mariadb_mempool_password }}"
+      MYSQL_ROOT_PASSWORD: "{{ mariadb_mempool_password }}"
+    volumes:
+      - {{ mempool_mysql_dir }}:/var/lib/mysql
+    healthcheck:
+      test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+      start_period: 30s
+
+  mempool-backend:
+    image: mempool/backend:{{ mempool_version }}
+    container_name: mempool-backend
+    restart: unless-stopped
+    network_mode: host
+    environment:
+      # Database (localhost since all containers share host network)
+      DATABASE_ENABLED: "true"
+      DATABASE_HOST: "127.0.0.1"
+      DATABASE_DATABASE: "{{ mariadb_database }}"
+      DATABASE_USERNAME: "{{ mariadb_user }}"
+      DATABASE_PASSWORD: "{{ mariadb_mempool_password }}"
+      # Bitcoin Core/Knots (via Tailnet MagicDNS)
+      CORE_RPC_HOST: "{{ bitcoin_host }}"
+      CORE_RPC_PORT: "{{ bitcoin_rpc_port }}"
+      CORE_RPC_USERNAME: "{{ bitcoin_rpc_user }}"
+      CORE_RPC_PASSWORD: "{{ bitcoin_rpc_password }}"
+      # Electrum (Fulcrum via Tailnet MagicDNS)
+      ELECTRUM_HOST: "{{ fulcrum_host }}"
+      ELECTRUM_PORT: "{{ fulcrum_port }}"
+      ELECTRUM_TLS_ENABLED: "{{ fulcrum_tls }}"
+      # Mempool settings
+      MEMPOOL_NETWORK: "{{ mempool_network }}"
+      MEMPOOL_BACKEND: "electrum"
+      MEMPOOL_CLEAR_PROTECTION_MINUTES: "20"
+      MEMPOOL_INDEXING_BLOCKS_AMOUNT: "52560"
+    volumes:
+      - {{ mempool_data_dir }}:/backend/cache
+    depends_on:
+      mariadb:
+        condition: service_healthy
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:8999/api/v1/backend-info"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 60s
+
+  mempool-frontend:
+    image: mempool/frontend:{{ mempool_version }}
+    container_name: mempool-frontend
+    restart: unless-stopped
+    network_mode: host
+    environment:
+      FRONTEND_HTTP_PORT: "{{ mempool_frontend_port }}"
+      BACKEND_MAINNET_HTTP_HOST: "127.0.0.1"
+    depends_on:
+      - mempool-backend
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:{{ mempool_frontend_port }}"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 30s

ansible/roles/mempool/templates/healthcheck-backend.sh.j2

@@ -0,0 +1,18 @@
+#!/bin/bash
+# Mempool backend health check — managed by Ansible (roles/mempool)
+# The exit code is the answer; systemd keeps it. Reporting is optional.
+PUSH_URL="${HEALTHCHECK_PUSH_URL:-}"
+BACKEND_PORT="{{ mempool_backend_port }}"
+
+check() {
+    curl -sf --max-time 5 "http://localhost:${BACKEND_PORT}/api/v1/backend-info" > /dev/null 2>&1
+}
+
+report() {
+    [ -n "$PUSH_URL" ] || return 0
+    curl -s --max-time 10 --retry 2 -o /dev/null \
+        "${PUSH_URL}?status=$1&msg=${2// /%20}&ping=" || true
+}
+
+if check; then report up "OK"; exit 0
+else echo "Mempool backend not responding"; report down "Mempool backend not responding"; exit 1; fi

ansible/roles/mempool/templates/healthcheck-frontend.sh.j2

@@ -0,0 +1,18 @@
+#!/bin/bash
+# Mempool frontend health check — managed by Ansible (roles/mempool)
+# The exit code is the answer; systemd keeps it. Reporting is optional.
+PUSH_URL="${HEALTHCHECK_PUSH_URL:-}"
+FRONTEND_PORT="{{ mempool_frontend_port }}"
+
+check() {
+    curl -sf --max-time 5 "http://localhost:${FRONTEND_PORT}" > /dev/null 2>&1
+}
+
+report() {
+    [ -n "$PUSH_URL" ] || return 0
+    curl -s --max-time 10 --retry 2 -o /dev/null \
+        "${PUSH_URL}?status=$1&msg=${2// /%20}&ping=" || true
+}
+
+if check; then report up "OK"; exit 0
+else echo "Mempool frontend not responding"; report down "Mempool frontend not responding"; exit 1; fi

ansible/roles/mempool/templates/healthcheck-mariadb.sh.j2

@@ -0,0 +1,21 @@
+#!/bin/bash
+# Mempool MariaDB health check — managed by Ansible (roles/mempool)
+# The exit code is the answer; systemd keeps it. Reporting is optional.
+PUSH_URL="${HEALTHCHECK_PUSH_URL:-}"
+
+check() {
+{% raw %}
+    [ "$(docker inspect --format='{{.State.Health.Status}}' mempool-db 2>/dev/null)" = "healthy" ]
+{% endraw %}
+}
+
+report() {
+    # No push URL is normal, not an error. The previous version logged
+    # "ERROR: UPTIME_KUMA_PUSH_URL not set" on every fire, once a minute.
+    [ -n "$PUSH_URL" ] || return 0
+    curl -s --max-time 10 --retry 2 -o /dev/null \
+        "${PUSH_URL}?status=$1&msg=${2// /%20}&ping=" || true
+}
+
+if check; then report up "OK"; exit 0
+else echo "MariaDB container unhealthy"; report down "MariaDB container unhealthy"; exit 1; fi

ansible/roles/mempool/templates/healthcheck.service.j2

@@ -0,0 +1,14 @@
+[Unit]
+Description=Mempool {{ hc.label }} Health Check
+After=network.target docker.service
+
+[Service]
+Type=oneshot
+User=root
+ExecStart=/usr/local/bin/mempool-{{ hc.name }}-healthcheck-push.sh
+Environment=HEALTHCHECK_PUSH_URL={{ hc.push_url }}
+StandardOutput=journal
+StandardError=journal
+
+[Install]
+WantedBy=multi-user.target

ansible/roles/mempool/templates/healthcheck.timer.j2

@@ -0,0 +1,10 @@
+[Unit]
+Description=Mempool {{ hc.name }} Health Check Timer
+
+[Timer]
+OnBootSec=2min
+OnUnitActiveSec=1min
+Persistent=true
+
+[Install]
+WantedBy=timers.target