counterweight/personal_infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

remove uptime kuma

Browse source
This commit is contained in:
counterweight 2026-09-20 17:37:47 +02:00
parent e791ffb183
commit defb2f6333
Signed by: counterweight
GPG key ID: 883EDBAA726BD96C
7 changed files with 0 additions and 1765 deletions
Download patch file Download diff file Expand all files Collapse all files

152
archive/uptime_kuma/MONITORS.md
View file

@ -1,152 +0,0 @@
# Uptime Kuma — monitor inventory (archived)
Captured from the live instance at `https://uptime.contrapeso.xyz` on 2026-09-11,
immediately before decommissioning. This is the **authoritative** record: most of
these monitors existed only in the Uptime Kuma UI and were never described by any
playbook in this repo.
**75 monitors total** — 16 group, 8 http, 3 port, 48 push. All were active.
Push tokens are deliberately **not** recorded here: they are live credentials, and
anything holding one could report a false 'up'. They die with the server.
---
## arbret - production *(7 monitors)*
| Monitor | Type | Target | Interval | Notes |
|---|---|---|---|---|
| arbret.com - arbret-analytics | push | — | 120s | Healthy when timer is scheduled and last run succeeded |
| arbret.com - arbret-backup | push | — | 120s | Healthy when timer is scheduled and last run succeeded |
| arbret.com - arbret-server | push | — | 120s | Healthy when arbret-server.service is active |
| arbret.com - arbret-worker | push | — | 120s | Healthy when arbret-worker.service is active |
| arbret.com - health | push | — | 120s | Healthy when GET /api/health returns status ok |
| arbret.com - https | push | — | 120s | Healthy when HTTPS front-door returns 200 |
| arbret.com - postgresql | push | — | 120s | Healthy when postgresql.service is active |
## arbret - staging *(7 monitors)*
| Monitor | Type | Target | Interval | Notes |
|---|---|---|---|---|
| arbretstaging.contrapeso.xyz - arbret-analytics | push | — | 120s | Healthy when timer is scheduled and last run succeeded |
| arbretstaging.contrapeso.xyz - arbret-backup | push | — | 120s | Healthy when timer is scheduled and last run succeeded |
| arbretstaging.contrapeso.xyz - arbret-server | push | — | 120s | Healthy when arbret-server.service is active |
| arbretstaging.contrapeso.xyz - arbret-worker | push | — | 120s | Healthy when arbret-worker.service is active |
| arbretstaging.contrapeso.xyz - health | push | — | 120s | Healthy when GET /api/health returns status ok |
| arbretstaging.contrapeso.xyz - https | push | — | 120s | Healthy when HTTPS front-door returns 200 |
| arbretstaging.contrapeso.xyz - postgresql | push | — | 120s | Healthy when postgresql.service is active |
## arbret-staging-box - infra *(2 monitors)*
| Monitor | Type | Target | Interval | Notes |
|---|---|---|---|---|
| disk-usage-arbret-staging-box-root | push | — | 960s | upside-down, Disk Usage: arbret-staging-box (/) - Alerts when usage excee |
| system-healthcheck-arbret-staging-box | push | — | 90s | System Healthcheck: arbret-staging-box - Regular healthcheck |
## forgejo-runner-box - infra *(2 monitors)*
| Monitor | Type | Target | Interval | Notes |
|---|---|---|---|---|
| disk-usage-forgejo-runner-box-root | push | — | 960s | upside-down, Disk Usage: forgejo-runner-box (/) - Alerts when usage excee |
| system-healthcheck-forgejo-runner-box | push | — | 90s | System Healthcheck: forgejo-runner-box - Regular healthcheck |
## fulcrum-box - infra *(2 monitors)*
| Monitor | Type | Target | Interval | Notes |
|---|---|---|---|---|
| disk-usage-fulcrum-box-root | push | — | 960s | upside-down, Disk Usage: fulcrum-box (/) - Alerts when usage exceeds 80% |
| system-healthcheck-fulcrum-box | push | — | 90s | System Healthcheck: fulcrum-box - Regular healthcheck ping e |
## knots-box - infra *(2 monitors)*
| Monitor | Type | Target | Interval | Notes |
|---|---|---|---|---|
| disk-usage-knots-box-root | push | — | 960s | upside-down, Disk Usage: knots-box (/) - Alerts when usage exceeds 80% |
| system-healthcheck-knots-box | push | — | 90s | System Healthcheck: knots-box - Regular healthcheck ping eve |
## memos-box - infra *(2 monitors)*
| Monitor | Type | Target | Interval | Notes |
|---|---|---|---|---|
| disk-usage-memos-box-root | push | — | 960s | upside-down, Disk Usage: memos-box (/) - Alerts when usage exceeds 80% |
| system-healthcheck-memos-box | push | — | 90s | System Healthcheck: memos-box - Regular healthcheck ping eve |
## mempool-box - infra *(2 monitors)*
| Monitor | Type | Target | Interval | Notes |
|---|---|---|---|---|
| disk-usage-mempool-box-root | push | — | 960s | upside-down, Disk Usage: mempool-box (/) - Alerts when usage exceeds 80% |
| system-healthcheck-mempool-box | push | — | 90s | System Healthcheck: mempool-box - Regular healthcheck ping e |
## nodito - infra *(4 monitors)*
| Monitor | Type | Target | Interval | Notes |
|---|---|---|---|---|
| UPS ONLINE | push | https:// | 90s | — |
| cpu-temp-nodito | push | — | 120s | upside-down, CPU Temperature: nodito - Alerts when temperature exceeds 80 |
| system-healthcheck-nodito | push | — | 300s | System Healthcheck: nodito - Regular healthcheck ping every |
| zfs-health-nodito | push | — | 90000s | ZFS Pool Health: nodito - Daily health check for pool proxmo |
## nonkeiwaisi-box - infra *(0 monitors)*
_(empty)_
## prd-arbret - infra *(2 monitors)*
| Monitor | Type | Target | Interval | Notes |
|---|---|---|---|---|
| disk-usage-prd-arbret-root | push | — | 960s | upside-down, Disk Usage: prd-arbret (/) - Alerts when usage exceeds 80% |
| system-healthcheck-prd-arbret | push | — | 90s | System Healthcheck: prd-arbret - Regular healthcheck ping ev |
## prd-spacey - infra *(2 monitors)*
| Monitor | Type | Target | Interval | Notes |
|---|---|---|---|---|
| disk-usage-prd-spacey-root | push | — | 960s | upside-down, Disk Usage: prd-spacey (/) - Alerts when usage exceeds 80% |
| system-healthcheck-prd-spacey | push | — | 90s | System Healthcheck: prd-spacey - Regular healthcheck ping ev |
## prd-vipy - infra *(2 monitors)*
| Monitor | Type | Target | Interval | Notes |
|---|---|---|---|---|
| disk-usage-prd-vipy-root | push | — | 960s | upside-down, Disk Usage: prd-vipy (/) - Alerts when usage exceeds 80% |
| system-healthcheck-prd-vipy | push | — | 90s | System Healthcheck: prd-vipy - Regular healthcheck ping ever |
## prd-watchtower - infra *(2 monitors)*
| Monitor | Type | Target | Interval | Notes |
|---|---|---|---|---|
| disk-usage-prd-watchtower-root | push | — | 960s | upside-down, Disk Usage: prd-watchtower (/) - Alerts when usage exceeds 8 |
| system-healthcheck-prd-watchtower | push | — | 90s | System Healthcheck: prd-watchtower - Regular healthcheck pin |
## services *(19 monitors)*
| Monitor | Type | Target | Interval | Notes |
|---|---|---|---|---|
| Forgejo | http | https://forgejo.contrapeso.xyz/api/healthz | 90s | — |
| Headscale | http | https://headscale.contrapeso.xyz/health | 60s | — |
| LNBits | http | https://wallet.contrapeso.xyz/api/v1/health | 60s | — |
| Memos | http | https://memos.contrapeso.xyz/healthz | 60s | — |
| Mempool Public Access | http | https://mempool.contrapeso.xyz | 60s | — |
| Personal Blog | http | https://pablohere.contrapeso.xyz | 60s | — |
| Vaultwarden | http | https://vault.contrapeso.xyz/alive | 60s | — |
| ntfy-emergency-app | http | https://avisame.contrapeso.xyz | 60s | — |
| Bitcoin Knots P2P Public | port | 167.172.107.33:8333 | 60s | — |
| DATUM Stratum (public) | port | 167.172.107.33:23334 | 60s | — |
| Fulcrum SSL Public | port | 167.172.107.33:50002 | 60s | — |
| Bitcoin Knots | push | — | 90s | — |
| DATUM Gateway | push | — | 90s | — |
| Fulcrum | push | — | 90s | — |
| Mempool Backend | push | — | 180s | — |
| Mempool Frontend | push | — | 90s | — |
| Mempool MariaDB | push | — | 90s | — |
| Phoenixd | push | — | 90s | — |
| forgejo-runner-healthcheck | push | — | 90s | Forgejo Runner healthcheck - ping every 60s |
## small-backups-box - infra *(2 monitors)*
| Monitor | Type | Target | Interval | Notes |
|---|---|---|---|---|
| disk-usage-small-backups-box-root | push | — | 960s | upside-down, Disk Usage: small-backups-box (/) - Alerts when usage exceed |
| system-healthcheck-small-backups-box | push | — | 90s | System Healthcheck: small-backups-box - Regular healthcheck |

51
archive/uptime_kuma/README.md
View file

@ -1,51 +0,0 @@
# Uptime Kuma — archived
Uptime Kuma was the monitoring stack for this infrastructure until **2026-09-11**, when
it was decommissioned. Everything that referenced it has been removed from the live
playbooks; this folder is the record of what it was, kept so the setup can be understood
later without digging through git history.
## Contents
| File | What it is |
|---|---|
| `MONITORS.md` | Every monitor that existed, grouped as it was in the UI. The authoritative record. |
| `monitors.json` | The same data, machine-readable, as returned by the API. |
| `deploy_uptime_kuma_playbook.yml` | How the server itself was deployed (Docker Compose on `monitoring`, behind Caddy). |
| `uptime_kuma_vars.yml` | Variables the deploy playbook needed — it is unreadable without these. |
| `setup_backup_uptime_kuma_to_lapy.yml` | How its data was backed up, and to where. Useful when disposing of the old volumes. |
## Why the inventory was captured from the live server, not the repo
The playbooks only ever created **17** monitors. The live instance had **75**. The
difference was created by hand in the UI and existed nowhere else — so a repo-derived
list would have silently lost two thirds of the picture. `MONITORS.md` is a snapshot of
the real thing, taken immediately before removal.
Push tokens are deliberately excluded. They are live credentials — anything holding one
can report a false "up" — and they become meaningless once the server is gone.
## What removal did NOT do
Removing the playbook code does not touch the machines. **48 push monitors** were driven
by scripts and systemd timers installed *on the hosts*, which keep firing on their
schedule and curling an endpoint that no longer answers. They are harmless but they are
still there, writing logs and failing quietly.
Left behind, per host:
- `/opt/disk-monitoring` + `disk-usage-monitor.{service,timer}` — on all 12 `managed` hosts
- `/opt/system-healthcheck` + `system-healthcheck.{service,timer}` — on all 12 `managed` hosts
- `/opt/nodito-monitoring` + `nodito-cpu-temp-monitor.{service,timer}` — `nodito`
- `/opt/zfs-monitoring` — `nodito`
- `/opt/ups-monitoring` — `nodito`
- `bitcoin-knots-healthcheck.{service,timer}` — `bitcoin`
- `datum-gateway-healthcheck.{service,timer}` — `bitcoin`
- `fulcrum-healthcheck.{service,timer}` — `electrum`
- `mempool-{backend,frontend,mariadb}-healthcheck.service` — `mempool`
- phoenixd and forgejo-runner healthcheck units — `edge`, `ci_runner`
Note `nut-monitor.service` on `nodito` is **NUT's own daemon**, not a monitoring
leftover — do not remove it with the rest.
Cleaning these up is a separate decommissioning pass and was not part of the removal.

74
archive/uptime_kuma/deploy_uptime_kuma_playbook.yml
View file

@ -1,74 +0,0 @@
- name: Deploy Uptime Kuma with Docker Compose and configure Caddy reverse proxy
hosts: monitoring
become: yes
vars_files:
- ../../infra_vars.yml
- ../../services_config.yml
- ./uptime_kuma_vars.yml
vars:
uptime_kuma_subdomain: "{{ subdomains.uptime_kuma }}"
caddy_sites_dir: "{{ caddy_sites_dir }}"
uptime_kuma_domain: "{{ uptime_kuma_subdomain }}.{{ root_domain }}"
tasks:
- name: Create uptime kuma directory
file:
path: "{{ uptime_kuma_dir }}"
state: directory
owner: "{{ ansible_user }}"
group: "{{ ansible_user }}"
mode: '0755'
- name: Create docker-compose.yml for uptime kuma
copy:
dest: "{{ uptime_kuma_dir }}/docker-compose.yml"
content: |
version: "3"
services:
uptime-kuma:
image: louislam/uptime-kuma:latest
container_name: uptime-kuma
restart: unless-stopped
ports:
- "{{ uptime_kuma_port }}:3001"
volumes:
- ./data:/app/data
dns:
- 1.1.1.1
- 9.9.9.9
- 8.8.8.8
- name: Deploy uptime kuma container with docker compose
command: docker compose up -d
args:
chdir: "{{ uptime_kuma_dir }}"
- name: Ensure Caddy sites-enabled directory exists
file:
path: /etc/caddy/sites-enabled
state: directory
owner: root
group: root
mode: '0755'
- name: Ensure Caddyfile includes import directive for sites-enabled
lineinfile:
path: /etc/caddy/Caddyfile
line: 'import sites-enabled/*'
insertafter: EOF
state: present
backup: yes
- name: Create Caddy reverse proxy configuration for uptime kuma
copy:
dest: "{{ caddy_sites_dir }}/uptime-kuma.conf"
content: |
{{ uptime_kuma_domain }} {
reverse_proxy localhost:{{ uptime_kuma_port }}
}
owner: root
group: root
mode: '0644'
- name: Reload Caddy to apply new config
command: systemctl reload caddy

1202
archive/uptime_kuma/monitors.json
View file

File diff suppressed because it is too large Load diff

107
archive/uptime_kuma/setup_backup_uptime_kuma_to_lapy.yml
View file

@ -1,107 +0,0 @@
- name: Configure local backup for Uptime Kuma from remote
hosts: control
gather_facts: no
vars_files:
- ../../infra_vars.yml
- ./uptime_kuma_vars.yml
vars:
remote_data_path: "{{ uptime_kuma_data_dir }}"
local_backup_dir: "{{ lookup('env', 'HOME') }}/uptime-kuma-backups"
backup_script_path: "{{ lookup('env', 'HOME') }}/.local/bin/uptime_kuma_backup.sh"
tasks:
- name: Debug remote backup vars
debug:
msg:
- "remote_host={{ remote_host }}"
- "remote_user={{ remote_user }}"
- "remote_data_path='{{ remote_data_path }}'"
- "local_backup_dir={{ local_backup_dir }}"
- name: Ensure local backup directory exists
file:
path: "{{ local_backup_dir }}"
state: directory
mode: '0755'
- name: Ensure ~/.local/bin exists
file:
path: "{{ lookup('env', 'HOME') }}/.local/bin"
state: directory
mode: '0755'
- name: Create backup script
copy:
dest: "{{ backup_script_path }}"
mode: '0750'
content: |
#!/bin/bash
set -euo pipefail
TIMESTAMP=$(date +'%Y-%m-%d')
BACKUP_DIR="{{ local_backup_dir }}/$TIMESTAMP"
mkdir -p "$BACKUP_DIR"
{% if remote_key_file %}
SSH_CMD="ssh -i {{ remote_key_file }} -p {{ remote_port }}"
{% else %}
SSH_CMD="ssh -p {{ remote_port }}"
{% endif %}
rsync -az -e "$SSH_CMD" --delete {{ remote_user }}@{{ remote_host }}:{{ remote_data_path }}/ "$BACKUP_DIR/"
# Rotate old backups (keep 14 days)
# Calculate cutoff date (14 days ago) and delete backups older than that
CUTOFF_DATE=$(date -d '14 days ago' +'%Y-%m-%d')
for dir in "{{ local_backup_dir }}"/20*; do
if [ -d "$dir" ]; then
dir_date=$(basename "$dir")
if [ "$dir_date" != "$TIMESTAMP" ] && [ "$dir_date" \< "$CUTOFF_DATE" ]; then
rm -rf "$dir"
fi
fi
done
- name: Ensure cronjob for backup exists
cron:
name: "Uptime Kuma backup"
user: "{{ lookup('env', 'USER') }}"
job: "{{ backup_script_path }}"
minute: 0
hour: "9,12,15,18"
- name: Run the backup script to make the first backup
command: "{{ backup_script_path }}"
- name: Verify backup was created
block:
- name: Get today's date
command: date +'%Y-%m-%d'
register: today_date
changed_when: false
- name: Check backup directory exists and contains files
stat:
path: "{{ local_backup_dir }}/{{ today_date.stdout }}"
register: backup_dir_stat
- name: Verify backup directory exists
assert:
that:
- backup_dir_stat.stat.exists
- backup_dir_stat.stat.isdir
fail_msg: "Backup directory {{ local_backup_dir }}/{{ today_date.stdout }} was not created"
success_msg: "Backup directory {{ local_backup_dir }}/{{ today_date.stdout }} exists"
- name: Check if backup directory contains files
find:
paths: "{{ local_backup_dir }}/{{ today_date.stdout }}"
recurse: yes
register: backup_files
- name: Verify backup directory is not empty
assert:
that:
- backup_files.files | length > 0
fail_msg: "Backup directory {{ local_backup_dir }}/{{ today_date.stdout }} exists but is empty"
success_msg: "Backup directory contains {{ backup_files.files | length }} file(s)"

164
archive/uptime_kuma/setup_ntfy_uptime_kuma_notification.yml
View file

@ -1,164 +0,0 @@
# ═════════════════════════════════════════════════════════════════════════════
# DEPRECATED — Uptime Kuma was decommissioned on 2026-09-11.
#
# This play WILL FAIL if run as-is, and that is deliberate: uptime_kuma_username
# and uptime_kuma_password were removed from the vault, so the "Validate Uptime
# Kuma configuration" assert stops it before anything is installed or changed.
#
# It is kept because the CHECK LOGIC is the durable part — what gets measured,
# the thresholds, and the systemd timer plumbing. When something replaces Uptime
# Kuma, only the push transport needs rewriting; the rest still applies.
#
# What was being monitored: archive/uptime_kuma/MONITORS.md
# ═════════════════════════════════════════════════════════════════════════════
- name: Setup ntfy as Uptime Kuma Notification Channel
hosts: observability
become: no
vars_files:
- ./ntfy_vars.yml
vars:
ntfy_subdomain: "{{ subdomains.ntfy }}"
uptime_kuma_subdomain: "{{ subdomains.uptime_kuma }}"
ntfy_domain: "{{ ntfy_subdomain }}.{{ root_domain }}"
ntfy_server_url: "https://{{ ntfy_domain }}"
ntfy_priority: 4 # 1=min, 2=low, 3=default, 4=high, 5=max
uptime_kuma_api_url: "https://{{ uptime_kuma_subdomain }}.{{ root_domain }}"
tasks:
- name: Validate Uptime Kuma configuration
assert:
that:
- uptime_kuma_api_url is defined
- uptime_kuma_api_url != ""
- uptime_kuma_username is defined
- uptime_kuma_username != ""
- uptime_kuma_password is defined
- uptime_kuma_password != ""
fail_msg: "uptime_kuma_api_url, uptime_kuma_username and uptime_kuma_password must be set"
- name: Validate ntfy configuration
assert:
that:
- ntfy_domain is defined
- ntfy_domain != ""
- ntfy_topic is defined
- ntfy_topic != ""
- ntfy_username is defined
- ntfy_username != ""
- ntfy_password is defined
- ntfy_password != ""
fail_msg: "ntfy_domain, ntfy_topic, ntfy_username and ntfy_password must be set"
- name: Create Uptime Kuma ntfy notification setup script
copy:
dest: /tmp/setup_uptime_kuma_ntfy_notification.py
content: |
#!/usr/bin/env python3
import sys
import json
from uptime_kuma_api import UptimeKumaApi
def main():
api_url = sys.argv[1]
username = sys.argv[2]
password = sys.argv[3]
notification_name = sys.argv[4]
ntfy_server_url = sys.argv[5]
ntfy_topic = sys.argv[6]
ntfy_username = sys.argv[7]
ntfy_password = sys.argv[8]
ntfy_priority = int(sys.argv[9])
api = UptimeKumaApi(api_url, timeout=60, wait_events=2.0)
api.login(username, password)
# Get all notifications
notifications = api.get_notifications()
# Find existing ntfy notification by name
existing_notification = next((n for n in notifications if n.get('name') == notification_name), None)
notification_data = {
'name': notification_name,
'type': 'ntfy',
'isDefault': True, # Apply to all monitors by default
'applyExisting': True, # Apply to existing monitors
'ntfyserverurl': ntfy_server_url,
'ntfytopic': ntfy_topic,
'ntfyusername': ntfy_username,
'ntfypassword': ntfy_password,
'ntfyPriority': ntfy_priority
}
if existing_notification:
notification = api.edit_notification(existing_notification['id'], **notification_data)
action = "updated"
else:
notification = api.add_notification(**notification_data)
action = "created"
# Output result as JSON
result = {
'notification_id': notification['id'],
'notification_name': notification_name,
'ntfy_server': ntfy_server_url,
'ntfy_topic': ntfy_topic,
'action': action
}
print(json.dumps(result))
api.disconnect()
if __name__ == '__main__':
main()
mode: '0755'
delegate_to: localhost
become: no
- name: Run Uptime Kuma ntfy notification setup script
command: >
{{ ansible_playbook_python }}
/tmp/setup_uptime_kuma_ntfy_notification.py
"{{ uptime_kuma_api_url }}"
"{{ uptime_kuma_username }}"
"{{ uptime_kuma_password }}"
"ntfy ({{ ntfy_topic }})"
"{{ ntfy_server_url }}"
"{{ ntfy_topic }}"
"{{ ntfy_username }}"
"{{ ntfy_password }}"
"{{ ntfy_priority }}"
register: notification_setup_result
delegate_to: localhost
become: no
changed_when: false
- name: Parse notification setup result
set_fact:
notification_info_parsed: "{{ notification_setup_result.stdout | from_json }}"
- name: Display notification information
debug:
msg: |
✓ ntfy notification channel {{ notification_info_parsed.action }} successfully!
Notification Name: ntfy ({{ ntfy_topic }})
ntfy Server: {{ ntfy_server_url }}
ntfy Topic: {{ ntfy_topic }}
Priority: {{ ntfy_priority }} (4=high)
Default for all monitors: Yes
Applied to existing monitors: Yes
All Uptime Kuma monitors will now send alerts to your ntfy server
on the "{{ ntfy_topic }}" topic.
You can subscribe to alerts at: {{ ntfy_server_url }}/{{ ntfy_topic }}
- name: Clean up temporary Uptime Kuma setup script
file:
path: /tmp/setup_uptime_kuma_ntfy_notification.py
state: absent
delegate_to: localhost
become: no

15
archive/uptime_kuma/uptime_kuma_vars.yml
View file

@ -1,15 +0,0 @@
# General
uptime_kuma_dir: /opt/uptime-kuma
uptime_kuma_data_dir: "{{ uptime_kuma_dir }}/data"
uptime_kuma_port: 3001
# Remote access
remote_host_name: "{{ groups['monitoring'] | first }}"
remote_host: "{{ hostvars.get(remote_host_name, {}).get('ansible_host', remote_host_name) }}"
remote_user: "{{ hostvars.get(remote_host_name, {}).get('ansible_user', 'counterweight') }}"
remote_key_file: "{{ hostvars.get(remote_host_name, {}).get('ansible_ssh_private_key_file', '') }}"
remote_port: "{{ hostvars.get(remote_host_name, {}).get('ansible_port', 22) }}"
# Local backup
local_backup_dir: "{{ lookup('env', 'HOME') }}/uptime-kuma-backups"
backup_script_path: "{{ lookup('env', 'HOME') }}/.local/bin/uptime_kuma_backup.sh"