counterweight/personal_infra
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

forgejo-runner: convert to a role, de-Uptime-Kuma the health check

Browse source 
409-line playbook becomes a 16-line playbook plus a 318-line role with phases
split across tasks/{prerequisites,install,configure,service,healthcheck}.yml and
four templates. forgejo_runner_vars.yml is deleted; its content is the role's
defaults.

Applies the Plan 6 Stage 0 decision: keep whatever determines whether the
service is healthy, drop the Uptime Kuma specifics, make the reporting point
pluggable. Gone from the role: the embedded Python that created monitors over
the Kuma API, the /tmp credentials file, token extraction, the systemd
Environment= rewrite, and 8 `when: uptime_kuma_enabled` guards. What remains is
the check itself, its log, the systemd unit and timer, and an honest exit code -
`systemctl is-failed forgejo-runner-healthcheck.service` now answers the
question with no monitoring system involved at all.

Reporting is one variable, healthcheck_push_url, empty by default. Any endpoint
that accepts an HTTP ping plugs in there. A pull-based monitor wants it left
empty and reads unit state instead.

PREMISE CORRECTION: Uptime Kuma is NOT dead. Plan 3 recorded "48 push timers
curling an endpoint that no longer answers" and Plan 6 said the check had
"nowhere to report to". Both wrong - 24+ push scripts across 11 hosts are
pushing successfully right now (HTTP 200). Only the Ansible code and the vault
credentials were decommissioned; the service never stopped. So the existing push
URLs were harvested into a vaulted healthcheck_push_urls dict and are preserved,
keeping this refactor behaviour-neutral. Retiring Kuma stays a deliberate act
rather than a side effect. PLAN_3 and PLAN_6 are corrected.

Verified:
  - task-list diff vs the old playbook shows ONLY the five Kuma tasks removed,
    everything else identical and in the same order
  - first run ok=22 changed=1 (the rewritten health script); both systemd units
    and forgejo-runner.service came back ok, so the templates reproduce the
    previous files byte-for-byte
  - second run ok=22 changed=0, fully idempotent
  - still reports "Ping sent successfully (HTTP 200)" from a script containing
    zero Uptime Kuma references
  - the 4 skipped tasks are genuine already-configured guards, checked not assumed

Two things for the next service:

- import_tasks, not include_tasks. Dynamic includes are opaque to --list-tasks,
  which is the primary verification tool here; the first attempt produced a
  useless diff.
- `Assert runner is running` was guarded by uptime_kuma_enabled and so had not
  run since the decommissioning. It is not monitoring, it is the deployment
  checking its own work - the deprecation banner swept it up with the Kuma
  plumbing, and a runner that failed to start was deploying "successfully" in
  silence. Ungated now. The banner was applied to contiguous blocks, so read
  every uptime_kuma_enabled guard and ask whether it is monitoring or deployment.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
counterweight 2026-09-12 18:15:29 +02:00
parent 2ebb2f9a64
commit 73340d5fbe
Signed by: counterweight
GPG key ID: 883EDBAA726BD96C
16 changed files with 710 additions and 528 deletions
Download patch file Download diff file Expand all files Collapse all files

409
ansible/services/forgejo-runner/deploy_forgejo_runner_playbook.yml
View file

@ -1,3 +1,4 @@
---
- name: Install Forgejo Runner on Debian 13
hosts: ci_runner
become: yes
@ -5,405 +6,11 @@
- ../../infra_vars.yml
- ../../services_config.yml
- ../../infra_secrets.yml
- ./forgejo_runner_vars.yml
vars:
uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}"
ntfy_topic: "{{ service_settings.ntfy.topic }}"
healthcheck_interval_seconds: 60
healthcheck_timeout_seconds: 90
healthcheck_retries: 1
healthcheck_script_dir: /opt/forgejo-runner-healthcheck
healthcheck_script_path: "{{ healthcheck_script_dir }}/forgejo_runner_healthcheck.sh"
healthcheck_log_file: "{{ healthcheck_script_dir }}/forgejo_runner_healthcheck.log"
healthcheck_service_name: forgejo-runner-healthcheck
tasks:
# ── 1. Assert Docker is available ──────────────────────────────────
- name: Check if Docker is installed
command: docker --version
register: docker_check
changed_when: false
failed_when: docker_check.rc != 0
- name: Fail if Docker is not available
assert:
that:
- docker_check.rc == 0
fail_msg: >
Docker is not installed or not in PATH.
Please install Docker before running this playbook.
# ── 2. Download forgejo-runner binary ──────────────────────────────
- name: Download forgejo-runner binary
get_url:
url: "{{ forgejo_runner_url }}"
dest: "{{ forgejo_runner_bin_path }}"
mode: '0755'
# ── 3. Create runner system user ───────────────────────────────────
- name: Create runner system user
user:
name: "{{ forgejo_runner_user }}"
system: yes
shell: /usr/sbin/nologin
home: "{{ forgejo_runner_dir }}"
create_home: no
groups: docker
append: yes
comment: 'Forgejo Runner'
# ── 4. Create working directory ────────────────────────────────────
- name: Create forgejo-runner working directory
file:
path: "{{ forgejo_runner_dir }}"
state: directory
owner: "{{ forgejo_runner_user }}"
group: "{{ forgejo_runner_user }}"
mode: '0750'
# ── 5. Generate default config ─────────────────────────────────────
- name: Check if config already exists
stat:
path: "{{ forgejo_runner_config_path }}"
register: config_stat
- name: Generate default config
shell: "{{ forgejo_runner_bin_path }} generate-config > {{ forgejo_runner_config_path }}"
args:
chdir: "{{ forgejo_runner_dir }}"
when: not config_stat.stat.exists
- name: Set config file ownership
file:
path: "{{ forgejo_runner_config_path }}"
owner: "{{ forgejo_runner_user }}"
group: "{{ forgejo_runner_user }}"
when: not config_stat.stat.exists
# ── 6. Register runner ─────────────────────────────────────────────
- name: Check if runner is already registered
stat:
path: "{{ forgejo_runner_dir }}/.runner"
register: runner_stat
- name: Register runner with Forgejo instance
command: >
{{ forgejo_runner_bin_path }} register --no-interactive
--instance {{ forgejo_instance_url }}
--token {{ forgejo_runner_registration_token }}
--name forgejo-runner-box
--labels "{{ forgejo_runner_labels }}"
args:
chdir: "{{ forgejo_runner_dir }}"
when: not runner_stat.stat.exists
- name: Set runner registration file ownership
file:
path: "{{ forgejo_runner_dir }}/.runner"
owner: "{{ forgejo_runner_user }}"
group: "{{ forgejo_runner_user }}"
when: not runner_stat.stat.exists
# ── 7. Create systemd service ──────────────────────────────────────
- name: Create forgejo-runner systemd service
copy:
dest: /etc/systemd/system/forgejo-runner.service
content: |
[Unit]
Description=Forgejo Runner
Documentation=https://forgejo.org/docs/latest/admin/actions/
After=docker.service
Requires=docker.service
[Service]
Type=simple
User={{ forgejo_runner_user }}
Group={{ forgejo_runner_user }}
WorkingDirectory={{ forgejo_runner_dir }}
ExecStart={{ forgejo_runner_bin_path }} daemon --config {{ forgejo_runner_config_path }}
Restart=on-failure
RestartSec=10
[Install]
WantedBy=multi-user.target
owner: root
group: root
mode: '0644'
# ── 8. Reload systemd, enable and start ────────────────────────────
- name: Reload systemd
systemd:
daemon_reload: yes
- name: Enable and start forgejo-runner service
systemd:
name: forgejo-runner
enabled: yes
state: started
# ── 9. Verify runner is active ─────────────────────────────────────
- name: Verify forgejo-runner is active
command: systemctl is-active forgejo-runner
register: runner_active
changed_when: false
# ═════════════════════════════════════════════════════════════════════════
# DEPRECATED — Uptime Kuma was decommissioned on 2026-09-11.
#
# Every task below is inert: uptime_kuma_enabled is false in
# group_vars/all/main.yml, so they all skip and the deployment above still
# runs normally. Kept because the health-check logic is the durable part —
# when a replacement exists, rewire the push transport and flip the flag.
#
# What was being monitored: archive/uptime_kuma/MONITORS.md
# ═════════════════════════════════════════════════════════════════════════
- name: Assert runner is running
when: uptime_kuma_enabled | default(false)
assert:
that:
- runner_active.stdout == "active"
fail_msg: "forgejo-runner service is not active: {{ runner_active.stdout }}"
# ── 10. Set up Uptime Kuma push monitor ────────────────────────────
- name: Create Uptime Kuma push monitor setup script
when: uptime_kuma_enabled | default(false)
copy:
dest: /tmp/setup_forgejo_runner_monitor.py
content: |
#!/usr/bin/env python3
import sys
import json
from uptime_kuma_api import UptimeKumaApi
def main():
api_url = sys.argv[1]
username = sys.argv[2]
password = sys.argv[3]
group_name = sys.argv[4]
monitor_name = sys.argv[5]
monitor_description = sys.argv[6]
interval = int(sys.argv[7])
retries = int(sys.argv[8])
ntfy_topic = sys.argv[9] if len(sys.argv) > 9 else "alerts"
api = UptimeKumaApi(api_url, timeout=60, wait_events=2.0)
api.login(username, password)
# Get all monitors
monitors = api.get_monitors()
# Get all notifications and find ntfy notification
notifications = api.get_notifications()
ntfy_notification = next((n for n in notifications if n.get('name') == f'ntfy ({ntfy_topic})'), None)
notification_id_list = {}
if ntfy_notification:
notification_id_list[ntfy_notification['id']] = True
# Find or create group
group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None)
if not group:
group_result = api.add_monitor(type='group', name=group_name)
# Refresh to get the full group object with id
monitors = api.get_monitors()
group = next((m for m in monitors if m.get('name') == group_name and m.get('type') == 'group'), None)
# Find or create/update push monitor
existing_monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
monitor_data = {
'type': 'push',
'name': monitor_name,
'parent': group['id'],
'interval': interval,
'upsideDown': False,
'maxretries': retries,
'description': monitor_description,
'notificationIDList': notification_id_list
}
if existing_monitor:
monitor = api.edit_monitor(existing_monitor['id'], **monitor_data)
# Refresh to get the full monitor object with pushToken
monitors = api.get_monitors()
monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
else:
monitor_result = api.add_monitor(**monitor_data)
# Refresh to get the full monitor object with pushToken
monitors = api.get_monitors()
monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
result = {
'monitor_id': monitor['id'],
'push_token': monitor['pushToken'],
'group_name': group_name,
'group_id': group['id'],
'monitor_name': monitor_name
}
print(json.dumps(result))
api.disconnect()
if __name__ == '__main__':
main()
mode: '0755'
delegate_to: localhost
become: no
- name: Run Uptime Kuma push monitor setup
when: uptime_kuma_enabled | default(false)
command: >
{{ ansible_playbook_python }}
/tmp/setup_forgejo_runner_monitor.py
"{{ uptime_kuma_api_url }}"
"{{ uptime_kuma_username }}"
"{{ uptime_kuma_password }}"
"services"
"forgejo-runner-healthcheck"
"Forgejo Runner healthcheck - ping every {{ healthcheck_interval_seconds }}s"
"{{ healthcheck_timeout_seconds }}"
"{{ healthcheck_retries }}"
"{{ ntfy_topic }}"
register: monitor_setup_result
delegate_to: localhost
become: no
changed_when: false
- name: Parse monitor setup result
when: uptime_kuma_enabled | default(false)
set_fact:
monitor_info_parsed: "{{ monitor_setup_result.stdout | from_json }}"
- name: Set push URL
when: uptime_kuma_enabled | default(false)
set_fact:
uptime_kuma_push_url: "{{ uptime_kuma_api_url }}/api/push/{{ monitor_info_parsed.push_token }}"
- name: Create healthcheck script directory
file:
path: "{{ healthcheck_script_dir }}"
state: directory
owner: root
group: root
mode: '0755'
- name: Create forgejo-runner healthcheck script
when: uptime_kuma_enabled | default(false)
copy:
dest: "{{ healthcheck_script_path }}"
content: |
#!/bin/bash
# Forgejo Runner Healthcheck Script
# Checks if forgejo-runner is active and pings Uptime Kuma on success
LOG_FILE="{{ healthcheck_log_file }}"
UPTIME_KUMA_URL="{{ uptime_kuma_push_url }}"
log_message() {
echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
}
main() {
if systemctl is-active --quiet forgejo-runner; then
log_message "forgejo-runner is active, sending ping"
response=$(curl -s -w "\n%{http_code}" "$UPTIME_KUMA_URL?status=up&msg=forgejo-runner%20is%20active" 2>&1)
http_code=$(echo "$response" | tail -n1)
if [ "$http_code" = "200" ] || [ "$http_code" = "201" ]; then
log_message "Ping sent successfully (HTTP $http_code)"
else
log_message "ERROR: Failed to send ping (HTTP $http_code)"
exit 1
fi
else
log_message "ERROR: forgejo-runner is not active"
exit 1
fi
}
main
owner: root
group: root
mode: '0755'
- name: Create healthcheck systemd service
copy:
dest: "/etc/systemd/system/{{ healthcheck_service_name }}.service"
content: |
[Unit]
Description=Forgejo Runner Healthcheck
After=network.target
[Service]
Type=oneshot
ExecStart={{ healthcheck_script_path }}
User=root
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
owner: root
group: root
mode: '0644'
- name: Create healthcheck systemd timer
copy:
dest: "/etc/systemd/system/{{ healthcheck_service_name }}.timer"
content: |
[Unit]
Description=Run Forgejo Runner Healthcheck every minute
Requires={{ healthcheck_service_name }}.service
[Timer]
OnBootSec=30sec
OnUnitActiveSec={{ healthcheck_interval_seconds }}sec
Persistent=true
[Install]
WantedBy=timers.target
owner: root
group: root
mode: '0644'
- name: Reload systemd for healthcheck units
systemd:
daemon_reload: yes
- name: Enable and start healthcheck timer
systemd:
name: "{{ healthcheck_service_name }}.timer"
enabled: yes
state: started
- name: Test healthcheck script
command: "{{ healthcheck_script_path }}"
register: healthcheck_test
changed_when: false
- name: Verify healthcheck script works
assert:
that:
- healthcheck_test.rc == 0
fail_msg: "Healthcheck script failed to execute properly"
- name: Display deployment summary
debug:
msg: |
Forgejo Runner deployed successfully!
Runner Name: forgejo-runner-box
Instance: {{ forgejo_instance_url }}
Working Directory: {{ forgejo_runner_dir }}
Service: forgejo-runner.service ({{ runner_active.stdout }})
Healthcheck Monitor: forgejo-runner-healthcheck
Healthcheck Interval: Every {{ healthcheck_interval_seconds }}s
Timeout: {{ healthcheck_timeout_seconds }}s
- name: Clean up temporary monitor setup script
when: uptime_kuma_enabled | default(false)
file:
path: /tmp/setup_forgejo_runner_monitor.py
state: absent
delegate_to: localhost
become: no
# Preserves the push URL this host has been reporting to all along, so the
# move to a role changes no behaviour. The role itself knows nothing about
# Uptime Kuma — this is just "a URL that accepts a ping", and whatever
# replaces it sets the same variable.
healthcheck_push_url: "{{ healthcheck_push_urls.forgejo_runner | default('') }}"
roles:
- forgejo_runner

9
ansible/services/forgejo-runner/forgejo_runner_vars.yml
View file

@ -1,9 +0,0 @@
forgejo_runner_version: "6.3.1"
forgejo_runner_arch: "linux-amd64"
forgejo_runner_url: "https://code.forgejo.org/forgejo/runner/releases/download/v{{ forgejo_runner_version }}/forgejo-runner-{{ forgejo_runner_version }}-{{ forgejo_runner_arch }}"
forgejo_runner_bin_path: "/usr/local/bin/forgejo-runner"
forgejo_runner_user: "runner"
forgejo_runner_dir: "/opt/forgejo-runner"
forgejo_runner_config_path: "{{ forgejo_runner_dir }}/config.yml"
forgejo_runner_labels: "docker:docker://node:20-bookworm,ubuntu-latest:docker://node:20-bookworm,ubuntu-22.04:docker://node:20-bookworm,ubuntu-24.04:docker://node:20-bookworm"
forgejo_instance_url: "https://forgejo.contrapeso.xyz"