fulcrum: convert to a role, de-Uptime-Kuma the health check
685-line playbook becomes 33 lines plus a 426-line role (install/service/healthcheck phases, six templates, one handler). fulcrum_vars.yml is deleted; its content is the role's defaults. Verified: fulcrum untouched - active since 2026-07-29 (no restart), 192G datadir, height 966842, bitcoind and db_mem unchanged on disk. Second run changed=1 (the arming run, changed_when: false aside). vipy changed=0. THREE PRE-EXISTING LANDMINES the check-mode diff caught, any of which a faithful extraction would have detonated: - bitcoin_rpc_host was "192.168.1.140", commented "IP of knots_box_local". But .140 is fulcrum-box ITSELF; knots-box is .135. The DHCP leases had reshuffled - the fifth instance of this same disease in this estate. The live config had been hand-corrected to knots-box; running the playbook would have reverted it and pointed Fulcrum at itself. Now addressed by Tailscale name. - The `Restart fulcrum` handler was guarded by uptime_kuma_enabled, so the three tasks that notify it (SSL cert, fulcrum.conf, systemd unit) could not restart anything. A config change applied to disk, reported success, and silently never took effect. That is worse than the other banner casualties: it makes the deployment itself lie. Ungated. - db_mem was about to go 2048 -> 4448 (75% of 5931MB RAM), leaving ~1.4GB for the OS and Fulcrum's non-cache memory. The live value had been hand-tuned down. fulcrum_db_mem_mb_override pins it. Note set_fact outranks role defaults, so the calculation itself has to honour the override. MY OWN ERROR, third instance: retyping `copy:` as `template:` lost `owner:` on the banner and on fulcrum.conf. Rather than keep catching these by eye, every managed path's owner/group/mode is now compared against `git show HEAD:` mechanically - 12/12 match. The health check timer had not fired since 2026-02-17 while reporting `active` and `enabled`. It is OnBootSec + OnUnitActiveSec with no OnCalendar: OnBootSec elapses once, and OnUnitActiveSec needs the SERVICE to have run this boot to have anything to schedule from. Restarting the timer does not supply that; running the service does, so the role now runs the check once after enabling. Also dropped `Requires=fulcrum.service` from the timer - on a timer that means "stop watching when the watched thing stops". Diagnostic note: NextElapseUSecRealtime is always empty for a monotonic timer, so it reads as broken even when healthy. I misread it once and wrongly called the timer dead. Use NextElapseUSecMonotonic or systemctl list-timers. fulcrum_ssl_port and fulcrum_tailscale_hostname moved to services_config.yml - the socket-proxy play on the edge host needs them and a role default cannot reach a second play. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
356139290f
commit
e83191c029
15 changed files with 516 additions and 666 deletions
65
ansible/roles/fulcrum/README.md
Normal file
65
ansible/roles/fulcrum/README.md
Normal file
|
|
@ -0,0 +1,65 @@
|
||||||
|
# `fulcrum`
|
||||||
|
|
||||||
|
Deploys [Fulcrum](https://github.com/cculianu/Fulcrum), an Electrum server
|
||||||
|
indexing the Bitcoin Knots node, on `fulcrum-box`. The second play in the
|
||||||
|
calling playbook publishes its SSL port from the edge host via `socket_proxy`.
|
||||||
|
|
||||||
|
Converted from `deploy_fulcrum_playbook.yml` (685 lines) under Plan 6. The
|
||||||
|
playbook is now 33 lines.
|
||||||
|
|
||||||
|
## The index is the expensive thing
|
||||||
|
|
||||||
|
`{{ fulcrum_db_dir }}` is ~192 GB and takes days to rebuild. Nothing in this role
|
||||||
|
touches it beyond `state: directory` with the ownership it already has
|
||||||
|
(`fulcrum:fulcrum 0755`). Restarting Fulcrum re-opens the database; it does not
|
||||||
|
reindex.
|
||||||
|
|
||||||
|
## Three things this conversion fixed, all pre-existing
|
||||||
|
|
||||||
|
**`bitcoind` pointed at the wrong machine.** The vars file carried
|
||||||
|
`bitcoin_rpc_host: "192.168.1.140"` commented "IP of knots_box_local", but `.140`
|
||||||
|
is **fulcrum-box itself** — knots-box is `.135`. The DHCP leases had reshuffled.
|
||||||
|
The live config had already been hand-corrected to `knots-box`; running the
|
||||||
|
playbook would have reverted it and broken indexing. Now addressed by Tailscale
|
||||||
|
name, like everything else in this repo.
|
||||||
|
|
||||||
|
**The restart handler was inert.** It carried
|
||||||
|
`when: uptime_kuma_enabled | default(false)`, so the three tasks that notify it
|
||||||
|
(SSL certificate, `fulcrum.conf`, systemd unit) could not restart anything. A
|
||||||
|
configuration change would write to disk, report success, and silently never take
|
||||||
|
effect. Ungated.
|
||||||
|
|
||||||
|
**`db_mem` was about to quadruple.** The role computes a share of RAM; on this
|
||||||
|
5931 MB host 75% is 4448 MB, leaving ~1.4 GB for the OS and Fulcrum's non-cache
|
||||||
|
memory. The live value had been hand-tuned to 2048. `fulcrum_db_mem_mb_override`
|
||||||
|
pins it. Note `set_fact` outranks role defaults, so the *calculation* has to
|
||||||
|
honour the override — pinning it in `defaults/` alone is silently ignored.
|
||||||
|
|
||||||
|
## The health check timer, and how to read it
|
||||||
|
|
||||||
|
The timer is `OnBootSec` + `OnUnitActiveSec` with no `OnCalendar`. That
|
||||||
|
combination has a failure mode worth knowing: `OnBootSec` is monotonic and
|
||||||
|
elapses once; `OnUnitActiveSec` schedules relative to the **service** last being
|
||||||
|
active. If the service does not run in a given boot, there is no reference to
|
||||||
|
schedule from and the timer sits `active` and `enabled` doing nothing. That is
|
||||||
|
exactly what had happened here — last trigger **2026-02-17**, seven months of no
|
||||||
|
health check, with every surface-level indicator green.
|
||||||
|
|
||||||
|
Restarting the timer does not supply that reference; running the service does.
|
||||||
|
So the role runs the check once after enabling the timer, which is both the fix
|
||||||
|
and a smoke test.
|
||||||
|
|
||||||
|
**Diagnosing this is easy to get wrong**: `NextElapseUSecRealtime` is always
|
||||||
|
empty for a monotonic timer, so it looks broken even when it is fine. Read
|
||||||
|
`NextElapseUSecMonotonic`, or just use `systemctl list-timers`.
|
||||||
|
|
||||||
|
The timer also no longer carries `Requires=fulcrum.service`. On a timer that
|
||||||
|
means "stop watching when the watched thing stops", which is backwards for a
|
||||||
|
health check.
|
||||||
|
|
||||||
|
## Monitoring: one variable, no product knowledge
|
||||||
|
|
||||||
|
The check tests the Electrum TCP port and records the answer in its exit code,
|
||||||
|
which systemd keeps: `systemctl is-failed fulcrum-healthcheck.service`. To report
|
||||||
|
elsewhere set `healthcheck_push_url` to any endpoint accepting an HTTP ping. The
|
||||||
|
Uptime Kuma API calls, monitor creation and token handling are gone.
|
||||||
|
|
@ -12,13 +12,22 @@ fulcrum_binary_path: /usr/local/bin/Fulcrum
|
||||||
# Network - Bitcoin RPC connection
|
# Network - Bitcoin RPC connection
|
||||||
# Bitcoin Knots is on a different host (knots_box_local)
|
# Bitcoin Knots is on a different host (knots_box_local)
|
||||||
# Using RPC user/password authentication (credentials from infra_secrets.yml)
|
# Using RPC user/password authentication (credentials from infra_secrets.yml)
|
||||||
bitcoin_rpc_host: "192.168.1.140" # Bitcoin Knots RPC host (IP of knots_box_local)
|
# Addressed by Tailscale name, never a LAN IP. This was
|
||||||
|
# bitcoin_rpc_host: "192.168.1.140" # IP of knots_box_local
|
||||||
|
# but .140 is fulcrum-box ITSELF - knots-box is .135. The DHCP leases had
|
||||||
|
# reshuffled (the same drift that transposed the inventory), so running this
|
||||||
|
# playbook would have pointed Fulcrum at itself and broken indexing. The live
|
||||||
|
# config had already been hand-corrected to knots-box; this makes the repo
|
||||||
|
# agree with it.
|
||||||
|
bitcoin_rpc_host: "knots-box"
|
||||||
bitcoin_rpc_port: 8332 # Bitcoin Knots RPC port
|
bitcoin_rpc_port: 8332 # Bitcoin Knots RPC port
|
||||||
# Note: bitcoin_rpc_user and bitcoin_rpc_password are loaded from infra_secrets.yml
|
# Note: bitcoin_rpc_user and bitcoin_rpc_password are loaded from infra_secrets.yml
|
||||||
|
|
||||||
# Network - Fulcrum server
|
# Network - Fulcrum server
|
||||||
fulcrum_tcp_port: 50001
|
fulcrum_tcp_port: 50001
|
||||||
fulcrum_ssl_port: 50002
|
# Shared with the socket-proxy play on the edge host, so it lives in
|
||||||
|
# services_config.yml rather than only here.
|
||||||
|
fulcrum_ssl_port: "{{ service_settings.fulcrum.ssl_port }}"
|
||||||
# Binding address for Fulcrum TCP/SSL server:
|
# Binding address for Fulcrum TCP/SSL server:
|
||||||
# - "127.0.0.1" = localhost only (use when Caddy is on the same box)
|
# - "127.0.0.1" = localhost only (use when Caddy is on the same box)
|
||||||
# - "0.0.0.0" = all interfaces (use when Caddy is on a different box)
|
# - "0.0.0.0" = all interfaces (use when Caddy is on a different box)
|
||||||
|
|
@ -34,10 +43,14 @@ fulcrum_ssl_key_path: "{{ fulcrum_config_dir }}/fulcrum.key"
|
||||||
fulcrum_ssl_cert_days: 3650 # 10 years validity for self-signed cert
|
fulcrum_ssl_cert_days: 3650 # 10 years validity for self-signed cert
|
||||||
|
|
||||||
# Port forwarding configuration (for public access via VPS)
|
# Port forwarding configuration (for public access via VPS)
|
||||||
fulcrum_tailscale_hostname: "fulcrum-box"
|
fulcrum_tailscale_hostname: "{{ service_settings.fulcrum.tailscale_hostname }}"
|
||||||
|
|
||||||
# Performance
|
# Performance
|
||||||
# db_mem will be calculated as 75% of available RAM automatically in playbook
|
# db_mem will be calculated as 75% of available RAM automatically in playbook
|
||||||
|
# db_mem is computed as this share of RAM unless fulcrum_db_mem_mb is set
|
||||||
|
# explicitly. On a 5931 MB host 75% is 4448 MB, which leaves ~1.4 GB for the
|
||||||
|
# OS and for Fulcrum's non-cache memory; the live config had been hand-tuned
|
||||||
|
# down to 2048 and that setting is respected below.
|
||||||
fulcrum_db_mem_percent: 0.75 # 75% of RAM for database cache
|
fulcrum_db_mem_percent: 0.75 # 75% of RAM for database cache
|
||||||
|
|
||||||
# Configuration options
|
# Configuration options
|
||||||
|
|
@ -49,3 +62,17 @@ fulcrum_zmq_allow_hashtx: true # Allow ZMQ hashtx notifications
|
||||||
fulcrum_user: fulcrum
|
fulcrum_user: fulcrum
|
||||||
fulcrum_group: fulcrum
|
fulcrum_group: fulcrum
|
||||||
|
|
||||||
|
|
||||||
|
# --- Health check -----------------------------------------------------------
|
||||||
|
# Checks the Electrum TCP port and records the answer in its exit code, which
|
||||||
|
# systemd keeps: `systemctl is-failed fulcrum-healthcheck.service`.
|
||||||
|
#
|
||||||
|
# WHERE TO REPORT HEALTH — the one place to plug in monitoring. Empty means
|
||||||
|
# check, exit honestly, report nowhere. Any endpoint accepting an HTTP ping
|
||||||
|
# works; nothing here is specific to a monitoring product.
|
||||||
|
healthcheck_push_url: ""
|
||||||
|
|
||||||
|
# Explicit db_mem in MB. When set it wins over fulcrum_db_mem_percent; empty
|
||||||
|
# means compute from RAM. Set here because the live host had been hand-tuned to
|
||||||
|
# 2048 and a silent jump to 4448 is not something a refactor should do.
|
||||||
|
fulcrum_db_mem_mb_override: 2048
|
||||||
15
ansible/roles/fulcrum/handlers/main.yml
Normal file
15
ansible/roles/fulcrum/handlers/main.yml
Normal file
|
|
@ -0,0 +1,15 @@
|
||||||
|
---
|
||||||
|
# Ungated on purpose. The hand-written handler carried
|
||||||
|
# when: uptime_kuma_enabled | default(false)
|
||||||
|
# so it has been inert since the decommissioning: three tasks notify it (the SSL
|
||||||
|
# certificate, fulcrum.conf and the systemd unit), and none of them could
|
||||||
|
# actually restart Fulcrum. A configuration change therefore applied to disk and
|
||||||
|
# silently never took effect — the worst kind of quiet failure, because the
|
||||||
|
# playbook reports success and the running service keeps its old settings.
|
||||||
|
#
|
||||||
|
# Restarting Fulcrum re-opens its database; it does not reindex.
|
||||||
|
- name: Restart fulcrum
|
||||||
|
systemd:
|
||||||
|
name: fulcrum
|
||||||
|
state: restarted
|
||||||
|
daemon_reload: yes
|
||||||
62
ansible/roles/fulcrum/tasks/healthcheck.yml
Normal file
62
ansible/roles/fulcrum/tasks/healthcheck.yml
Normal file
|
|
@ -0,0 +1,62 @@
|
||||||
|
---
|
||||||
|
# Everything here answers "is Fulcrum healthy" and records the answer. The
|
||||||
|
# Uptime Kuma specifics that used to follow — an embedded Python script creating
|
||||||
|
# monitors over the API, a /tmp credentials file, push-URL extraction and a
|
||||||
|
# systemd Environment= rewrite — are gone. Where it reports is now one variable,
|
||||||
|
# healthcheck_push_url. See the role README.
|
||||||
|
- name: Create Fulcrum health check script
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: healthcheck.sh.j2
|
||||||
|
dest: /usr/local/bin/fulcrum-healthcheck-push.sh
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: '0755'
|
||||||
|
validate: "bash -n %s"
|
||||||
|
|
||||||
|
- name: Create systemd service for Fulcrum health check
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: healthcheck.service.j2
|
||||||
|
dest: /etc/systemd/system/fulcrum-healthcheck.service
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: '0644'
|
||||||
|
|
||||||
|
- name: Create systemd timer for Fulcrum health check
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: healthcheck.timer.j2
|
||||||
|
dest: /etc/systemd/system/fulcrum-healthcheck.timer
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: '0644'
|
||||||
|
|
||||||
|
- name: Reload systemd daemon for health check
|
||||||
|
systemd:
|
||||||
|
daemon_reload: yes
|
||||||
|
|
||||||
|
# state: restarted, not started. The hand-written timer had got itself stuck
|
||||||
|
# `active` with no next elapse and had not fired since 2026-02-17; `started` on
|
||||||
|
# an already-active timer is a no-op and would have left it stuck. Restarting
|
||||||
|
# re-arms it. See the note in healthcheck.timer.j2.
|
||||||
|
- name: Enable and restart the Fulcrum health check timer
|
||||||
|
systemd:
|
||||||
|
name: fulcrum-healthcheck.timer
|
||||||
|
enabled: yes
|
||||||
|
state: restarted
|
||||||
|
daemon_reload: yes
|
||||||
|
|
||||||
|
# Run the check once, which is both a smoke test and the thing that actually
|
||||||
|
# arms the timer.
|
||||||
|
#
|
||||||
|
# This timer is OnBootSec + OnUnitActiveSec with no OnCalendar. OnBootSec is
|
||||||
|
# monotonic and had long since elapsed; OnUnitActiveSec schedules relative to the
|
||||||
|
# SERVICE last being active, and the service had not run since 2026-02-17 — so
|
||||||
|
# there was no reference to schedule from and the timer sat `active` and
|
||||||
|
# `enabled` with NextElapseUSecMonotonic=infinity. Restarting the timer alone
|
||||||
|
# does not supply that reference; running the service does.
|
||||||
|
#
|
||||||
|
# (Diagnosing this is easy to get wrong: NextElapseUSecRealtime is always empty
|
||||||
|
# for a monotonic timer, so it looks broken even when it is fine. Read
|
||||||
|
# NextElapseUSecMonotonic, or just use `systemctl list-timers`.)
|
||||||
|
- name: Run the Fulcrum health check once to arm the timer
|
||||||
|
command: systemctl start fulcrum-healthcheck.service
|
||||||
|
changed_when: false
|
||||||
146
ansible/roles/fulcrum/tasks/install.yml
Normal file
146
ansible/roles/fulcrum/tasks/install.yml
Normal file
|
|
@ -0,0 +1,146 @@
|
||||||
|
---
|
||||||
|
- name: Calculate db_mem as a share of system RAM
|
||||||
|
set_fact:
|
||||||
|
fulcrum_db_mem_mb: "{{ (ansible_memtotal_mb | float * fulcrum_db_mem_percent) | int }}"
|
||||||
|
when: fulcrum_db_mem_mb_override | string | length == 0
|
||||||
|
|
||||||
|
- name: Use the explicit db_mem override
|
||||||
|
set_fact:
|
||||||
|
fulcrum_db_mem_mb: "{{ fulcrum_db_mem_mb_override }}"
|
||||||
|
when: fulcrum_db_mem_mb_override | string | length > 0
|
||||||
|
changed_when: false
|
||||||
|
|
||||||
|
- name: Display calculated db_mem value
|
||||||
|
debug:
|
||||||
|
msg: "Setting db_mem to {{ fulcrum_db_mem_mb }} MB ({{ (fulcrum_db_mem_percent * 100) | int }}% of {{ ansible_memtotal_mb }} MB total RAM)"
|
||||||
|
|
||||||
|
- name: Display Fulcrum version to install
|
||||||
|
debug:
|
||||||
|
msg: "Installing Fulcrum version {{ fulcrum_version }}"
|
||||||
|
|
||||||
|
- name: Install required packages
|
||||||
|
apt:
|
||||||
|
name:
|
||||||
|
- curl
|
||||||
|
- wget
|
||||||
|
- openssl
|
||||||
|
state: present
|
||||||
|
update_cache: yes
|
||||||
|
|
||||||
|
- name: Create fulcrum group
|
||||||
|
group:
|
||||||
|
name: "{{ fulcrum_group }}"
|
||||||
|
system: yes
|
||||||
|
state: present
|
||||||
|
|
||||||
|
- name: Create fulcrum user
|
||||||
|
user:
|
||||||
|
name: "{{ fulcrum_user }}"
|
||||||
|
group: "{{ fulcrum_group }}"
|
||||||
|
system: yes
|
||||||
|
shell: /usr/sbin/nologin
|
||||||
|
home: /home/{{ fulcrum_user }}
|
||||||
|
create_home: yes
|
||||||
|
state: present
|
||||||
|
|
||||||
|
- name: Create Fulcrum database directory (heavy data on special mount)
|
||||||
|
file:
|
||||||
|
path: "{{ fulcrum_db_dir }}"
|
||||||
|
state: directory
|
||||||
|
owner: "{{ fulcrum_user }}"
|
||||||
|
group: "{{ fulcrum_group }}"
|
||||||
|
mode: '0755'
|
||||||
|
|
||||||
|
- name: Create Fulcrum config directory
|
||||||
|
file:
|
||||||
|
path: "{{ fulcrum_config_dir }}"
|
||||||
|
state: directory
|
||||||
|
owner: root
|
||||||
|
group: "{{ fulcrum_group }}"
|
||||||
|
mode: '0755'
|
||||||
|
|
||||||
|
- name: Create Fulcrum lib directory (for banner and other data files)
|
||||||
|
file:
|
||||||
|
path: "{{ fulcrum_lib_dir }}"
|
||||||
|
state: directory
|
||||||
|
owner: "{{ fulcrum_user }}"
|
||||||
|
group: "{{ fulcrum_group }}"
|
||||||
|
mode: '0755'
|
||||||
|
|
||||||
|
# ===========================================
|
||||||
|
# SSL Certificate Generation
|
||||||
|
# ===========================================
|
||||||
|
- name: Check if SSL certificate already exists
|
||||||
|
stat:
|
||||||
|
path: "{{ fulcrum_ssl_cert_path }}"
|
||||||
|
register: fulcrum_ssl_cert_exists
|
||||||
|
when: fulcrum_ssl_enabled | default(false)
|
||||||
|
|
||||||
|
- name: Generate self-signed SSL certificate for Fulcrum
|
||||||
|
command: >
|
||||||
|
openssl req -x509 -newkey rsa:4096
|
||||||
|
-keyout {{ fulcrum_ssl_key_path }}
|
||||||
|
-out {{ fulcrum_ssl_cert_path }}
|
||||||
|
-sha256 -days {{ fulcrum_ssl_cert_days }}
|
||||||
|
-nodes
|
||||||
|
-subj "/C=XX/ST=Decentralized/L=Bitcoin/O=Fulcrum/OU=Electrum/CN=fulcrum.local"
|
||||||
|
args:
|
||||||
|
creates: "{{ fulcrum_ssl_cert_path }}"
|
||||||
|
when: fulcrum_ssl_enabled | default(false)
|
||||||
|
notify: Restart fulcrum
|
||||||
|
|
||||||
|
- name: Set SSL certificate permissions
|
||||||
|
file:
|
||||||
|
path: "{{ fulcrum_ssl_cert_path }}"
|
||||||
|
owner: "{{ fulcrum_user }}"
|
||||||
|
group: "{{ fulcrum_group }}"
|
||||||
|
mode: '0644'
|
||||||
|
when: fulcrum_ssl_enabled | default(false) and fulcrum_ssl_cert_exists.stat.exists | default(false) or fulcrum_ssl_enabled | default(false)
|
||||||
|
|
||||||
|
- name: Set SSL key permissions
|
||||||
|
file:
|
||||||
|
path: "{{ fulcrum_ssl_key_path }}"
|
||||||
|
owner: "{{ fulcrum_user }}"
|
||||||
|
group: "{{ fulcrum_group }}"
|
||||||
|
mode: '0600'
|
||||||
|
when: fulcrum_ssl_enabled | default(false)
|
||||||
|
|
||||||
|
- name: Check if Fulcrum binary already exists
|
||||||
|
stat:
|
||||||
|
path: "{{ fulcrum_binary_path }}"
|
||||||
|
register: fulcrum_binary_exists
|
||||||
|
changed_when: false
|
||||||
|
|
||||||
|
- name: Download Fulcrum binary tarball
|
||||||
|
get_url:
|
||||||
|
url: "https://github.com/cculianu/Fulcrum/releases/download/v{{ fulcrum_version }}/Fulcrum-{{ fulcrum_version }}-x86_64-linux.tar.gz"
|
||||||
|
dest: "/tmp/Fulcrum-{{ fulcrum_version }}-x86_64-linux.tar.gz"
|
||||||
|
mode: '0644'
|
||||||
|
when: not fulcrum_binary_exists.stat.exists
|
||||||
|
|
||||||
|
- name: Extract Fulcrum binary
|
||||||
|
unarchive:
|
||||||
|
src: "/tmp/Fulcrum-{{ fulcrum_version }}-x86_64-linux.tar.gz"
|
||||||
|
dest: "/tmp"
|
||||||
|
remote_src: yes
|
||||||
|
when: not fulcrum_binary_exists.stat.exists
|
||||||
|
|
||||||
|
- name: Install Fulcrum binary
|
||||||
|
copy:
|
||||||
|
src: "/tmp/Fulcrum-{{ fulcrum_version }}-x86_64-linux/Fulcrum"
|
||||||
|
dest: "{{ fulcrum_binary_path }}"
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: '0755'
|
||||||
|
remote_src: yes
|
||||||
|
when: not fulcrum_binary_exists.stat.exists
|
||||||
|
|
||||||
|
- name: Verify Fulcrum binary installation
|
||||||
|
command: "{{ fulcrum_binary_path }} --version"
|
||||||
|
register: fulcrum_version_check
|
||||||
|
changed_when: false
|
||||||
|
|
||||||
|
- name: Display Fulcrum version
|
||||||
|
debug:
|
||||||
|
msg: "{{ fulcrum_version_check.stdout_lines }}"
|
||||||
|
|
||||||
6
ansible/roles/fulcrum/tasks/main.yml
Normal file
6
ansible/roles/fulcrum/tasks/main.yml
Normal file
|
|
@ -0,0 +1,6 @@
|
||||||
|
---
|
||||||
|
# import_tasks, not include_tasks: static imports stay visible to --list-tasks,
|
||||||
|
# which is how this conversion was verified against the playbook it replaced.
|
||||||
|
- ansible.builtin.import_tasks: install.yml
|
||||||
|
- ansible.builtin.import_tasks: service.yml
|
||||||
|
- ansible.builtin.import_tasks: healthcheck.yml
|
||||||
54
ansible/roles/fulcrum/tasks/service.yml
Normal file
54
ansible/roles/fulcrum/tasks/service.yml
Normal file
|
|
@ -0,0 +1,54 @@
|
||||||
|
---
|
||||||
|
- name: Create Fulcrum banner file
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: banner.txt.j2
|
||||||
|
dest: "{{ fulcrum_lib_dir }}/fulcrum-banner.txt"
|
||||||
|
owner: "{{ fulcrum_user }}"
|
||||||
|
group: "{{ fulcrum_group }}"
|
||||||
|
mode: '0644'
|
||||||
|
|
||||||
|
- name: Create Fulcrum configuration file
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: fulcrum.conf.j2
|
||||||
|
dest: "{{ fulcrum_config_dir }}/fulcrum.conf"
|
||||||
|
owner: "{{ fulcrum_user }}"
|
||||||
|
group: "{{ fulcrum_group }}"
|
||||||
|
mode: '0640'
|
||||||
|
notify: Restart fulcrum
|
||||||
|
|
||||||
|
- name: Create systemd service file for Fulcrum
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: fulcrum.service.j2
|
||||||
|
dest: /etc/systemd/system/fulcrum.service
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
mode: '0644'
|
||||||
|
notify: Restart fulcrum
|
||||||
|
|
||||||
|
- name: Reload systemd daemon
|
||||||
|
systemd:
|
||||||
|
daemon_reload: yes
|
||||||
|
|
||||||
|
- name: Enable and start Fulcrum service
|
||||||
|
systemd:
|
||||||
|
name: fulcrum
|
||||||
|
enabled: yes
|
||||||
|
state: started
|
||||||
|
|
||||||
|
- name: Wait for Fulcrum to start
|
||||||
|
wait_for:
|
||||||
|
port: "{{ fulcrum_tcp_port }}"
|
||||||
|
host: "{{ fulcrum_tcp_bind }}"
|
||||||
|
delay: 5
|
||||||
|
timeout: 30
|
||||||
|
ignore_errors: yes
|
||||||
|
|
||||||
|
- name: Check Fulcrum service status
|
||||||
|
systemd:
|
||||||
|
name: fulcrum
|
||||||
|
register: fulcrum_service_status
|
||||||
|
changed_when: false
|
||||||
|
|
||||||
|
- name: Display Fulcrum service status
|
||||||
|
debug:
|
||||||
|
msg: "Fulcrum service is {{ 'running' if fulcrum_service_status.status.ActiveState == 'active' else 'not running' }}"
|
||||||
3
ansible/roles/fulcrum/templates/banner.txt.j2
Normal file
3
ansible/roles/fulcrum/templates/banner.txt.j2
Normal file
|
|
@ -0,0 +1,3 @@
|
||||||
|
counterinfra
|
||||||
|
|
||||||
|
PER ASPERA AD ASTRA
|
||||||
29
ansible/roles/fulcrum/templates/fulcrum.conf.j2
Normal file
29
ansible/roles/fulcrum/templates/fulcrum.conf.j2
Normal file
|
|
@ -0,0 +1,29 @@
|
||||||
|
# Fulcrum Configuration
|
||||||
|
# Generated by Ansible
|
||||||
|
|
||||||
|
# Bitcoin Core/Knots RPC settings
|
||||||
|
bitcoind = {{ bitcoin_rpc_host }}:{{ bitcoin_rpc_port }}
|
||||||
|
rpcuser = {{ bitcoin_rpc_user }}
|
||||||
|
rpcpassword = {{ bitcoin_rpc_password }}
|
||||||
|
|
||||||
|
# Fulcrum server general settings
|
||||||
|
datadir = {{ fulcrum_db_dir }}
|
||||||
|
tcp = {{ fulcrum_tcp_bind }}:{{ fulcrum_tcp_port }}
|
||||||
|
peering = {{ 'true' if fulcrum_peering else 'false' }}
|
||||||
|
zmq_allow_hashtx = {{ 'true' if fulcrum_zmq_allow_hashtx else 'false' }}
|
||||||
|
|
||||||
|
# SSL/TLS Configuration
|
||||||
|
{% if fulcrum_ssl_enabled | default(false) %}
|
||||||
|
ssl = {{ fulcrum_ssl_bind }}:{{ fulcrum_ssl_port }}
|
||||||
|
cert = {{ fulcrum_ssl_cert_path }}
|
||||||
|
key = {{ fulcrum_ssl_key_path }}
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
# Anonymize client IP addresses and TxIDs in logs
|
||||||
|
anon_logs = {{ 'true' if fulcrum_anon_logs else 'false' }}
|
||||||
|
|
||||||
|
# Max RocksDB Memory in MiB
|
||||||
|
db_mem = {{ fulcrum_db_mem_mb }}.0
|
||||||
|
|
||||||
|
# Banner
|
||||||
|
banner = {{ fulcrum_lib_dir }}/fulcrum-banner.txt
|
||||||
24
ansible/roles/fulcrum/templates/fulcrum.service.j2
Normal file
24
ansible/roles/fulcrum/templates/fulcrum.service.j2
Normal file
|
|
@ -0,0 +1,24 @@
|
||||||
|
# MiniBolt: systemd unit for Fulcrum
|
||||||
|
# /etc/systemd/system/fulcrum.service
|
||||||
|
|
||||||
|
[Unit]
|
||||||
|
Description=Fulcrum
|
||||||
|
After=network.target
|
||||||
|
|
||||||
|
StartLimitBurst=2
|
||||||
|
StartLimitIntervalSec=20
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
ExecStart={{ fulcrum_binary_path }} {{ fulcrum_config_dir }}/fulcrum.conf
|
||||||
|
|
||||||
|
User={{ fulcrum_user }}
|
||||||
|
Group={{ fulcrum_group }}
|
||||||
|
|
||||||
|
# Process management
|
||||||
|
####################
|
||||||
|
Type=simple
|
||||||
|
KillSignal=SIGINT
|
||||||
|
TimeoutStopSec=300
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=multi-user.target
|
||||||
14
ansible/roles/fulcrum/templates/healthcheck.service.j2
Normal file
14
ansible/roles/fulcrum/templates/healthcheck.service.j2
Normal file
|
|
@ -0,0 +1,14 @@
|
||||||
|
[Unit]
|
||||||
|
Description=Fulcrum Health Check
|
||||||
|
After=network.target fulcrum.service
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=oneshot
|
||||||
|
User=root
|
||||||
|
ExecStart=/usr/local/bin/fulcrum-healthcheck-push.sh
|
||||||
|
Environment=HEALTHCHECK_PUSH_URL={{ healthcheck_push_url }}
|
||||||
|
StandardOutput=journal
|
||||||
|
StandardError=journal
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=multi-user.target
|
||||||
34
ansible/roles/fulcrum/templates/healthcheck.sh.j2
Normal file
34
ansible/roles/fulcrum/templates/healthcheck.sh.j2
Normal file
|
|
@ -0,0 +1,34 @@
|
||||||
|
#!/bin/bash
|
||||||
|
# Fulcrum health check — managed by Ansible (roles/fulcrum)
|
||||||
|
#
|
||||||
|
# Checks that Fulcrum's Electrum TCP port is accepting connections, and records
|
||||||
|
# the answer in the exit code, which systemd keeps:
|
||||||
|
# systemctl is-failed fulcrum-healthcheck.service
|
||||||
|
# That is a complete answer with no monitoring system involved. Reporting
|
||||||
|
# elsewhere is optional and generic — set healthcheck_push_url.
|
||||||
|
|
||||||
|
FULCRUM_HOST="{{ fulcrum_tcp_bind }}"
|
||||||
|
FULCRUM_PORT={{ fulcrum_tcp_port }}
|
||||||
|
PUSH_URL="${HEALTHCHECK_PUSH_URL:-}"
|
||||||
|
|
||||||
|
check_fulcrum() {
|
||||||
|
timeout 5 bash -c "echo > /dev/tcp/${FULCRUM_HOST}/${FULCRUM_PORT}" 2>/dev/null
|
||||||
|
}
|
||||||
|
|
||||||
|
report() {
|
||||||
|
local status=$1 msg=$2
|
||||||
|
# No push URL is normal, not an error: the exit code below is still a
|
||||||
|
# complete answer for anything reading unit state.
|
||||||
|
[ -n "$PUSH_URL" ] || return 0
|
||||||
|
curl -s --max-time 10 --retry 2 -o /dev/null \
|
||||||
|
"${PUSH_URL}?status=${status}&msg=${msg// /%20}&ping=" || true
|
||||||
|
}
|
||||||
|
|
||||||
|
if check_fulcrum; then
|
||||||
|
report "up" "OK"
|
||||||
|
exit 0
|
||||||
|
else
|
||||||
|
echo "Fulcrum TCP port ${FULCRUM_PORT} not responding"
|
||||||
|
report "down" "Fulcrum TCP port not responding"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
17
ansible/roles/fulcrum/templates/healthcheck.timer.j2
Normal file
17
ansible/roles/fulcrum/templates/healthcheck.timer.j2
Normal file
|
|
@ -0,0 +1,17 @@
|
||||||
|
[Unit]
|
||||||
|
Description=Fulcrum Health Check Timer
|
||||||
|
# NOTE: this deliberately does NOT carry `Requires=fulcrum.service`, which the
|
||||||
|
# hand-written unit had. Requires on a timer means the timer is stopped when the
|
||||||
|
# required unit stops — i.e. "if the thing I am watching goes down, stop
|
||||||
|
# watching it", which is backwards for a health check and leaves nothing to
|
||||||
|
# re-arm the timer when the service returns. The live timer had been `active`
|
||||||
|
# and `enabled` with NextElapseUSecMonotonic=infinity and a last trigger of
|
||||||
|
# 2026-02-17: seven months with no health check and no outward sign of it.
|
||||||
|
|
||||||
|
[Timer]
|
||||||
|
OnBootSec=1min
|
||||||
|
OnUnitActiveSec=1min
|
||||||
|
Persistent=true
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=timers.target
|
||||||
|
|
@ -1,3 +1,7 @@
|
||||||
|
---
|
||||||
|
# Fulcrum: Electrum server indexing the Bitcoin Knots node.
|
||||||
|
# The index takes days to rebuild, so nothing here touches its data directory
|
||||||
|
# beyond asserting that it exists.
|
||||||
- name: Deploy Fulcrum Electrum Server
|
- name: Deploy Fulcrum Electrum Server
|
||||||
hosts: electrum
|
hosts: electrum
|
||||||
become: yes
|
become: yes
|
||||||
|
|
@ -5,540 +9,12 @@
|
||||||
- ../../infra_vars.yml
|
- ../../infra_vars.yml
|
||||||
- ../../services_config.yml
|
- ../../services_config.yml
|
||||||
- ../../infra_secrets.yml
|
- ../../infra_secrets.yml
|
||||||
- ./fulcrum_vars.yml
|
|
||||||
vars:
|
vars:
|
||||||
uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}"
|
# Preserves the push URL this check has been configured with. The role knows
|
||||||
|
# nothing about Uptime Kuma — this is just "a URL that accepts a ping".
|
||||||
tasks:
|
healthcheck_push_url: "{{ healthcheck_push_urls.fulcrum | default('') }}"
|
||||||
- name: Calculate 75% of system RAM for db_mem
|
roles:
|
||||||
set_fact:
|
- fulcrum
|
||||||
fulcrum_db_mem_mb: "{{ (ansible_memtotal_mb | float * fulcrum_db_mem_percent) | int }}"
|
|
||||||
changed_when: false
|
|
||||||
|
|
||||||
- name: Display calculated db_mem value
|
|
||||||
debug:
|
|
||||||
msg: "Setting db_mem to {{ fulcrum_db_mem_mb }} MB ({{ (fulcrum_db_mem_percent * 100) | int }}% of {{ ansible_memtotal_mb }} MB total RAM)"
|
|
||||||
|
|
||||||
- name: Display Fulcrum version to install
|
|
||||||
debug:
|
|
||||||
msg: "Installing Fulcrum version {{ fulcrum_version }}"
|
|
||||||
|
|
||||||
- name: Install required packages
|
|
||||||
apt:
|
|
||||||
name:
|
|
||||||
- curl
|
|
||||||
- wget
|
|
||||||
- openssl
|
|
||||||
state: present
|
|
||||||
update_cache: yes
|
|
||||||
|
|
||||||
- name: Create fulcrum group
|
|
||||||
group:
|
|
||||||
name: "{{ fulcrum_group }}"
|
|
||||||
system: yes
|
|
||||||
state: present
|
|
||||||
|
|
||||||
- name: Create fulcrum user
|
|
||||||
user:
|
|
||||||
name: "{{ fulcrum_user }}"
|
|
||||||
group: "{{ fulcrum_group }}"
|
|
||||||
system: yes
|
|
||||||
shell: /usr/sbin/nologin
|
|
||||||
home: /home/{{ fulcrum_user }}
|
|
||||||
create_home: yes
|
|
||||||
state: present
|
|
||||||
|
|
||||||
- name: Create Fulcrum database directory (heavy data on special mount)
|
|
||||||
file:
|
|
||||||
path: "{{ fulcrum_db_dir }}"
|
|
||||||
state: directory
|
|
||||||
owner: "{{ fulcrum_user }}"
|
|
||||||
group: "{{ fulcrum_group }}"
|
|
||||||
mode: '0755'
|
|
||||||
|
|
||||||
- name: Create Fulcrum config directory
|
|
||||||
file:
|
|
||||||
path: "{{ fulcrum_config_dir }}"
|
|
||||||
state: directory
|
|
||||||
owner: root
|
|
||||||
group: "{{ fulcrum_group }}"
|
|
||||||
mode: '0755'
|
|
||||||
|
|
||||||
- name: Create Fulcrum lib directory (for banner and other data files)
|
|
||||||
file:
|
|
||||||
path: "{{ fulcrum_lib_dir }}"
|
|
||||||
state: directory
|
|
||||||
owner: "{{ fulcrum_user }}"
|
|
||||||
group: "{{ fulcrum_group }}"
|
|
||||||
mode: '0755'
|
|
||||||
|
|
||||||
# ===========================================
|
|
||||||
# SSL Certificate Generation
|
|
||||||
# ===========================================
|
|
||||||
- name: Check if SSL certificate already exists
|
|
||||||
stat:
|
|
||||||
path: "{{ fulcrum_ssl_cert_path }}"
|
|
||||||
register: fulcrum_ssl_cert_exists
|
|
||||||
when: fulcrum_ssl_enabled | default(false)
|
|
||||||
|
|
||||||
- name: Generate self-signed SSL certificate for Fulcrum
|
|
||||||
command: >
|
|
||||||
openssl req -x509 -newkey rsa:4096
|
|
||||||
-keyout {{ fulcrum_ssl_key_path }}
|
|
||||||
-out {{ fulcrum_ssl_cert_path }}
|
|
||||||
-sha256 -days {{ fulcrum_ssl_cert_days }}
|
|
||||||
-nodes
|
|
||||||
-subj "/C=XX/ST=Decentralized/L=Bitcoin/O=Fulcrum/OU=Electrum/CN=fulcrum.local"
|
|
||||||
args:
|
|
||||||
creates: "{{ fulcrum_ssl_cert_path }}"
|
|
||||||
when: fulcrum_ssl_enabled | default(false)
|
|
||||||
notify: Restart fulcrum
|
|
||||||
|
|
||||||
- name: Set SSL certificate permissions
|
|
||||||
file:
|
|
||||||
path: "{{ fulcrum_ssl_cert_path }}"
|
|
||||||
owner: "{{ fulcrum_user }}"
|
|
||||||
group: "{{ fulcrum_group }}"
|
|
||||||
mode: '0644'
|
|
||||||
when: fulcrum_ssl_enabled | default(false) and fulcrum_ssl_cert_exists.stat.exists | default(false) or fulcrum_ssl_enabled | default(false)
|
|
||||||
|
|
||||||
- name: Set SSL key permissions
|
|
||||||
file:
|
|
||||||
path: "{{ fulcrum_ssl_key_path }}"
|
|
||||||
owner: "{{ fulcrum_user }}"
|
|
||||||
group: "{{ fulcrum_group }}"
|
|
||||||
mode: '0600'
|
|
||||||
when: fulcrum_ssl_enabled | default(false)
|
|
||||||
|
|
||||||
- name: Check if Fulcrum binary already exists
|
|
||||||
stat:
|
|
||||||
path: "{{ fulcrum_binary_path }}"
|
|
||||||
register: fulcrum_binary_exists
|
|
||||||
changed_when: false
|
|
||||||
|
|
||||||
- name: Download Fulcrum binary tarball
|
|
||||||
get_url:
|
|
||||||
url: "https://github.com/cculianu/Fulcrum/releases/download/v{{ fulcrum_version }}/Fulcrum-{{ fulcrum_version }}-x86_64-linux.tar.gz"
|
|
||||||
dest: "/tmp/Fulcrum-{{ fulcrum_version }}-x86_64-linux.tar.gz"
|
|
||||||
mode: '0644'
|
|
||||||
when: not fulcrum_binary_exists.stat.exists
|
|
||||||
|
|
||||||
- name: Extract Fulcrum binary
|
|
||||||
unarchive:
|
|
||||||
src: "/tmp/Fulcrum-{{ fulcrum_version }}-x86_64-linux.tar.gz"
|
|
||||||
dest: "/tmp"
|
|
||||||
remote_src: yes
|
|
||||||
when: not fulcrum_binary_exists.stat.exists
|
|
||||||
|
|
||||||
- name: Install Fulcrum binary
|
|
||||||
copy:
|
|
||||||
src: "/tmp/Fulcrum-{{ fulcrum_version }}-x86_64-linux/Fulcrum"
|
|
||||||
dest: "{{ fulcrum_binary_path }}"
|
|
||||||
owner: root
|
|
||||||
group: root
|
|
||||||
mode: '0755'
|
|
||||||
remote_src: yes
|
|
||||||
when: not fulcrum_binary_exists.stat.exists
|
|
||||||
|
|
||||||
- name: Verify Fulcrum binary installation
|
|
||||||
command: "{{ fulcrum_binary_path }} --version"
|
|
||||||
register: fulcrum_version_check
|
|
||||||
changed_when: false
|
|
||||||
|
|
||||||
- name: Display Fulcrum version
|
|
||||||
debug:
|
|
||||||
msg: "{{ fulcrum_version_check.stdout_lines }}"
|
|
||||||
|
|
||||||
- name: Create Fulcrum banner file
|
|
||||||
copy:
|
|
||||||
dest: "{{ fulcrum_lib_dir }}/fulcrum-banner.txt"
|
|
||||||
content: |
|
|
||||||
counterinfra
|
|
||||||
|
|
||||||
PER ASPERA AD ASTRA
|
|
||||||
owner: "{{ fulcrum_user }}"
|
|
||||||
group: "{{ fulcrum_group }}"
|
|
||||||
mode: '0644'
|
|
||||||
|
|
||||||
- name: Create Fulcrum configuration file
|
|
||||||
copy:
|
|
||||||
dest: "{{ fulcrum_config_dir }}/fulcrum.conf"
|
|
||||||
content: |
|
|
||||||
# Fulcrum Configuration
|
|
||||||
# Generated by Ansible
|
|
||||||
|
|
||||||
# Bitcoin Core/Knots RPC settings
|
|
||||||
bitcoind = {{ bitcoin_rpc_host }}:{{ bitcoin_rpc_port }}
|
|
||||||
rpcuser = {{ bitcoin_rpc_user }}
|
|
||||||
rpcpassword = {{ bitcoin_rpc_password }}
|
|
||||||
|
|
||||||
# Fulcrum server general settings
|
|
||||||
datadir = {{ fulcrum_db_dir }}
|
|
||||||
tcp = {{ fulcrum_tcp_bind }}:{{ fulcrum_tcp_port }}
|
|
||||||
peering = {{ 'true' if fulcrum_peering else 'false' }}
|
|
||||||
zmq_allow_hashtx = {{ 'true' if fulcrum_zmq_allow_hashtx else 'false' }}
|
|
||||||
|
|
||||||
# SSL/TLS Configuration
|
|
||||||
{% if fulcrum_ssl_enabled | default(false) %}
|
|
||||||
ssl = {{ fulcrum_ssl_bind }}:{{ fulcrum_ssl_port }}
|
|
||||||
cert = {{ fulcrum_ssl_cert_path }}
|
|
||||||
key = {{ fulcrum_ssl_key_path }}
|
|
||||||
{% endif %}
|
|
||||||
|
|
||||||
# Anonymize client IP addresses and TxIDs in logs
|
|
||||||
anon_logs = {{ 'true' if fulcrum_anon_logs else 'false' }}
|
|
||||||
|
|
||||||
# Max RocksDB Memory in MiB
|
|
||||||
db_mem = {{ fulcrum_db_mem_mb }}.0
|
|
||||||
|
|
||||||
# Banner
|
|
||||||
banner = {{ fulcrum_lib_dir }}/fulcrum-banner.txt
|
|
||||||
owner: "{{ fulcrum_user }}"
|
|
||||||
group: "{{ fulcrum_group }}"
|
|
||||||
mode: '0640'
|
|
||||||
notify: Restart fulcrum
|
|
||||||
|
|
||||||
- name: Create systemd service file for Fulcrum
|
|
||||||
copy:
|
|
||||||
dest: /etc/systemd/system/fulcrum.service
|
|
||||||
content: |
|
|
||||||
# MiniBolt: systemd unit for Fulcrum
|
|
||||||
# /etc/systemd/system/fulcrum.service
|
|
||||||
|
|
||||||
[Unit]
|
|
||||||
Description=Fulcrum
|
|
||||||
After=network.target
|
|
||||||
|
|
||||||
StartLimitBurst=2
|
|
||||||
StartLimitIntervalSec=20
|
|
||||||
|
|
||||||
[Service]
|
|
||||||
ExecStart={{ fulcrum_binary_path }} {{ fulcrum_config_dir }}/fulcrum.conf
|
|
||||||
|
|
||||||
User={{ fulcrum_user }}
|
|
||||||
Group={{ fulcrum_group }}
|
|
||||||
|
|
||||||
# Process management
|
|
||||||
####################
|
|
||||||
Type=simple
|
|
||||||
KillSignal=SIGINT
|
|
||||||
TimeoutStopSec=300
|
|
||||||
|
|
||||||
[Install]
|
|
||||||
WantedBy=multi-user.target
|
|
||||||
owner: root
|
|
||||||
group: root
|
|
||||||
mode: '0644'
|
|
||||||
notify: Restart fulcrum
|
|
||||||
|
|
||||||
- name: Reload systemd daemon
|
|
||||||
systemd:
|
|
||||||
daemon_reload: yes
|
|
||||||
|
|
||||||
- name: Enable and start Fulcrum service
|
|
||||||
systemd:
|
|
||||||
name: fulcrum
|
|
||||||
enabled: yes
|
|
||||||
state: started
|
|
||||||
|
|
||||||
- name: Wait for Fulcrum to start
|
|
||||||
wait_for:
|
|
||||||
port: "{{ fulcrum_tcp_port }}"
|
|
||||||
host: "{{ fulcrum_tcp_bind }}"
|
|
||||||
delay: 5
|
|
||||||
timeout: 30
|
|
||||||
ignore_errors: yes
|
|
||||||
|
|
||||||
- name: Check Fulcrum service status
|
|
||||||
systemd:
|
|
||||||
name: fulcrum
|
|
||||||
register: fulcrum_service_status
|
|
||||||
changed_when: false
|
|
||||||
|
|
||||||
- name: Display Fulcrum service status
|
|
||||||
debug:
|
|
||||||
msg: "Fulcrum service is {{ 'running' if fulcrum_service_status.status.ActiveState == 'active' else 'not running' }}"
|
|
||||||
|
|
||||||
# ═════════════════════════════════════════════════════════════════════════
|
|
||||||
# DEPRECATED — Uptime Kuma was decommissioned on 2026-09-11.
|
|
||||||
#
|
|
||||||
# Every task below is inert: uptime_kuma_enabled is false in
|
|
||||||
# group_vars/all/main.yml, so they all skip and the deployment above still
|
|
||||||
# runs normally. Kept because the health-check logic is the durable part —
|
|
||||||
# when a replacement exists, rewire the push transport and flip the flag.
|
|
||||||
#
|
|
||||||
# What was being monitored: archive/uptime_kuma/MONITORS.md
|
|
||||||
# ═════════════════════════════════════════════════════════════════════════
|
|
||||||
- name: Create Fulcrum health check and push script
|
|
||||||
when: uptime_kuma_enabled | default(false)
|
|
||||||
copy:
|
|
||||||
dest: /usr/local/bin/fulcrum-healthcheck-push.sh
|
|
||||||
content: |
|
|
||||||
#!/bin/bash
|
|
||||||
#
|
|
||||||
# Fulcrum Health Check and Push to Uptime Kuma
|
|
||||||
# Checks if Fulcrum TCP port is responding and pushes status to Uptime Kuma
|
|
||||||
#
|
|
||||||
|
|
||||||
FULCRUM_HOST="{{ fulcrum_tcp_bind }}"
|
|
||||||
FULCRUM_PORT={{ fulcrum_tcp_port }}
|
|
||||||
UPTIME_KUMA_PUSH_URL="${UPTIME_KUMA_PUSH_URL}"
|
|
||||||
|
|
||||||
# Check if Fulcrum TCP port is responding
|
|
||||||
check_fulcrum() {
|
|
||||||
# Try to connect to TCP port
|
|
||||||
timeout 5 bash -c "echo > /dev/tcp/${FULCRUM_HOST}/${FULCRUM_PORT}" 2>/dev/null
|
|
||||||
return $?
|
|
||||||
}
|
|
||||||
|
|
||||||
# Push status to Uptime Kuma
|
|
||||||
push_to_uptime_kuma() {
|
|
||||||
local status=$1
|
|
||||||
local msg=$2
|
|
||||||
|
|
||||||
if [ -z "$UPTIME_KUMA_PUSH_URL" ]; then
|
|
||||||
echo "ERROR: UPTIME_KUMA_PUSH_URL not set"
|
|
||||||
return 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# URL encode spaces in message
|
|
||||||
local encoded_msg="${msg// /%20}"
|
|
||||||
|
|
||||||
if ! curl -s --max-time 10 --retry 2 -o /dev/null \
|
|
||||||
"${UPTIME_KUMA_PUSH_URL}?status=${status}&msg=${encoded_msg}&ping="; then
|
|
||||||
echo "ERROR: Failed to push to Uptime Kuma"
|
|
||||||
return 1
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
# Main health check
|
|
||||||
if check_fulcrum; then
|
|
||||||
push_to_uptime_kuma "up" "OK"
|
|
||||||
exit 0
|
|
||||||
else
|
|
||||||
push_to_uptime_kuma "down" "Fulcrum TCP port not responding"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
owner: root
|
|
||||||
group: root
|
|
||||||
mode: '0755'
|
|
||||||
|
|
||||||
- name: Create systemd timer for Fulcrum health check
|
|
||||||
copy:
|
|
||||||
dest: /etc/systemd/system/fulcrum-healthcheck.timer
|
|
||||||
content: |
|
|
||||||
[Unit]
|
|
||||||
Description=Fulcrum Health Check Timer
|
|
||||||
Requires=fulcrum.service
|
|
||||||
|
|
||||||
[Timer]
|
|
||||||
OnBootSec=1min
|
|
||||||
OnUnitActiveSec=1min
|
|
||||||
Persistent=true
|
|
||||||
|
|
||||||
[Install]
|
|
||||||
WantedBy=timers.target
|
|
||||||
owner: root
|
|
||||||
group: root
|
|
||||||
mode: '0644'
|
|
||||||
|
|
||||||
- name: Create systemd service for Fulcrum health check
|
|
||||||
when: uptime_kuma_enabled | default(false)
|
|
||||||
copy:
|
|
||||||
dest: /etc/systemd/system/fulcrum-healthcheck.service
|
|
||||||
content: |
|
|
||||||
[Unit]
|
|
||||||
Description=Fulcrum Health Check and Push to Uptime Kuma
|
|
||||||
After=network.target fulcrum.service
|
|
||||||
|
|
||||||
[Service]
|
|
||||||
Type=oneshot
|
|
||||||
User=root
|
|
||||||
ExecStart=/usr/local/bin/fulcrum-healthcheck-push.sh
|
|
||||||
Environment=UPTIME_KUMA_PUSH_URL=
|
|
||||||
StandardOutput=journal
|
|
||||||
StandardError=journal
|
|
||||||
|
|
||||||
[Install]
|
|
||||||
WantedBy=multi-user.target
|
|
||||||
owner: root
|
|
||||||
group: root
|
|
||||||
mode: '0644'
|
|
||||||
|
|
||||||
- name: Reload systemd daemon for health check
|
|
||||||
systemd:
|
|
||||||
daemon_reload: yes
|
|
||||||
|
|
||||||
- name: Enable and start Fulcrum health check timer
|
|
||||||
systemd:
|
|
||||||
name: fulcrum-healthcheck.timer
|
|
||||||
enabled: yes
|
|
||||||
state: started
|
|
||||||
|
|
||||||
- name: Create Uptime Kuma push monitor setup script for Fulcrum
|
|
||||||
when: uptime_kuma_enabled | default(false)
|
|
||||||
delegate_to: localhost
|
|
||||||
become: no
|
|
||||||
copy:
|
|
||||||
dest: /tmp/setup_fulcrum_monitor.py
|
|
||||||
content: |
|
|
||||||
#!/usr/bin/env python3
|
|
||||||
import sys
|
|
||||||
import traceback
|
|
||||||
import yaml
|
|
||||||
from uptime_kuma_api import UptimeKumaApi, MonitorType
|
|
||||||
|
|
||||||
try:
|
|
||||||
# Load configs
|
|
||||||
with open('/tmp/ansible_config.yml', 'r') as f:
|
|
||||||
config = yaml.safe_load(f)
|
|
||||||
|
|
||||||
url = config['uptime_kuma_url']
|
|
||||||
username = config['username']
|
|
||||||
password = config['password']
|
|
||||||
monitor_name = config['monitor_name']
|
|
||||||
|
|
||||||
# Connect to Uptime Kuma
|
|
||||||
api = UptimeKumaApi(url, timeout=30)
|
|
||||||
api.login(username, password)
|
|
||||||
|
|
||||||
# Get all monitors
|
|
||||||
monitors = api.get_monitors()
|
|
||||||
|
|
||||||
# Find or create "services" group
|
|
||||||
group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None)
|
|
||||||
if not group:
|
|
||||||
group_result = api.add_monitor(type='group', name='services')
|
|
||||||
# Refresh to get the group with id
|
|
||||||
monitors = api.get_monitors()
|
|
||||||
group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None)
|
|
||||||
|
|
||||||
# Check if monitor already exists
|
|
||||||
existing_monitor = None
|
|
||||||
for monitor in monitors:
|
|
||||||
if monitor.get('name') == monitor_name:
|
|
||||||
existing_monitor = monitor
|
|
||||||
break
|
|
||||||
|
|
||||||
# Get ntfy notification ID
|
|
||||||
notifications = api.get_notifications()
|
|
||||||
ntfy_notification_id = None
|
|
||||||
for notif in notifications:
|
|
||||||
if notif.get('type') == 'ntfy':
|
|
||||||
ntfy_notification_id = notif.get('id')
|
|
||||||
break
|
|
||||||
|
|
||||||
if existing_monitor:
|
|
||||||
print(f"Monitor '{monitor_name}' already exists (ID: {existing_monitor['id']})")
|
|
||||||
push_token = existing_monitor.get('pushToken') or existing_monitor.get('push_token')
|
|
||||||
if not push_token:
|
|
||||||
raise ValueError("Could not find push token for monitor")
|
|
||||||
push_url = f"{url}/api/push/{push_token}"
|
|
||||||
print(f"Push URL: {push_url}")
|
|
||||||
else:
|
|
||||||
print(f"Creating push monitor '{monitor_name}'...")
|
|
||||||
api.add_monitor(
|
|
||||||
type=MonitorType.PUSH,
|
|
||||||
name=monitor_name,
|
|
||||||
parent=group['id'],
|
|
||||||
interval=60,
|
|
||||||
maxretries=3,
|
|
||||||
retryInterval=60,
|
|
||||||
notificationIDList={ntfy_notification_id: True} if ntfy_notification_id else {}
|
|
||||||
)
|
|
||||||
monitors = api.get_monitors()
|
|
||||||
new_monitor = next((m for m in monitors if m.get('name') == monitor_name), None)
|
|
||||||
if new_monitor:
|
|
||||||
push_token = new_monitor.get('pushToken') or new_monitor.get('push_token')
|
|
||||||
if not push_token:
|
|
||||||
raise ValueError("Could not find push token for new monitor")
|
|
||||||
push_url = f"{url}/api/push/{push_token}"
|
|
||||||
print(f"Push URL: {push_url}")
|
|
||||||
|
|
||||||
api.disconnect()
|
|
||||||
print("SUCCESS")
|
|
||||||
|
|
||||||
except Exception as e:
|
|
||||||
error_msg = str(e) if str(e) else repr(e)
|
|
||||||
print(f"ERROR: {error_msg}", file=sys.stderr)
|
|
||||||
traceback.print_exc(file=sys.stderr)
|
|
||||||
sys.exit(1)
|
|
||||||
mode: '0755'
|
|
||||||
|
|
||||||
- name: Create temporary config for monitor setup
|
|
||||||
when: uptime_kuma_enabled | default(false)
|
|
||||||
delegate_to: localhost
|
|
||||||
become: no
|
|
||||||
copy:
|
|
||||||
dest: /tmp/ansible_config.yml
|
|
||||||
content: |
|
|
||||||
uptime_kuma_url: "{{ uptime_kuma_api_url }}"
|
|
||||||
username: "{{ uptime_kuma_username }}"
|
|
||||||
password: "{{ uptime_kuma_password }}"
|
|
||||||
monitor_name: "Fulcrum"
|
|
||||||
mode: '0644'
|
|
||||||
|
|
||||||
- name: Run Uptime Kuma push monitor setup
|
|
||||||
when: uptime_kuma_enabled | default(false)
|
|
||||||
command: python3 /tmp/setup_fulcrum_monitor.py
|
|
||||||
delegate_to: localhost
|
|
||||||
become: no
|
|
||||||
register: monitor_setup
|
|
||||||
changed_when: "'SUCCESS' in monitor_setup.stdout"
|
|
||||||
ignore_errors: yes
|
|
||||||
|
|
||||||
- name: Extract push URL from monitor setup output
|
|
||||||
set_fact:
|
|
||||||
uptime_kuma_push_url: "{{ monitor_setup.stdout | regex_search('Push URL: (https?://[^\\s]+)', '\\1') | first | default('') }}"
|
|
||||||
delegate_to: localhost
|
|
||||||
become: no
|
|
||||||
when: monitor_setup.stdout is defined
|
|
||||||
|
|
||||||
- name: Display extracted push URL
|
|
||||||
debug:
|
|
||||||
msg: "Uptime Kuma Push URL: {{ uptime_kuma_push_url }}"
|
|
||||||
when: uptime_kuma_push_url | default('') != ''
|
|
||||||
|
|
||||||
- name: Set push URL in systemd service environment
|
|
||||||
lineinfile:
|
|
||||||
path: /etc/systemd/system/fulcrum-healthcheck.service
|
|
||||||
regexp: '^Environment=UPTIME_KUMA_PUSH_URL='
|
|
||||||
line: "Environment=UPTIME_KUMA_PUSH_URL={{ uptime_kuma_push_url }}"
|
|
||||||
state: present
|
|
||||||
insertafter: '^\[Service\]'
|
|
||||||
when: uptime_kuma_push_url | default('') != ''
|
|
||||||
|
|
||||||
- name: Reload systemd daemon after push URL update
|
|
||||||
systemd:
|
|
||||||
daemon_reload: yes
|
|
||||||
when: uptime_kuma_push_url | default('') != ''
|
|
||||||
|
|
||||||
- name: Restart health check timer to pick up new environment
|
|
||||||
systemd:
|
|
||||||
name: fulcrum-healthcheck.timer
|
|
||||||
state: restarted
|
|
||||||
when: uptime_kuma_push_url | default('') != ''
|
|
||||||
|
|
||||||
- name: Clean up temporary files
|
|
||||||
when: uptime_kuma_enabled | default(false)
|
|
||||||
delegate_to: localhost
|
|
||||||
become: no
|
|
||||||
file:
|
|
||||||
path: "{{ item }}"
|
|
||||||
state: absent
|
|
||||||
loop:
|
|
||||||
- /tmp/setup_fulcrum_monitor.py
|
|
||||||
- /tmp/ansible_config.yml
|
|
||||||
- /tmp/Fulcrum-{{ fulcrum_version }}-x86_64-linux.tar.gz
|
|
||||||
- /tmp/Fulcrum-{{ fulcrum_version }}-x86_64-linux
|
|
||||||
|
|
||||||
handlers:
|
|
||||||
- name: Restart fulcrum
|
|
||||||
when: uptime_kuma_enabled | default(false)
|
|
||||||
systemd:
|
|
||||||
name: fulcrum
|
|
||||||
state: restarted
|
|
||||||
|
|
||||||
|
|
||||||
- name: Setup public Fulcrum SSL forwarding on the edge host
|
- name: Setup public Fulcrum SSL forwarding on the edge host
|
||||||
hosts: edge
|
hosts: edge
|
||||||
|
|
@ -546,11 +22,6 @@
|
||||||
vars_files:
|
vars_files:
|
||||||
- ../../infra_vars.yml
|
- ../../infra_vars.yml
|
||||||
- ../../services_config.yml
|
- ../../services_config.yml
|
||||||
- ../../infra_secrets.yml
|
|
||||||
- ./fulcrum_vars.yml
|
|
||||||
vars:
|
|
||||||
uptime_kuma_api_url: "https://{{ subdomains.uptime_kuma }}.{{ root_domain }}"
|
|
||||||
|
|
||||||
tasks:
|
tasks:
|
||||||
- name: Expose Fulcrum SSL through a socket proxy
|
- name: Expose Fulcrum SSL through a socket proxy
|
||||||
ansible.builtin.include_role:
|
ansible.builtin.include_role:
|
||||||
|
|
@ -558,128 +29,5 @@
|
||||||
vars:
|
vars:
|
||||||
socket_proxy_name: fulcrum-ssl
|
socket_proxy_name: fulcrum-ssl
|
||||||
socket_proxy_description: "Fulcrum SSL"
|
socket_proxy_description: "Fulcrum SSL"
|
||||||
socket_proxy_listen_port: "{{ fulcrum_ssl_port }}"
|
socket_proxy_listen_port: "{{ service_settings.fulcrum.ssl_port }}"
|
||||||
socket_proxy_upstream_host: "{{ fulcrum_tailscale_hostname }}"
|
socket_proxy_upstream_host: "{{ service_settings.fulcrum.tailscale_hostname }}"
|
||||||
|
|
||||||
- name: Display public endpoint
|
|
||||||
when: uptime_kuma_enabled | default(false)
|
|
||||||
debug:
|
|
||||||
msg: "Fulcrum SSL public endpoint: {{ ansible_host }}:{{ fulcrum_ssl_port }}"
|
|
||||||
|
|
||||||
# ===========================================
|
|
||||||
# Uptime Kuma TCP Monitor for Public SSL Port
|
|
||||||
# ===========================================
|
|
||||||
- name: Create Uptime Kuma TCP monitor setup script for Fulcrum SSL
|
|
||||||
when: uptime_kuma_enabled | default(false)
|
|
||||||
delegate_to: localhost
|
|
||||||
become: no
|
|
||||||
copy:
|
|
||||||
dest: /tmp/setup_fulcrum_ssl_tcp_monitor.py
|
|
||||||
content: |
|
|
||||||
#!/usr/bin/env python3
|
|
||||||
import sys
|
|
||||||
import traceback
|
|
||||||
import yaml
|
|
||||||
from uptime_kuma_api import UptimeKumaApi, MonitorType
|
|
||||||
|
|
||||||
try:
|
|
||||||
with open('/tmp/ansible_fulcrum_ssl_config.yml', 'r') as f:
|
|
||||||
config = yaml.safe_load(f)
|
|
||||||
|
|
||||||
url = config['uptime_kuma_url']
|
|
||||||
username = config['username']
|
|
||||||
password = config['password']
|
|
||||||
monitor_host = config['monitor_host']
|
|
||||||
monitor_port = config['monitor_port']
|
|
||||||
monitor_name = config['monitor_name']
|
|
||||||
|
|
||||||
api = UptimeKumaApi(url, timeout=30)
|
|
||||||
api.login(username, password)
|
|
||||||
|
|
||||||
monitors = api.get_monitors()
|
|
||||||
|
|
||||||
# Find or create "services" group
|
|
||||||
group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None)
|
|
||||||
if not group:
|
|
||||||
api.add_monitor(type='group', name='services')
|
|
||||||
monitors = api.get_monitors()
|
|
||||||
group = next((m for m in monitors if m.get('name') == 'services' and m.get('type') == 'group'), None)
|
|
||||||
|
|
||||||
# Check if monitor already exists
|
|
||||||
existing = next((m for m in monitors if m.get('name') == monitor_name), None)
|
|
||||||
|
|
||||||
# Get ntfy notification ID
|
|
||||||
notifications = api.get_notifications()
|
|
||||||
ntfy_notification_id = None
|
|
||||||
for notif in notifications:
|
|
||||||
if notif.get('type') == 'ntfy':
|
|
||||||
ntfy_notification_id = notif.get('id')
|
|
||||||
break
|
|
||||||
|
|
||||||
if existing:
|
|
||||||
print(f"Monitor '{monitor_name}' already exists (ID: {existing['id']})")
|
|
||||||
print("Skipping - monitor already configured")
|
|
||||||
else:
|
|
||||||
print(f"Creating TCP monitor '{monitor_name}'...")
|
|
||||||
api.add_monitor(
|
|
||||||
type=MonitorType.PORT,
|
|
||||||
name=monitor_name,
|
|
||||||
hostname=monitor_host,
|
|
||||||
port=monitor_port,
|
|
||||||
parent=group['id'],
|
|
||||||
interval=60,
|
|
||||||
maxretries=3,
|
|
||||||
retryInterval=60,
|
|
||||||
notificationIDList={ntfy_notification_id: True} if ntfy_notification_id else {}
|
|
||||||
)
|
|
||||||
|
|
||||||
api.disconnect()
|
|
||||||
print("SUCCESS")
|
|
||||||
|
|
||||||
except Exception as e:
|
|
||||||
print(f"ERROR: {str(e)}", file=sys.stderr)
|
|
||||||
traceback.print_exc(file=sys.stderr)
|
|
||||||
sys.exit(1)
|
|
||||||
mode: '0755'
|
|
||||||
|
|
||||||
- name: Create temporary config for TCP monitor setup
|
|
||||||
when: uptime_kuma_enabled | default(false)
|
|
||||||
delegate_to: localhost
|
|
||||||
become: no
|
|
||||||
copy:
|
|
||||||
dest: /tmp/ansible_fulcrum_ssl_config.yml
|
|
||||||
content: |
|
|
||||||
uptime_kuma_url: "{{ uptime_kuma_api_url }}"
|
|
||||||
username: "{{ uptime_kuma_username }}"
|
|
||||||
password: "{{ uptime_kuma_password }}"
|
|
||||||
monitor_host: "{{ ansible_host }}"
|
|
||||||
monitor_port: {{ fulcrum_ssl_port }}
|
|
||||||
monitor_name: "Fulcrum SSL Public"
|
|
||||||
mode: '0644'
|
|
||||||
|
|
||||||
- name: Run Uptime Kuma TCP monitor setup
|
|
||||||
when: uptime_kuma_enabled | default(false)
|
|
||||||
command: python3 /tmp/setup_fulcrum_ssl_tcp_monitor.py
|
|
||||||
delegate_to: localhost
|
|
||||||
become: no
|
|
||||||
register: tcp_monitor_setup
|
|
||||||
changed_when: "'SUCCESS' in tcp_monitor_setup.stdout"
|
|
||||||
ignore_errors: yes
|
|
||||||
|
|
||||||
- name: Display TCP monitor setup output
|
|
||||||
debug:
|
|
||||||
msg: "{{ tcp_monitor_setup.stdout_lines }}"
|
|
||||||
when: tcp_monitor_setup.stdout is defined
|
|
||||||
|
|
||||||
- name: Clean up TCP monitor temporary files
|
|
||||||
when: uptime_kuma_enabled | default(false)
|
|
||||||
delegate_to: localhost
|
|
||||||
become: no
|
|
||||||
file:
|
|
||||||
path: "{{ item }}"
|
|
||||||
state: absent
|
|
||||||
loop:
|
|
||||||
- /tmp/setup_fulcrum_ssl_tcp_monitor.py
|
|
||||||
- /tmp/ansible_fulcrum_ssl_config.yml
|
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -46,3 +46,9 @@ service_settings:
|
||||||
# from the edge host. A role default cannot serve the second play, so it
|
# from the edge host. A role default cannot serve the second play, so it
|
||||||
# lives here rather than in roles/mempool/defaults.
|
# lives here rather than in roles/mempool/defaults.
|
||||||
frontend_port: 8080
|
frontend_port: 8080
|
||||||
|
fulcrum:
|
||||||
|
# Same shape as mempool: the fulcrum role deploys on fulcrum-box, and the
|
||||||
|
# socket-proxy play publishes the SSL port from the edge host. A role default
|
||||||
|
# is invisible to that second play.
|
||||||
|
ssl_port: 50002
|
||||||
|
tailscale_hostname: fulcrum-box
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue