monerod v0.18.5.1 on monerod-box (VMID 2400), Debian 13, systemd, no Docker.
Syncing at 24% as of this commit, 0 restarts, 0 OOM kills.
── The build-from-source requirement could not be met ──────────────────────
The intent was to build it ourselves and check the result against the published
checksums. That is not achievable for this release, and the role's README says
so at length so nobody re-derives it:
contrib/guix - the reproducible build that yields a byte-identical tarball -
exists ONLY on monero master. At tag v0.18.5.1 contrib/ contains `gitian`,
whose README wants an Ubuntu 18.04 host (EOL 2023) and python-cheetah.
A plain `make release-static` is not reproducible at all, so it produces a
binary whose hash matches nothing - failing both halves of the ask.
So we install the published binary and prove its provenance instead:
1. binaryFate's key is VENDORED at files/binaryfate.asc. A trust root fetched
over the network at the moment you need it is not a trust root. Its
fingerprint is asserted after import regardless.
2. hashes.txt is verified, and the assertion is on gpg's `VALIDSIG <fpr>`
status line - NOT on the string "Good signature", which is what
bitcoin_knots greps for and which passes for any key in the keyring.
3. The hash is looked up in the output of `gpg --decrypt`, not in the raw
clearsigned file, so nothing unsigned is ever searched.
4. The lookup requires hash AND filename on one line.
Tested with a positive and a negative control before deploying: the real
tarball matches, a corrupted hash correctly fails. monero_sha256 in defaults/
is a second in-git assertion so a version bump cannot happen without the
checksum moving in the same diff.
When v0.19 ships with guix on the release branch this gets easier, and the
README records why: the published monero-linux-x64 tarball unpacks to
monero-x86_64-linux-gnu-v0.18.5.1/, i.e. it IS the guix DISTNAME renamed, so
sha256(guix output) will equal the hashes.txt entry exactly.
── The 500 GB disk is provisioned in code ──────────────────────────────────
knots-box mounts /mnt/knots_data (~875 GB of chain) entirely by hand; that
mount appears nowhere in Ansible and a rebuild of that host would come up with
the blockchain silently missing. tasks/storage.yml does not repeat it.
Whole device, no partition table. Everything after mkfs goes by UUID, because
/dev/sdX is assigned by the kernel at boot and this estate has been bitten
eight times by an identifier written down once whose thing then moved.
monero_data_device is used ONLY to create the filesystem. Guards refuse to
touch the root disk or to overwrite an unrecognised filesystem, and a final
assert confirms the data directory is not still on /.
fstab gets `nofail` so a missing disk cannot drop a headless VM into an
emergency shell - and the unit gets `RequiresMountsFor=`, without which monerod
would then start, find an empty directory, and write a 300 GB chain onto a
10 GB root disk. With it the unit refuses to start and infra/401 says so within
five minutes.
The mount-point task and the ownership task deliberately share owner and mode.
Writing the obvious root:root 0755 on the first made them disagree and every
run flipped the directory back and forth; caught by re-running --check.
── Exposure: one HTTPS vhost, zero new open ports ──────────────────────────
Two RPC servers, not one: unrestricted on 127.0.0.1:18081 for administration
and the health check, restricted on :18089 for everything else. That is
`rpc-restricted-bind-port` and deliberately NOT `restricted-rpc=1`, which would
restrict the loopback server too. Verified live: the restricted port reports
restricted=true and blanks version and start_time.
--rpc-login is HTTP Digest (contrib/epee/src/http_auth.cpp) and
core_rpc_server::init hands the same login to both servers. Caddy on vipy does
TLS and routing only - no caddy_site_basic_auth, because a Basic challenge
would collide with the Digest challenge Feather answers. Verified end to end
from the WAN: 401 unauthenticated, get_block_count over /json_rpc when
authenticated, 401 again with a wrong password.
No UFW rule, and that is correct: tailnet traffic bypasses UFW entirely because
`-A INPUT -j ts-input` sits ahead of every ufw-* chain. Confirmed on mempool-box
before relying on it here.
P2P is outbound only. Nothing forwards 18080, so the node dials out, syncs and
serves us without relaying for strangers; no-igd follows from that.
── Monitoring: infra, systemd, public. No backups ──────────────────────────
Six Gatus endpoints, all green: liveness, disk, services/monerod-box-local/
monerod, probe/monerod, dns/monero.contrapeso.xyz, public/monero.
The probe does NOT assert `synchronized`. A full sync takes days and a check
requiring it would report DOWN for every one of them - an alarm with the
batteries taken out. Height and sync state go in the message instead.
The timer carries no `Requires=monerod.service`. Fulcrum's did, and sat
active+enabled for seven months without firing once. public/monero asserts 401,
not 200, so it goes red if rpc-login ever stops being enforced.
No backups, deliberately, following the phoenixd precedent: the chain is
re-syncable public data and there is no wallet on this node.
── Known constraint ────────────────────────────────────────────────────────
The VM has 2 GB and no swap; monerod's RSS is ~1.3 GB mid-sync. It is holding
(0 OOM kills, ~1 GB available, kernel reclaiming cache) but 8 GB is the
recommendation once the initial sync is done. out-peers/in-peers are held at
16/8 partly for this reason. Bumping it is a Proxmox resize plus memory_mb
here, since the vm resource sets ignore_changes = all.
The 500 GB data disk is likewise invisible to tofu - it was attached in the UI
and the vm resource ignores it - so vms.auto.tfvars now carries a comment
saying where it is and what mounts it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three changes: detection windows tightened, a Signal transport deployed, and
alerts attached to all 84 non-transport endpoints.
── Failing faster ──────────────────────────────────────────────────────────
The heartbeat is a TICKER, not a deadline: Gatus wakes every interval and asks
"did anything arrive in the last interval", so real detection is 1-2x the
window. And a window can only ever be as tight as the push frequency - which is
why two checks moved rather than just having their numbers changed.
liveness, cpu, ups, service-health, probes 16m -> 11m
disk, zfs daily/30h -> 6-hourly/7h
backup store + pull job daily/30h -> 6-hourly/7h
DNS records 24h -> 6h
backup dump 30h -> 26h
backup-dump stays slow because the dump genuinely is daily. The store-side check
catches the same fault within 6h by reading the source's dump timestamp out of
the artefact filename, so 26h is a backstop rather than the primary signal.
── Thresholds differ by check type, deliberately ───────────────────────────
`failure-threshold` counts consecutive failures, but "consecutive" is a
different amount of wall-clock time per check: a push endpoint produces one
failure per heartbeat window, a pulled one per interval. The default of 3 would
mean 33 minutes on an 11m heartbeat and over a day on a 7h one.
More importantly the heartbeat window ALREADY encodes the tolerance - an 11m
window on a 5-minute push is exactly "one missed push forgiven" - so stacking a
threshold of 3 triples a tolerance that was already chosen. Hence:
push/heartbeat endpoints failure-threshold 1
pulled, 5m (public) failure-threshold 3 (= 15 minutes)
pulled, 6h/24h (dns, domain) failure-threshold 1
── The Signal transport ────────────────────────────────────────────────────
roles/signal_api runs signal-cli-rest-api on the observability host, pinned by
digest, MODE=native.
It publishes NO PORTS. The API has no authentication of any kind - anything that
reaches it can send messages as you and read your Signal. Gatus talks to it over
a shared docker network by service name, which is also WHY the network exists:
Gatus runs in a container, so the host's loopback is unreachable from it and a
port published on 127.0.0.1 would not have worked.
MODE=native and not json-rpc because this VPS has 464MB of RAM and already runs
Gatus and Caddy. The json-rpc modes hold a resident JVM; native runs a binary
per request, and alerts are rare enough that startup cost per alert is the right
trade.
Monitored - Gatus polls /v1/health over the same network path the alerts take,
so it proves the delivery route rather than mere container liveness. Deliberately
NOT backed up: the data directory holds Signal private keys and the recovery
path is to link the device again from the phone.
Neither the signal-api endpoint nor Gatus's self-check carries a Signal alert.
If either is down, Signal is precisely what cannot deliver the alert.
── Four traps hit while linking, all now in the role README ────────────────
* /v1/qrcodelink is BROKEN in native mode - returns "no data to encode" while
the binary itself emits a perfectly good URI. Not worked around by switching
MODE, which would put a JVM in the path of every alert permanently.
* The data dir must be owned by uid 1000, not root. A root-owned 0700 dir
cannot be traversed by the container user, so linking silently never
completes and /v1/accounts returns "Failed to read local accounts list".
* `docker exec` runs as ROOT while the service runs as uid 1000, so without
--config the account is written to /root/... on the container's ephemeral
layer. It reports success and is destroyed on the next recreate.
* The phone reporting "network error" was IPv6: chat.signal.org resolves to
dualstack AAAA records first, the container has no IPv6 address, and this
host's IPv6 path is broken - the same edge that 404'd the Go tarball. Fixed
with a mounted gai.conf that prefers IPv4.
Verified: provider loads (configuredProviders=[signal]), a test message was
delivered and confirmed received, 84 endpoints carry alerts, 86 UP / 0 DOWN.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
watchtower is being destroyed. Removed from [vps], with its host_vars, its push
token, and the six Gatus endpoints that referenced it (liveness, disk, two
systemd services, the ntfy DNS record and the ntfy HTTP check).
ntfy went with it - it ran nowhere else - so services/ntfy is deleted,
subdomains.ntfy and ntfy_topic are gone from group_vars, and the ntfy playbook
is out of site.yml. ntfy_topic already had no readers: the three infra/4xx plays
that used it were deleted when their checks were superseded.
Two things this exposed.
services/ntfy/deploy_ntfy_playbook.yml was pointing at the WRONG MACHINE. It
said `hosts: observability`, which resolves to the host `monitoring`
(64.226.70.190) - but ntfy ran on watchtower, and ntfy.contrapeso.xyz pointed
there. Running it would have installed ntfy on the new VPS. Moot now, but it is
the same stale-identity failure as the rest: the group meant watchtower when the
play was written, and nobody revisited it when the group changed. Watchtower was
in [vps] and NO role group at all, while running caddy, ntfy and Uptime Kuma -
nothing in the repo managed any of it.
More seriously: ntfy-emergency-app on vipy (avisame.contrapeso.xyz) sends its
notifications to https://ntfy.contrapeso.xyz, topic "emergencia". Destroying
watchtower breaks it, and it is an EMERGENCY notifier - it would fail silently
at exactly the moment it matters. That is NOT resolved here, deliberately:
standing ntfy up elsewhere, pointing at ntfy.sh, or retiring the app are all
decisions, not cleanups.
What this change does is make the break impossible to miss. The URL was derived
from subdomains.ntfy, so deleting that would have turned it into an undefined
variable buried in a template. It is now an explicit ntfy_service_url in the
app's own vars, still holding the old value, with the three options written
above it. The ntfy credentials stay in the vault because that app still needs
them - the vault was restored from HEAD and only watchtower's push token
removed, rather than re-handling the plaintext.
Verified: no reference to watchtower or its IP anywhere in the repo; Gatus down
from 91 to 85 endpoints, 85 UP, 0 DOWN.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Domains are now a list in group_vars/all (monitored_domains) rather than the
single root_domain hardcoded in the playbook, so adding one is a line of data
instead of a change to the task.
arbret.com resolves to prd-arbret (167.99.242.62) and its RDAP record exposes
an expiry of 2027-02-18, so the check reads real data rather than silently
passing on a missing field - verified against rdap.verisign.com before wiring
it up.
Both domains checked daily with two weeks of runway, because registration
renewal is a manual act at the registrar and losing a domain is not recoverable
the way losing a host is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four more check types, 42 endpoints, taking the estate from 39 to 81.
── systemd services (infra/401) ────────────────────────────────────────────
Every unit we deploy, checked every 5 minutes with a 16-minute heartbeat.
This closes the gap that let a real bug run unnoticed earlier today: a backup
script left forgejo, lnbits, headscale and memos stopped, and NOTHING caught
it. The dumps exited 0, the artefacts were correct, the deploy said failed=0,
and liveness only proves the HOST is up - not that anything on it serves.
One endpoint PER UNIT but only ONE timer per host: the check iterates that
host's units and pushes a result for each, the way check-backups.sh reports per
source. Four units on vipy would otherwise mean four scripts, services and
timers. A single host-level red light would also say "something on vipy is
down" without saying which, which is not the question anyone has.
Keys are host-qualified because unit names collide - caddy runs on four
machines. Which units a host runs lives in host_vars/<host>/monitored_services,
because "what runs here" is a property of the machine, the same reasoning as
the cross-host ports.
nut-driver-enumerator is deliberately excluded: it is a oneshot generator that
is `enabled` but always `inactive`, so it would report down forever. Checked
live before excluding it.
── domain, DNS and public endpoints (infra/402) ────────────────────────────
The first checks that PULL rather than push, and that is the right way round:
all three are about how the outside world sees us, so they must be measured
from outside. Nothing is installed anywhere - no script, no timer, no token.
They also have no heartbeat, because a heartbeat answers "did the reporter
report"; when Gatus does the checking itself, failure is immediate.
domain 1 endpoint, 24h, [DOMAIN_EXPIRATION] > 336h (two weeks)
dns 11 endpoints, 24h, [DNS_RCODE] == NOERROR and [BODY] == the IP
public 14 endpoints, 5m, 11 HTTPS + 3 TCP
Expected A records are derived from inventory (hostvars[host].ansible_host),
not written down again. The estate's recurring bug is an address recorded in a
second place and left behind when the machine moved; asserting against
inventory means a renumbered box is one edit, not two.
Expected HTTP status was checked live per site rather than assumed. Two return
401 - the Gatus dashboard and the DATUM dashboard, both behind basic auth - and
that is what is asserted: expecting 200 there would go green precisely when the
auth broke. headscale asserts /health rather than /, which is a 404 by design.
Every HTTPS check also carries [CERTIFICATE_EXPIRATION] > 168h, which is free
on an endpoint already being polled and catches a renewal that silently stops.
Two things learned the hard way, both now in comments:
* A domain-expiry endpoint needs a URL SCHEME. Gatus derives the endpoint type
from the prefix (endpoint.Type()), so a bare "contrapeso.xyz" is UNKNOWN and
the whole config is rejected. It is https:// plus a DOMAIN_EXPIRATION
condition and no status assertion, so the registrar's parking page at the
apex is irrelevant. Upstream also enforces a 5m minimum interval for that
placeholder, because it uses a free whois service.
* That rejection proved skip-invalid-config-update was worth adding. Gatus
logged "the configuration file was updated, but it is not valid, the old
configuration will continue being used" and kept running. Without it the
reload path calls panic() and one malformed contributed file takes the
monitor down.
Verified: 15/15 service endpoints UP; 26/27 public-facing UP. The single DOWN is
public_memos, correctly - memos-box is powered off, and the condition result
reads [STATUS] (502) == 200. memos-box and arbret-staging-box were shut down
deliberately from the Proxmox UI to test the liveness endpoints; their endpoints
stay registered and will go green when the VMs return.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
