xmrblocks: the Monero explorer, built off-box because 2 GB cannot compile it
The Onion Monero Blockchain Explorer on monerod-box, published at
exmrplore.contrapeso.xyz. Ten Gatus endpoints green across the node and the
explorer.
── It needs a compiled monero source tree, not a running monerod ───────────
xmrblocks is not an RPC client. cmake/FindMonero.cmake imports 21 STATIC
libraries out of a monero build tree and includes its headers, because it opens
the LMDB directly. The release tarball we install for the daemon ships only
executables - no .a files, no headers - so there is nothing on the box to link
against, and building it means compiling monero first.
Measured on Debian 13, building only the libraries it links:
peak RSS, one cc1plus ~3.2 GB
peak RSS at -j4 ~5.5 GB
wall time at -j4 167 s
source + build tree 195 MB
toolchain 1.6 GB
That rules the box out at ANY parallelism: 768 MiB and 1800 MiB were both
OOM-killed at -j1 and -j2, because a single translation unit over 3 GB is not
something lower -j works around. My first estimate of "~10 GB of disk" was
wrong by two orders of magnitude - that figure describes upstream's full `make`,
which also builds monerod, both wallets and the test suite, none of it linked
here. Disk was never the constraint.
So scripts/build-xmrblocks.sh does the build in a debian:13 container. An
artefact only one machine can produce is one nobody can audit; this way any host
with Docker and ~6 GB reproduces it. It verifies the monero source against
binaryFate's signature exactly as roles/monero does for the daemon, and the role
asserts the resulting sha256 ON THE CONTROL NODE before copying a single byte.
── Three upstream problems, none documented anywhere ───────────────────────
Found by building, not by reading:
1. libasio-dev is absent from upstream's dependency list. That list is
Ubuntu-based; Crow wants standalone asio.hpp, not boost/asio.hpp, and on
Debian 13 the build dies outright.
2. The explorer does not compile against monero v0.18.5.1. v0.18.5.0 changed
get_pruned_transaction_hash from returning the hash to returning bool with
an out-param, and src/page.h still calls the old form. Upstream's
Dockerfile pins v0.18.4.0, which is why nobody has hit it. Patched from an
exact-line anchor, so if upstream fixes it the build STOPS rather than
quietly doing something else. We patch instead of building against 0.18.4.0
because the signed hashes.txt only covers the current release - using the
older source would mean dropping the provenance chain. LMDB is VERSION 5 in
both, so either reads the other's database.
3. FindMonero.cmake hunts for miniupnpc inside the monero build tree with
NO_DEFAULT_PATH, but monero stopped vendoring miniupnp as a submodule.
Fixed with a cache override rather than patching their cmake.
USE_DEVICE_TREZOR=OFF as well - an explorer has no use for hardware wallets and
it drops the generated protobuf units. Verified that xmrblocks still links.
── Two things in the unit that look wrong and are not ──────────────────────
WorkingDirectory is load-bearing: page.h hardcodes TMPL_DIR as "./templates",
resolved relative to cwd at REQUEST time. Get it wrong and the service starts,
binds 8081, and 500s every page. That is also why the health check fetches a
real page and greps it instead of probing the port.
ReadWritePaths must include the blockchain directory even though the database is
opened read-only: monero sets mdb_flags = MDB_RDONLY WITHOUT MDB_NOLOCK, so
LMDB still registers this reader in lock.mdb. With ProtectSystem=strict and
nothing writable the open fails. "It only reads, so grant it nothing" is wrong.
── Timeouts are measured, not guessed ──────────────────────────────────────
proxmox-tank-1 is a spinning-disk mirror and xmrblocks says so on startup.
First start took 98s to open the LMDB before binding (wait_for now allows 300s),
and a COLD index page took 68 SECONDS where the next two took 0.87s and 0.012s
(the check now allows 90s, its unit 180s). The first pass used 90s and 25s and
produced exactly the false failures you would expect.
A stat() bug was caught the same way: `stat` computes a sha1 by default, so
checking that data.mdb exists was hashing the entire 40 GB blockchain. --check
found it before a real run did.
── Exposure ────────────────────────────────────────────────────────────────
Caddy on vipy terminates TLS; the upstream is reached over the tailnet, which
needed a headscale ACL entry for tag:monerod-box:8081 - without it the node is
invisible, not merely unreachable, and the symptom is a 502 "no such host".
No authentication, deliberately: it is a block explorer over public chain data.
That includes --enable-pusher, so a visitor can relay a raw transaction through
our node. A choice, not an oversight; flip xmrblocks_enable_pusher to close it.
── Also ────────────────────────────────────────────────────────────────────
monerod-box raised 2048 -> 6144 MiB. Not for the explorer: monerod alone had
outgrown 2 GB past ~39% of the chain and was OOM-killed twice in a morning at
1.74 GB anon-rss. Recorded in vms.auto.tfvars because ignore_changes = all means
tofu will not reconcile it. The box now idles at ~1.2 GB of 5.9 GB with both
services up.
No backups, deliberately - binary, templates and chain are all reproducible.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
3c50cab58d
commit
e791ffb183
GPG key ID:
883EDBAA726BD96C
20 changed files with 897 additions and 1 deletions
180
scripts/build-xmrblocks.sh
Executable file
180
scripts/build-xmrblocks.sh
Executable file
|
|
@ -0,0 +1,180 @@
|
|||
#!/usr/bin/env bash
|
||||
#
|
||||
# Build xmrblocks (the Onion Monero Blockchain Explorer) reproducibly.
|
||||
#
|
||||
# WHY THIS SCRIPT EXISTS
|
||||
# ----------------------
|
||||
# xmrblocks is not a client of monerod. It links 21 static libraries out of a
|
||||
# compiled monero source tree and opens the LMDB directly, so building it means
|
||||
# first compiling monero itself. Measured on Debian 13: a single cc1plus peaks
|
||||
# at ~3.1 GB and a -j4 build peaks at ~5.5 GB. monerod-box has 2 GB, so it
|
||||
# cannot build this at any level of parallelism - the binary has to be built
|
||||
# elsewhere and shipped.
|
||||
#
|
||||
# Shipping a binary that only one machine can produce is how estates end up
|
||||
# with artefacts nobody can rebuild. This script is the answer to that: it runs
|
||||
# the whole build inside a debian:13 container, so any machine with Docker and
|
||||
# ~6 GB of RAM produces the same binary. Nothing is installed on the host.
|
||||
#
|
||||
# WHAT IT GUARANTEES
|
||||
# - the monero source is the GPG-signed release tarball, verified against
|
||||
# binaryFate's key (the same chain roles/monero uses for the daemon)
|
||||
# - the explorer is pinned to an exact commit
|
||||
# - the one upstream patch is applied from an anchor, and the script FAILS
|
||||
# if the anchor is missing rather than silently skipping it
|
||||
#
|
||||
# WHAT IT IS NOT
|
||||
# Not bit-for-bit reproducible. Debian's toolchain, timestamps and build
|
||||
# paths all leak in. It is repeatable, not deterministic; the MANIFEST
|
||||
# records what went in so two builds can be compared.
|
||||
#
|
||||
# Usage: scripts/build-xmrblocks.sh [output-dir]
|
||||
# Needs: docker, ~6 GB RAM available to it, ~2 GB disk
|
||||
set -euo pipefail
|
||||
|
||||
# --- Pins. Change these together and rebuild. --------------------------------
|
||||
MONERO_VERSION="0.18.5.1"
|
||||
MONERO_SRC_TARBALL="monero-source-v${MONERO_VERSION}.tar.bz2"
|
||||
MONERO_SRC_URL="https://downloads.getmonero.org/cli/${MONERO_SRC_TARBALL}"
|
||||
# From the GPG-signed hashes.txt. Verified by the container, not trusted from here.
|
||||
MONERO_SRC_SHA256="9ec6ed0fd37db9d81cf7738a5f0536cf9aec6ed8ef8fd48649a59a6aaf20de3d"
|
||||
BINARYFATE_FPR="81AC591FE9C4B65C5806AFC3F0AF4D462A0BDF92"
|
||||
|
||||
EXPLORER_REPO="https://github.com/moneroexamples/onion-monero-blockchain-explorer.git"
|
||||
EXPLORER_COMMIT="29593b709edeb23daba035009ccd134a5862543a"
|
||||
|
||||
BUILD_IMAGE="debian:13" # must match the TARGET host's Debian release
|
||||
JOBS="${JOBS:-4}"
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
SIGNING_KEY="${REPO_ROOT}/ansible/roles/monero/files/binaryfate.asc"
|
||||
OUT_DIR="${1:-$HOME/personal_infra_artifacts/xmrblocks}"
|
||||
|
||||
command -v docker >/dev/null || { echo "docker is required" >&2; exit 1; }
|
||||
[ -f "$SIGNING_KEY" ] || { echo "missing signing key: $SIGNING_KEY" >&2; exit 1; }
|
||||
|
||||
WORK="$(mktemp -d)"
|
||||
trap 'rm -rf "$WORK"' EXIT
|
||||
cp "$SIGNING_KEY" "$WORK/binaryfate.asc"
|
||||
mkdir -p "$OUT_DIR"
|
||||
|
||||
echo "==> building xmrblocks"
|
||||
echo " monero ${MONERO_VERSION} (signed source tarball)"
|
||||
echo " explorer ${EXPLORER_COMMIT}"
|
||||
echo " image ${BUILD_IMAGE}, -j${JOBS}"
|
||||
echo " output ${OUT_DIR}"
|
||||
|
||||
cat > "$WORK/build.sh" <<INNER
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
apt-get update -qq
|
||||
# libasio-dev is NOT in upstream's dependency list. That list is Ubuntu-based
|
||||
# and omits it; on Debian 13 Crow fails with "asio.hpp: No such file".
|
||||
# libminiupnpc-dev is needed because monero stopped vendoring miniupnp as a
|
||||
# submodule while the explorer's FindMonero.cmake still expects it in-tree.
|
||||
apt-get install -y -qq --no-install-recommends \\
|
||||
build-essential cmake pkg-config git ca-certificates curl bzip2 gnupg \\
|
||||
libssl-dev libzmq3-dev libunbound-dev libsodium-dev libunwind-dev \\
|
||||
libreadline-dev libhidapi-dev libusb-1.0-0-dev libprotobuf-dev \\
|
||||
protobuf-compiler libboost-all-dev libcurl4-openssl-dev \\
|
||||
libasio-dev libminiupnpc-dev >/dev/null
|
||||
|
||||
cd /build
|
||||
|
||||
echo "--> verifying the monero source tarball"
|
||||
export GNUPGHOME=/build/gnupg; mkdir -p \$GNUPGHOME; chmod 700 \$GNUPGHOME
|
||||
gpg --batch --import /build/binaryfate.asc 2>/dev/null
|
||||
gpg --batch --with-colons --fingerprint | awk -F: '/^fpr:/{print \$10}' \\
|
||||
| grep -qx "${BINARYFATE_FPR}" || { echo "signing key is not binaryFate's"; exit 1; }
|
||||
curl -sSL -o hashes.txt https://www.getmonero.org/downloads/hashes.txt
|
||||
gpg --batch --status-fd 2 --output hashes-verified.txt --decrypt hashes.txt 2>gpg.err
|
||||
grep -q "VALIDSIG.*${BINARYFATE_FPR}" gpg.err || { echo "hashes.txt did not verify"; exit 1; }
|
||||
curl -sSL -o "${MONERO_SRC_TARBALL}" "${MONERO_SRC_URL}"
|
||||
ACTUAL=\$(sha256sum "${MONERO_SRC_TARBALL}" | cut -d' ' -f1)
|
||||
[ "\$ACTUAL" = "${MONERO_SRC_SHA256}" ] || { echo "tarball sha mismatch"; exit 1; }
|
||||
grep -Fxq "\$ACTUAL ${MONERO_SRC_TARBALL}" hashes-verified.txt \\
|
||||
|| { echo "tarball is not the one binaryFate signed"; exit 1; }
|
||||
echo " source verified against the signed hashes"
|
||||
|
||||
tar xjf "${MONERO_SRC_TARBALL}"
|
||||
SRC=/build/monero-source-v${MONERO_VERSION}
|
||||
|
||||
echo "--> building the monero libraries xmrblocks links against"
|
||||
# Only the libraries. Upstream's Dockerfile runs a full \`make\`, which also
|
||||
# builds monerod, both wallets and the test suite - none of which are linked
|
||||
# into xmrblocks. USE_DEVICE_TREZOR=OFF drops the generated protobuf units too;
|
||||
# an explorer has no use for hardware wallets.
|
||||
mkdir -p \$SRC/build/release && cd \$SRC/build/release
|
||||
cmake -D CMAKE_BUILD_TYPE=Release -D BUILD_TESTS=OFF -D USE_DEVICE_TREZOR=OFF ../.. >/dev/null
|
||||
cmake --build . -j${JOBS} --target \\
|
||||
common blocks cryptonote_basic cryptonote_core multisig cryptonote_protocol \\
|
||||
daemonizer mnemonics epee lmdb device wallet-crypto blockchain_db ringct \\
|
||||
wallet cncrypto easylogging version checkpoints randomx hardforks >/dev/null
|
||||
|
||||
echo "--> fetching the explorer at ${EXPLORER_COMMIT}"
|
||||
cd /build
|
||||
git clone -q "${EXPLORER_REPO}" explorer
|
||||
cd explorer && git checkout -q "${EXPLORER_COMMIT}"
|
||||
|
||||
# monero v0.18.5.0 changed get_pruned_transaction_hash from returning the hash
|
||||
# to returning bool with an out-param. The explorer still calls the old form,
|
||||
# so it does not compile against the daemon we actually run. Anchored on the
|
||||
# exact line: if upstream fixes this, the anchor vanishes and we stop loudly
|
||||
# rather than building something unexpected.
|
||||
OLD=' txd.hash = get_pruned_transaction_hash(tx, tx.prunable_hash);'
|
||||
NEW=' get_pruned_transaction_hash(tx, tx.prunable_hash, txd.hash);'
|
||||
if grep -qF "\$OLD" src/page.h; then
|
||||
sed -i "s|\$OLD|\$NEW|" src/page.h
|
||||
echo " applied get_pruned_transaction_hash patch"
|
||||
elif grep -qF "\$NEW" src/page.h; then
|
||||
echo " upstream already uses the new signature - patch not needed"
|
||||
else
|
||||
echo "PATCH ANCHOR NOT FOUND in src/page.h - upstream changed; review before building" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "--> building xmrblocks"
|
||||
mkdir -p build && cd build
|
||||
# Xmr_MINIUPNPC_LIBRARY: FindMonero.cmake searches for miniupnpc *inside* the
|
||||
# monero build tree with NO_DEFAULT_PATH, but monero no longer vendors it.
|
||||
# Point it at the system library instead of patching their cmake.
|
||||
cmake -D MONERO_DIR=\$SRC \\
|
||||
-D Xmr_MINIUPNPC_LIBRARY=/usr/lib/x86_64-linux-gnu/libminiupnpc.a .. >/dev/null
|
||||
cmake --build . -j${JOBS} >/dev/null
|
||||
[ -x xmrblocks ] || { echo "no xmrblocks produced"; exit 1; }
|
||||
./xmrblocks --help >/dev/null 2>&1 || true
|
||||
|
||||
echo "--> collecting"
|
||||
install -m 0755 xmrblocks /out/xmrblocks
|
||||
rm -rf /out/templates && cp -r templates /out/templates
|
||||
chmod -R a+rX /out/templates
|
||||
|
||||
{
|
||||
echo "artifact: xmrblocks"
|
||||
echo "built: \$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||
echo "build-image: ${BUILD_IMAGE}"
|
||||
echo "monero-version: ${MONERO_VERSION}"
|
||||
echo "monero-src-sha: ${MONERO_SRC_SHA256}"
|
||||
echo "explorer-commit: ${EXPLORER_COMMIT}"
|
||||
echo "cmake-flags: USE_DEVICE_TREZOR=OFF BUILD_TESTS=OFF CMAKE_BUILD_TYPE=Release"
|
||||
echo "patches: get_pruned_transaction_hash -> out-param form (monero >= 0.18.5.0)"
|
||||
echo "xmrblocks-sha256: \$(sha256sum /out/xmrblocks | cut -d' ' -f1)"
|
||||
echo "templates-sha256: \$(find /out/templates -type f | sort | xargs sha256sum | sha256sum | cut -d' ' -f1)"
|
||||
echo "needed-libs:"
|
||||
objdump -p /out/xmrblocks 2>/dev/null | awk '/NEEDED/{print " " \$2}' | sort
|
||||
} > /out/MANIFEST
|
||||
INNER
|
||||
chmod +x "$WORK/build.sh"
|
||||
|
||||
docker run --rm \
|
||||
-v "$WORK":/build \
|
||||
-v "$OUT_DIR":/out \
|
||||
"$BUILD_IMAGE" bash /build/build.sh
|
||||
|
||||
echo
|
||||
echo "==> done"
|
||||
sed -n '1,12p' "$OUT_DIR/MANIFEST"
|
||||
echo
|
||||
echo " pin xmrblocks_sha256 in ansible/roles/xmrblocks/defaults/main.yml to:"
|
||||
grep '^xmrblocks-sha256:' "$OUT_DIR/MANIFEST" | awk '{print " " $2}'
|
||||
Loading…
Add table
Add a link
Reference in a new issue