navidrome: a music server on its own box, with beets in front of it

Navidrome 0.64.0 on music-box (nodito VM 8200, 1 core / 1 GB / 50 GB), public
at music.contrapeso.xyz. A single Go binary with a SQLite database; the only
external dependency is ffmpeg for transcoding. Plus beets, the CLI that fixes
tags before Navidrome ever indexes them.

── Exposed without basic auth, on purpose ─────────────────────────────────
Navidrome speaks two protocols on one port: its web UI, and the Subsonic API
(/rest/*) that every mobile client actually uses. A Subsonic client
authenticates with ?u=&t=md5(password+salt)&s= in the query string and has
nowhere to carry a second credential, so Basic Auth in front of the site is
invisible to most of them. The choice is a proxy credential or a working phone.

Same call this estate already made for monerod's restricted RPC, for the same
reason: proxied bare so Feather can reach it, with the service's own auth doing
the authenticating.

The cost is stated rather than glossed: the login page and the whole /rest/*
surface are reachable by anyone who finds the hostname, and Navidrome has no
rate limiting and no 2FA.

A fail2ban jail on vipy reads a per-vhost Caddy access log and catches WEB UI
brute force, where a bad login returns 401. It very likely does NOT catch
Subsonic brute force, because that protocol answers a bad password with HTTP
200 and puts the failure in the body - which Caddy never sees. Nor can it be
fixed on the music box: the request arrives from vipy over the tailnet, so the
only address fail2ban could ban there is vipy's, which would take down every
service behind the edge. Banning has to happen at the edge; the edge cannot see
the failure. So /rest/* is protected by password strength alone, and the README
carries a one-line curl to re-check that after every upgrade.

── Passwords are stored reversibly, and cannot not be ─────────────────────
The Subsonic API requires the server to compute md5(password+salt) per request,
so Navidrome cannot hash passwords one-way. It encrypts them at rest with
PasswordEncryptionKey instead. Two real consequences, both in the README: root
on that box means every Navidrome password in plaintext, so the password must
be unique to this service; and changing the key invalidates every stored
password, so it is set before the first account and left alone.

── beets is not a service, deliberately ───────────────────────────────────
`beet import` is interactive and it MOVES AND RENAMES FILES. A timer that
reorganises a music library unattended is a way to lose one. Installed and
configured, run by hand.

/srv/music is counterweight:navidrome mode 2750 and the setgid bit is
load-bearing: without it beets' files land in the curator's own group,
Navidrome cannot read them, and the symptom is an EMPTY LIBRARY rather than a
permission error. ReplayGain and lyrics are auto: no - on one core they turn a
two-minute import into a twenty-minute one.

── Monitoring ─────────────────────────────────────────────────────────────
Host liveness and disk (400), the systemd unit (401), a probe that fetches
/app/ and greps it (403), and public reachability plus cert expiry (402).

/app/ was chosen by testing all three candidates against 0.64.0: /ping returns
a bare "." and proves only that something listens; /rest/ping.view identifies
the server but logs two warnings per call, ~576 a day; /app/ returns
<title>Navidrome</title> and logs nothing. 402 asserts 200 on /app/ and not on
"/", because "/" answers 302 - checked live rather than assumed.

── Also ───────────────────────────────────────────────────────────────────
The headscale ACL grants tag:prd-vipy exactly tag:music-box:4533. Without a
rule the node is invisible rather than unreachable, which surfaces as a Caddy
502 "no such host".

inventory lan_ip was read off nodito's ARP table for the VM's own MAC, not
guessed - and .149 had belonged to another machine, whose stale host key made
SSH refuse the connection. That is the designed behaviour, not a fault.

50 GB is the LIBRARY ceiling, not just the OS: /srv/music is on the root disk,
so roughly 300-450 albums in FLAC. 1 GB is fine for serving and tight for the
initial scan, which is the memory peak.

No backups, deliberately. The music files would swamp the age-encrypted nightly
path, and navidrome.db is a rebuildable index. What is NOT rebuildable is
playlists, play counts, ratings and accounts - there are none today, which is
what makes the trade fair, and the README names this as the decision to revisit
if that changes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ansible/roles/navidrome/templates/healthcheck.sh.j2

@@ -0,0 +1,65 @@
+#!/bin/bash
+# Navidrome health check — managed by Ansible (roles/navidrome)
+#
+# The exit code is the answer and systemd keeps it:
+#     systemctl is-failed {{ navidrome_healthcheck_service_name }}.service
+# Reporting anywhere else is optional and generic.
+#
+# /app/ rather than /ping, established by testing against 0.64.0 on the box:
+#
+#   /ping            200, Content-Type text/plain, body is a single "."  - it
+#                    proves something is listening and nothing more.
+#   /rest/ping.view  identifies itself ("type":"navidrome") but logs TWO
+#                    warning lines per call. At one call per 5 minutes that is
+#                    ~576 spurious warnings a day, which is how you train
+#                    yourself to ignore the warning level.
+#   /app/            200, 2.8 KB, <title>Navidrome</title>, and SILENT.
+#
+# So /app/ is the only one that proves it is really Navidrome serving its real
+# UI without polluting the journal. Same reasoning as the xmrblocks check,
+# which fetches a real page because a missing templates dir serves 200s of
+# nothing.
+
+URL="${NAVIDROME_URL:-http://127.0.0.1:{{ navidrome_port }}}"
+PUSH_URL="${HEALTHCHECK_PUSH_URL:-}"
+PUSH_TOKEN="${HEALTHCHECK_PUSH_TOKEN:-}"
+
+report() {
+    local status=$1
+    local msg=$2
+
+    # No push URL is normal, not an error: the exit code below is still a
+    # complete answer for anything reading unit state.
+    [ -n "$PUSH_URL" ] || return 0
+
+    local encoded_msg="${msg// /%20}"
+    local _ok=false
+    [ "${status}" = "up" ] && _ok=true
+    curl -s --max-time 15 --retry 2 -o /dev/null -X POST \
+        -H "Authorization: Bearer ${PUSH_TOKEN}" \
+        "${PUSH_URL}?success=${_ok}&error=${encoded_msg}" || true
+}
+
+# 30s: Navidrome answers /ping in milliseconds normally, but this box has one
+# core and a library scan will happily eat it. A tight timeout here would
+# manufacture false DOWNs every time a scan runs.
+body=$(curl -s --max-time 30 -w '\n%{http_code}' "${URL}/app/" 2>/dev/null)
+code=$(printf '%s' "$body" | tail -1)
+
+if [ -z "$code" ] || [ "$code" = "000" ]; then
+    report "down" "navidrome not responding"
+    exit 1
+fi
+
+if [ "$code" != "200" ]; then
+    report "down" "navidrome HTTP ${code}"
+    exit 1
+fi
+
+if ! printf '%s' "$body" | grep -qi "navidrome"; then
+    report "down" "navidrome served 200 but not its own UI (something else on the port?)"
+    exit 1
+fi
+
+report "up" "OK"
+exit 0