navidrome: a music server on its own box, with beets in front of it

Navidrome 0.64.0 on music-box (nodito VM 8200, 1 core / 1 GB / 50 GB), public
at music.contrapeso.xyz. A single Go binary with a SQLite database; the only
external dependency is ffmpeg for transcoding. Plus beets, the CLI that fixes
tags before Navidrome ever indexes them.

── Exposed without basic auth, on purpose ─────────────────────────────────
Navidrome speaks two protocols on one port: its web UI, and the Subsonic API
(/rest/*) that every mobile client actually uses. A Subsonic client
authenticates with ?u=&t=md5(password+salt)&s= in the query string and has
nowhere to carry a second credential, so Basic Auth in front of the site is
invisible to most of them. The choice is a proxy credential or a working phone.

Same call this estate already made for monerod's restricted RPC, for the same
reason: proxied bare so Feather can reach it, with the service's own auth doing
the authenticating.

The cost is stated rather than glossed: the login page and the whole /rest/*
surface are reachable by anyone who finds the hostname, and Navidrome has no
rate limiting and no 2FA.

A fail2ban jail on vipy reads a per-vhost Caddy access log and catches WEB UI
brute force, where a bad login returns 401. It very likely does NOT catch
Subsonic brute force, because that protocol answers a bad password with HTTP
200 and puts the failure in the body - which Caddy never sees. Nor can it be
fixed on the music box: the request arrives from vipy over the tailnet, so the
only address fail2ban could ban there is vipy's, which would take down every
service behind the edge. Banning has to happen at the edge; the edge cannot see
the failure. So /rest/* is protected by password strength alone, and the README
carries a one-line curl to re-check that after every upgrade.

── Passwords are stored reversibly, and cannot not be ─────────────────────
The Subsonic API requires the server to compute md5(password+salt) per request,
so Navidrome cannot hash passwords one-way. It encrypts them at rest with
PasswordEncryptionKey instead. Two real consequences, both in the README: root
on that box means every Navidrome password in plaintext, so the password must
be unique to this service; and changing the key invalidates every stored
password, so it is set before the first account and left alone.

── beets is not a service, deliberately ───────────────────────────────────
`beet import` is interactive and it MOVES AND RENAMES FILES. A timer that
reorganises a music library unattended is a way to lose one. Installed and
configured, run by hand.

/srv/music is counterweight:navidrome mode 2750 and the setgid bit is
load-bearing: without it beets' files land in the curator's own group,
Navidrome cannot read them, and the symptom is an EMPTY LIBRARY rather than a
permission error. ReplayGain and lyrics are auto: no - on one core they turn a
two-minute import into a twenty-minute one.

── Monitoring ─────────────────────────────────────────────────────────────
Host liveness and disk (400), the systemd unit (401), a probe that fetches
/app/ and greps it (403), and public reachability plus cert expiry (402).

/app/ was chosen by testing all three candidates against 0.64.0: /ping returns
a bare "." and proves only that something listens; /rest/ping.view identifies
the server but logs two warnings per call, ~576 a day; /app/ returns
<title>Navidrome</title> and logs nothing. 402 asserts 200 on /app/ and not on
"/", because "/" answers 302 - checked live rather than assumed.

── Also ───────────────────────────────────────────────────────────────────
The headscale ACL grants tag:prd-vipy exactly tag:music-box:4533. Without a
rule the node is invisible rather than unreachable, which surfaces as a Caddy
502 "no such host".

inventory lan_ip was read off nodito's ARP table for the VM's own MAC, not
guessed - and .149 had belonged to another machine, whose stale host key made
SSH refuse the connection. That is the designed behaviour, not a fault.

50 GB is the LIBRARY ceiling, not just the OS: /srv/music is on the root disk,
so roughly 300-450 albums in FLAC. 1 GB is fine for serving and tight for the
initial scan, which is the memory peak.

No backups, deliberately. The music files would swamp the age-encrypted nightly
path, and navidrome.db is a rebuildable index. What is NOT rebuildable is
playlists, play counts, ratings and accounts - there are none today, which is
what makes the trade fair, and the README names this as the decision to revisit
if that changes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ansible/roles/navidrome/README.md

@@ -0,0 +1,179 @@
+# navidrome
+
+Navidrome — a music server — plus [beets](https://beets.io), the CLI that fixes
+tags before Navidrome ever indexes them. Both live on `music-box`.
+
+Navidrome is a single Go binary with a SQLite database and no external
+dependencies beyond `ffmpeg` for transcoding. It speaks two protocols on one
+port: its own web UI, and the **Subsonic API** (`/rest/*`), which is what every
+mobile client in the ecosystem — Symfonium, Amperfy, Tempo, substreamer —
+actually talks to.
+
+| | |
+|---|---|
+| Host | `music-box` (nodito VM 8200, 1 core / 1 GB / 50 GB) |
+| Port | 4533 |
+| Public at | `music.contrapeso.xyz`, proxied by Caddy on `vipy` |
+| Library | `/srv/music` — beets-managed, Navidrome reads only |
+| Staging | `/srv/incoming` — unsorted, Navidrome never looks here |
+| State | `/var/lib/navidrome` — SQLite, cache, transcode scratch |
+| Backups | none, deliberately — see below |
+
+## Exposure
+
+Public, authenticated by **Navidrome's own login**. There is no `basic_auth` in
+front of it, and that is a decision rather than an omission.
+
+The reason is the Subsonic API. A Subsonic client authenticates with
+`?u=<user>&t=<md5(password+salt)>&s=<salt>` in the query string and has nowhere
+to put a second set of credentials. HTTP Basic Auth in front of the whole site
+is invisible to most of them, so the choice is between a proxy credential and a
+working phone. This estate has made the same call before, for the same reason:
+`monerod`'s restricted RPC is proxied bare so Feather can reach it, and its own
+Digest auth does the authenticating.
+
+The cost is stated plainly: Navidrome's login page and the whole `/rest/*`
+surface are reachable by anyone who finds the hostname. Navidrome has **no rate
+limiting and no 2FA**. The fail2ban jail below covers part of that gap.
+
+### The fail2ban jail on `vipy`, and what it does not cover
+
+Installed by the edge play in `services/navidrome/deploy_navidrome_playbook.yml`,
+reading a per-vhost Caddy access log at `/var/log/caddy/music.log`.
+
+It catches **web UI brute force**: a bad login on `/auth/login` returns HTTP
+401, which lands in the access log and trips the filter.
+
+It very likely does **not** catch Subsonic API brute force, because the
+Subsonic protocol answers a bad password with **HTTP 200** and puts the failure
+in the body (`<subsonic-response status="failed"><error code="40" .../>`).
+Caddy never sees response bodies, so a status-code filter is blind to it.
+
+The obvious fix does not work either. Navidrome's own journal logs those
+failures, but the request arrives from `vipy` over the tailnet, so the only
+address fail2ban could ban on this box is `vipy`'s — which would take down every
+service behind the edge. Banning has to happen at the edge; the edge cannot see
+the failure.
+
+So: `/rest/*` is protected by password strength alone. That is the reason the
+Navidrome password must be long, random, and used nowhere else.
+
+To re-check whether this is still true after a Navidrome upgrade:
+
+```bash
+curl -s -o /dev/null -w '%{http_code}\n' \
+  'https://music.contrapeso.xyz/rest/ping.view?u=x&p=wrong&v=1.16.1&c=test'
+```
+
+`200` means the caveat holds. `401` means it no longer does — widen the jail
+and delete this section.
+
+## Passwords are stored reversibly
+
+Navidrome **cannot** hash user passwords one-way. The Subsonic API requires the
+server to compute `md5(password + salt)` on every request, so it must be able to
+recover the plaintext. It encrypts passwords at rest with
+`PasswordEncryptionKey` instead (`navidrome_password_encryption_key`, from
+`group_vars/all/vault.yml`).
+
+Two consequences, both real:
+
+- Root on this box means every Navidrome password in plaintext. **Use a password
+  unique to this service.**
+- **Changing the key invalidates every stored password.** Set it before the first
+  account exists and leave it alone.
+
+## Backups: none, deliberately
+
+Recorded as a decision so it does not read as an oversight — the same treatment
+`phoenixd`, `monero` and `xmrblocks` get.
+
+The music files are large and would swamp the age-encrypted nightly path that
+`backup_source` uses. `navidrome.db` is mostly a rebuildable index: delete it and
+a rescan reconstructs the library.
+
+What is **not** rebuildable is the curation on top — playlists, play counts,
+ratings, starred items, user accounts. Today there is none of it, which is why
+this is a fair trade. **If that changes, this is the decision to revisit**, and
+the fix is small: `navidrome.db` is a single SQLite file and would fit the
+`backup_source` pattern in about fifteen lines.
+
+The beets database (`/var/lib/beets/library.db`) is in the same position — it
+can be rebuilt from the tags beets itself wrote into the files.
+
+## beets
+
+Installed and configured by this role; **not** a service, and deliberately so.
+`beet import` is interactive and it *moves and renames files*. A timer that
+reorganises a music library unattended is a way to lose one.
+
+Config at `~counterweight/.config/beets/config.yaml`, library DB in
+`/var/lib/beets`.
+
+```bash
+# the usual loop
+rsync -avP ~/Music/newstuff/ music-box:/srv/incoming/
+ssh music-box
+beet import /srv/incoming
+```
+
+Navidrome picks up the result on its next scan.
+
+The single most important line in that config is
+`paths.default: $albumartist/...`. Organising by `$artist` instead scatters
+every compilation and every "feat." track into a phantom artist, which is how a
+library ends up unusable and Navidrome gets blamed for it.
+
+ReplayGain and lyrics are set `auto: no` — on one core they turn a two-minute
+import into a twenty-minute one. Run `beet replaygain` and `beet lyrics`
+afterwards, separately.
+
+### Ownership, which is the fiddly part
+
+`/srv/music` is `counterweight:navidrome` mode **2750**. The setgid bit is
+load-bearing: without it, files beets creates land in the curator's own group,
+Navidrome cannot read them, and the symptom is an **empty library rather than a
+permission error**.
+
+## Monitoring
+
+| Layer | What it answers | Where |
+|---|---|---|
+| `infra/400` | is the host alive, is the disk filling | automatic, needs `gatus_push_tokens.music_box_local` in the vault |
+| `infra/401` | is the `navidrome` unit active | `monitored_services` in `host_vars/music_box_local/main.yml` |
+| `infra/403` + this role | does Navidrome actually serve its own UI (`/app/`) | `probe_navidrome` |
+| `infra/402` | does `music.contrapeso.xyz` answer from outside, is the cert valid | `public_sites` |
+
+The role's own check fetches `/app/` and greps for `navidrome`. A unit can be
+perfectly `active` while serving nothing useful.
+
+`/app/` was chosen after testing all three candidates against 0.64.0:
+`/ping` returns a bare `.` (proves only that something listens);
+`/rest/ping.view` identifies the server but logs two warnings per call,
+~576 a day; `/app/` returns `<title>Navidrome</title>` and logs nothing.
+
+## Upgrading
+
+```bash
+curl -sL https://github.com/navidrome/navidrome/releases/download/v<ver>/navidrome_checksums.txt \
+  | grep linux_amd64.tar.gz
+```
+
+Bump `navidrome_version` and `navidrome_sha256` **together** in
+`defaults/main.yml`, then re-run the playbook. The install is version-gated, so
+an unchanged version is a no-op.
+
+## Sizing
+
+1 core / 1 GB / 50 GB. Two things to know:
+
+- **50 GB is the library ceiling**, not just the OS — `/srv/music` is on the root
+  disk. Roughly 300–450 albums in FLAC. Growing it means a Proxmox resize plus
+  `growpart`/`resize2fs` (see `03_vm_disk_enlargement.md`), because
+  `ignore_changes = all` in the tofu config means tofu will not do it.
+- **1 GB is fine for serving, tight for scanning.** The initial scan of a large
+  library is the memory peak. If it gets OOM-killed, do what `monerod-box`
+  needed: raise it in the Proxmox UI and record it back in `vms.auto.tfvars`.
+
+Transcoding is the CPU cost, and it is avoidable: clients that direct-play
+(Symfonium, Amperfy) make the server a file shipper. Roku and browsers transcode.