navidrome: a music server on its own box, with beets in front of it

Navidrome 0.64.0 on music-box (nodito VM 8200, 1 core / 1 GB / 50 GB), public
at music.contrapeso.xyz. A single Go binary with a SQLite database; the only
external dependency is ffmpeg for transcoding. Plus beets, the CLI that fixes
tags before Navidrome ever indexes them.

── Exposed without basic auth, on purpose ─────────────────────────────────
Navidrome speaks two protocols on one port: its web UI, and the Subsonic API
(/rest/*) that every mobile client actually uses. A Subsonic client
authenticates with ?u=&t=md5(password+salt)&s= in the query string and has
nowhere to carry a second credential, so Basic Auth in front of the site is
invisible to most of them. The choice is a proxy credential or a working phone.

Same call this estate already made for monerod's restricted RPC, for the same
reason: proxied bare so Feather can reach it, with the service's own auth doing
the authenticating.

The cost is stated rather than glossed: the login page and the whole /rest/*
surface are reachable by anyone who finds the hostname, and Navidrome has no
rate limiting and no 2FA.

A fail2ban jail on vipy reads a per-vhost Caddy access log and catches WEB UI
brute force, where a bad login returns 401. It very likely does NOT catch
Subsonic brute force, because that protocol answers a bad password with HTTP
200 and puts the failure in the body - which Caddy never sees. Nor can it be
fixed on the music box: the request arrives from vipy over the tailnet, so the
only address fail2ban could ban there is vipy's, which would take down every
service behind the edge. Banning has to happen at the edge; the edge cannot see
the failure. So /rest/* is protected by password strength alone, and the README
carries a one-line curl to re-check that after every upgrade.

── Passwords are stored reversibly, and cannot not be ─────────────────────
The Subsonic API requires the server to compute md5(password+salt) per request,
so Navidrome cannot hash passwords one-way. It encrypts them at rest with
PasswordEncryptionKey instead. Two real consequences, both in the README: root
on that box means every Navidrome password in plaintext, so the password must
be unique to this service; and changing the key invalidates every stored
password, so it is set before the first account and left alone.

── beets is not a service, deliberately ───────────────────────────────────
`beet import` is interactive and it MOVES AND RENAMES FILES. A timer that
reorganises a music library unattended is a way to lose one. Installed and
configured, run by hand.

/srv/music is counterweight:navidrome mode 2750 and the setgid bit is
load-bearing: without it beets' files land in the curator's own group,
Navidrome cannot read them, and the symptom is an EMPTY LIBRARY rather than a
permission error. ReplayGain and lyrics are auto: no - on one core they turn a
two-minute import into a twenty-minute one.

── Monitoring ─────────────────────────────────────────────────────────────
Host liveness and disk (400), the systemd unit (401), a probe that fetches
/app/ and greps it (403), and public reachability plus cert expiry (402).

/app/ was chosen by testing all three candidates against 0.64.0: /ping returns
a bare "." and proves only that something listens; /rest/ping.view identifies
the server but logs two warnings per call, ~576 a day; /app/ returns
<title>Navidrome</title> and logs nothing. 402 asserts 200 on /app/ and not on
"/", because "/" answers 302 - checked live rather than assumed.

── Also ───────────────────────────────────────────────────────────────────
The headscale ACL grants tag:prd-vipy exactly tag:music-box:4533. Without a
rule the node is invisible rather than unreachable, which surfaces as a Caddy
502 "no such host".

inventory lan_ip was read off nodito's ARP table for the VM's own MAC, not
guessed - and .149 had belonged to another machine, whose stale host key made
SSH refuse the connection. That is the designed behaviour, not a fault.

50 GB is the LIBRARY ceiling, not just the OS: /srv/music is on the root disk,
so roughly 300-450 albums in FLAC. 1 GB is fine for serving and tight for the
initial scan, which is the memory peak.

No backups, deliberately. The music files would swamp the age-encrypted nightly
path, and navidrome.db is a rebuildable index. What is NOT rebuildable is
playlists, play counts, ratings and accounts - there are none today, which is
what makes the trade fair, and the README names this as the decision to revisit
if that changes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ansible/services/headscale/templates/acl.json.j2

@@ -42,6 +42,7 @@
         "tag:fulcrum-box:50002",
         "tag:mempool-box:8080",
         "tag:memos-box:5230",
+        "tag:music-box:4533",
         "tag:monerod-box:18089",
         "tag:monerod-box:8081",
         "tag:arbret-staging-box:80",
@@ -113,6 +114,9 @@
     "tag:monerod-box": [
       "{{ headscale_namespace }}@"
     ],
+    "tag:music-box": [
+      "{{ headscale_namespace }}@"
+    ],
     "tag:nodito": [
       "{{ headscale_namespace }}@"
     ],

ansible/services/navidrome/deploy_navidrome_playbook.yml

@@ -0,0 +1,163 @@
+---
+# Navidrome + beets on music-box, published at music.contrapeso.xyz.
+#
+# See roles/navidrome/README.md for the exposure and backup decisions. The two
+# that matter here: no basic_auth in front (the Subsonic API the mobile clients
+# speak has nowhere to carry a second credential), and the fail2ban jail below
+# covers the web UI only.
+- name: Deploy Navidrome on music-box
+  hosts: music
+  become: yes
+  vars:
+    # Vault key is navidrome_encryption_key, not the role's own variable name -
+    # assigning a var to itself is a recursive reference, not an override.
+    navidrome_password_encryption_key: "{{ navidrome_encryption_key }}"
+    healthcheck_push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/probe_navidrome/external"
+    healthcheck_push_token: "{{ gatus_push_tokens[inventory_hostname] }}"
+  roles:
+    - navidrome
+
+- name: Publish Navidrome from the edge host
+  hosts: edge
+  become: yes
+  vars:
+    navidrome_access_log: /var/log/caddy/music.log
+  tasks:
+    # The Debian package usually creates this, but `caddy validate` passes with
+    # the directory missing and the RELOAD is what fails - so the vhost would
+    # look deployed and serve nothing. Make it explicit.
+    - name: Ensure the Caddy log directory exists
+      ansible.builtin.file:
+        path: "{{ navidrome_access_log | dirname }}"
+        state: directory
+        owner: caddy
+        group: caddy
+        mode: '0750'
+
+    # The FILE, not just the directory, and this is load-bearing.
+    #
+    # `caddy validate` below runs as root and provisions the logging module,
+    # which CREATES this file root:root 0600. Caddy itself runs as the caddy
+    # user, so the subsequent reload then dies with:
+    #   loading new config: setting up custom log 'log0': ...
+    #   open /var/log/caddy/music.log: permission denied
+    # and - the nasty part - `systemctl reload` hangs rather than failing,
+    # leaving the unit stuck in "reloading" while the OLD config keeps serving.
+    # So the vhost looks deployed, validate passes, nothing reports an error,
+    # and the site is simply absent.
+    #
+    # Creating it up front with the right owner makes validate's open a no-op.
+    - name: Ensure the Caddy access log file exists and is owned by caddy
+      ansible.builtin.file:
+        path: "{{ navidrome_access_log }}"
+        state: touch
+        owner: caddy
+        group: caddy
+        mode: '0640'
+        access_time: preserve
+        modification_time: preserve
+
+    # caddy_site_body rather than caddy_site_upstream + caddy_site_resolvers:
+    # this vhost needs a `log` directive so fail2ban has something to read, and
+    # the role's template has no knob for one. The escape hatch is the
+    # documented way to do this - gatus and headscale already use it - and it
+    # leaves the other vhosts untouched.
+    - name: Expose Navidrome over HTTPS
+      ansible.builtin.include_role:
+        name: caddy_site
+      vars:
+        caddy_site_name: music
+        caddy_site_domain: "{{ subdomains.music }}.{{ root_domain }}"
+        caddy_site_body: |
+          log {
+              output file {{ navidrome_access_log }} {
+                  roll_size 10MiB
+                  roll_keep 5
+              }
+              format json
+          }
+          reverse_proxy {{ hostvars['music_box_local'].ansible_host }}:{{ hostvars['music_box_local'].navidrome_port }} {
+              # Tailscale MagicDNS: music-box has no public address.
+              transport http {
+                  resolvers 100.100.100.100
+              }
+          }
+
+    # The per-site validate cannot catch a conflict between sites.
+    - name: Validate the whole Caddyfile
+      ansible.builtin.command: caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile
+      changed_when: false
+
+    # ── fail2ban ────────────────────────────────────────────────────────────
+    # The estate's first jail: 02_firewall_and_fail2ban_playbook.yml installs
+    # fail2ban everywhere but leaves it on stock config. Kept inline here
+    # because one jail does not justify a role; if a second service ever wants
+    # one, that is the moment to promote this to roles/fail2ban_jail alongside
+    # caddy_site and gatus_endpoint.
+    #
+    # Scope, stated so it is not mistaken for more than it is: this catches web
+    # UI brute force (401 on /auth/login). It does NOT catch Subsonic API brute
+    # force, because Subsonic answers a bad password with HTTP 200 and the error
+    # in the body. See roles/navidrome/README.md.
+    - name: Install the navidrome fail2ban filter
+      ansible.builtin.copy:
+        dest: /etc/fail2ban/filter.d/navidrome-caddy.conf
+        owner: root
+        group: root
+        mode: '0644'
+        content: |
+          # Managed by Ansible - services/navidrome/deploy_navidrome_playbook.yml
+          #
+          # The log file is per-vhost, so any 401 in it is a failed Navidrome
+          # login and needs no path matching.
+          [Definition]
+          failregex = "remote_ip":"<HOST>".*"status":401
+          ignoreregex =
+      notify: Restart fail2ban
+
+    - name: Install the navidrome fail2ban jail
+      ansible.builtin.copy:
+        dest: /etc/fail2ban/jail.d/navidrome.conf
+        owner: root
+        group: root
+        mode: '0644'
+        content: |
+          # Managed by Ansible - services/navidrome/deploy_navidrome_playbook.yml
+          [navidrome]
+          enabled   = true
+          filter    = navidrome-caddy
+          logpath   = {{ navidrome_access_log }}
+          # ufw, not the Debian default iptables-multiport: UFW is what manages
+          # the firewall on every host here (infra/02_firewall_and_fail2ban).
+          banaction = ufw
+          maxretry  = 5
+          findtime  = 10m
+          bantime   = 1h
+      notify: Restart fail2ban
+
+    - name: Ensure fail2ban is enabled and running
+      systemd:
+        name: fail2ban
+        enabled: yes
+        state: started
+
+    - name: Flush handlers so the jail is loaded before we assert on it
+      meta: flush_handlers
+
+    # A jail that silently failed to load is worse than no jail: it reads as
+    # protection in `ls /etc/fail2ban/jail.d` and bans nothing.
+    - name: Assert the navidrome jail is actually active
+      ansible.builtin.command: fail2ban-client status navidrome
+      register: navidrome_jail_status
+      changed_when: false
+      failed_when: navidrome_jail_status.rc != 0
+
+    - name: Display the navidrome jail status
+      ansible.builtin.debug:
+        var: navidrome_jail_status.stdout_lines
+
+  handlers:
+    - name: Restart fail2ban
+      systemd:
+        name: fail2ban
+        state: restarted