2025-10-22 23:58:38 +02:00
- name : Deploy headscale and configure Caddy reverse proxy
2026-09-11 21:56:14 +02:00
hosts : vpn_control
2025-10-22 23:58:38 +02:00
become : no
vars_files :
- ./headscale_vars.yml
vars :
2025-11-06 23:09:44 +01:00
headscale_subdomain : "{{ subdomains.headscale }}"
2025-10-22 23:58:38 +02:00
headscale_domain : "{{ headscale_subdomain }}.{{ root_domain }}"
2025-11-03 16:51:53 +01:00
headscale_base_domain : "tailnet.{{ root_domain }}"
2025-10-22 23:58:38 +02:00
tasks :
- name : Install required packages
become : yes
apt :
name :
- wget
- gnupg
state : present
update_cache : yes
- name : Download headscale DEB package
get_url :
url : "https://github.com/juanfont/headscale/releases/download/v{{ headscale_version }}/headscale_{{ headscale_version }}_linux_amd64.deb"
dest : /tmp/headscale.deb
mode : '0644'
- name : Install headscale package
become : yes
apt :
deb : /tmp/headscale.deb
state : present
- name : Remove temporary DEB file
file :
path : /tmp/headscale.deb
state : absent
2025-11-03 16:51:53 +01:00
- name : Ensure headscale user exists
become : yes
user :
name : headscale
system : yes
shell : /usr/sbin/nologin
home : /var/lib/headscale
create_home : yes
state : present
2025-10-22 23:58:38 +02:00
- name : Create headscale data directory
become : yes
file :
path : /var/lib/headscale
state : directory
owner : headscale
group : headscale
mode : '0750'
- name : Create headscale run directory
become : yes
file :
path : /var/run/headscale
state : directory
owner : headscale
group : headscale
2025-11-03 16:51:53 +01:00
mode : '0770'
2025-10-22 23:58:38 +02:00
- name : Ensure headscale user owns data directory
become : yes
file :
path : /var/lib/headscale
owner : headscale
group : headscale
recurse : yes
2025-11-03 16:51:53 +01:00
mode : '0750'
- name : Add counterweight user to headscale group
become : yes
user :
name : counterweight
groups : headscale
append : yes
2025-10-22 23:58:38 +02:00
headscale: the ACL was `{}` in code and real on the box. Put it in git
`Create ACL policies file` wrote this, verbatim, to /etc/headscale/acl.json:
content: |
{}
An empty policy. The real rules - who may reach which host on which port -
were maintained by hand on spacey and existed nowhere else. So the repo and
the box disagreed about the single file that governs every machine's access
to every other machine, and the repo's version was "nothing is allowed".
It had already fired once. /etc/headscale/acl.json.937464.2026-09-12@14:03:08~
is Ansible's own backup of a 1596-byte policy it replaced with {} on 12 Sep;
the live file is dated 17 Sep, so somebody rebuilt it by hand afterwards
without the playbook ever being corrected. Every subsequent `site.yml` run was
one command away from flattening the whole tailnet's access rules again.
This commit changes nothing on spacey. The policy now lives in
templates/acl.json.j2, captured verbatim from the live file, and the task is a
`template:` instead of an inline `{}`. Rendering it produces sha256
1ed14e7247d957db5e9f8d4f4c53846dc646131ad900054c868483a9a84ed8fd, byte for byte
the file that was already there - verified before applying, and again after.
Three deliberate choices:
- A template file, not the inline `content:` used elsewhere in this playbook.
The whole failure was three invisible characters passing review; a 2KB
policy needs to show up as a diff.
- `validate: python3 -m json.tool %s`. headscale refuses to load a malformed
policy, and a policy it will not load is the same outage as an empty one.
- `"counter-net@"` is now `"{{ headscale_namespace }}@"`, which is already a
fact in group_vars/all/main.yml rather than a fourteenth hand-typed copy.
The header comment records why a node with no ACL rule is INVISIBLE rather than
merely unreachable - it never appears in `tailscale status`, MagicDNS does not
resolve it, and the symptom is a downstream "no such host" that looks like a
DNS problem. That cost an hour today.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-19 17:49:54 +02:00
# This used to write `{}` - an empty policy - over the real ACL, which was
# maintained by hand on the box. It had already flattened it once
# (/etc/headscale/acl.json.937464.2026-09-12@14:03:08~ is Ansible's own
# backup of the 1596-byte policy it replaced), and every subsequent
# site.yml run would have taken the whole tailnet's access rules with it.
#
# The policy now lives in templates/acl.json.j2, captured from the live
# file on 2026-09-19. validate first: a malformed policy is one headscale
# refuses to load, which is the same outage as an empty one.
- name : Deploy the Headscale ACL policy
2025-10-22 23:58:38 +02:00
become : yes
headscale: the ACL was `{}` in code and real on the box. Put it in git
`Create ACL policies file` wrote this, verbatim, to /etc/headscale/acl.json:
content: |
{}
An empty policy. The real rules - who may reach which host on which port -
were maintained by hand on spacey and existed nowhere else. So the repo and
the box disagreed about the single file that governs every machine's access
to every other machine, and the repo's version was "nothing is allowed".
It had already fired once. /etc/headscale/acl.json.937464.2026-09-12@14:03:08~
is Ansible's own backup of a 1596-byte policy it replaced with {} on 12 Sep;
the live file is dated 17 Sep, so somebody rebuilt it by hand afterwards
without the playbook ever being corrected. Every subsequent `site.yml` run was
one command away from flattening the whole tailnet's access rules again.
This commit changes nothing on spacey. The policy now lives in
templates/acl.json.j2, captured verbatim from the live file, and the task is a
`template:` instead of an inline `{}`. Rendering it produces sha256
1ed14e7247d957db5e9f8d4f4c53846dc646131ad900054c868483a9a84ed8fd, byte for byte
the file that was already there - verified before applying, and again after.
Three deliberate choices:
- A template file, not the inline `content:` used elsewhere in this playbook.
The whole failure was three invisible characters passing review; a 2KB
policy needs to show up as a diff.
- `validate: python3 -m json.tool %s`. headscale refuses to load a malformed
policy, and a policy it will not load is the same outage as an empty one.
- `"counter-net@"` is now `"{{ headscale_namespace }}@"`, which is already a
fact in group_vars/all/main.yml rather than a fourteenth hand-typed copy.
The header comment records why a node with no ACL rule is INVISIBLE rather than
merely unreachable - it never appears in `tailscale status`, MagicDNS does not
resolve it, and the symptom is a downstream "no such host" that looks like a
DNS problem. That cost an hour today.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-19 17:49:54 +02:00
ansible.builtin.template :
src : acl.json.j2
2025-10-22 23:58:38 +02:00
dest : /etc/headscale/acl.json
owner : headscale
group : headscale
2025-11-03 16:51:53 +01:00
mode : '0640'
headscale: the ACL was `{}` in code and real on the box. Put it in git
`Create ACL policies file` wrote this, verbatim, to /etc/headscale/acl.json:
content: |
{}
An empty policy. The real rules - who may reach which host on which port -
were maintained by hand on spacey and existed nowhere else. So the repo and
the box disagreed about the single file that governs every machine's access
to every other machine, and the repo's version was "nothing is allowed".
It had already fired once. /etc/headscale/acl.json.937464.2026-09-12@14:03:08~
is Ansible's own backup of a 1596-byte policy it replaced with {} on 12 Sep;
the live file is dated 17 Sep, so somebody rebuilt it by hand afterwards
without the playbook ever being corrected. Every subsequent `site.yml` run was
one command away from flattening the whole tailnet's access rules again.
This commit changes nothing on spacey. The policy now lives in
templates/acl.json.j2, captured verbatim from the live file, and the task is a
`template:` instead of an inline `{}`. Rendering it produces sha256
1ed14e7247d957db5e9f8d4f4c53846dc646131ad900054c868483a9a84ed8fd, byte for byte
the file that was already there - verified before applying, and again after.
Three deliberate choices:
- A template file, not the inline `content:` used elsewhere in this playbook.
The whole failure was three invisible characters passing review; a 2KB
policy needs to show up as a diff.
- `validate: python3 -m json.tool %s`. headscale refuses to load a malformed
policy, and a policy it will not load is the same outage as an empty one.
- `"counter-net@"` is now `"{{ headscale_namespace }}@"`, which is already a
fact in group_vars/all/main.yml rather than a fourteenth hand-typed copy.
The header comment records why a node with no ACL rule is INVISIBLE rather than
merely unreachable - it never appears in `tailscale status`, MagicDNS does not
resolve it, and the symptom is a downstream "no such host" that looks like a
DNS problem. That cost an hour today.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-19 17:49:54 +02:00
validate : "python3 -m json.tool %s"
2025-10-22 23:58:38 +02:00
notify : Restart headscale
- name : Deploy headscale configuration file
become : yes
copy :
dest : /etc/headscale/config.yaml
content : |
server_url : https://{{ headscale_domain }}
listen_addr : 0.0 .0 .0 : {{ headscale_port }}
grpc_listen_addr : 0.0 .0 .0 : {{ headscale_grpc_port }}
grpc_allow_insecure : false
private_key_path : /var/lib/headscale/private.key
noise :
private_key_path : /var/lib/headscale/noise_private.key
prefixes :
v4 : 100.64 .0 .0 /10
v6 : fd7a:115c:a1e0::/48
derp :
server :
enabled : true
region_id : 999
region_code : "headscale"
region_name : "Headscale Embedded DERP"
verify_clients : true
stun_listen_addr : "0.0.0.0:3478"
private_key_path : /var/lib/headscale/derp_server_private.key
automatically_add_embedded_derp_region : true
urls :
- https://controlplane.tailscale.com/derpmap/default
database :
type : sqlite3
sqlite :
path : /var/lib/headscale/db.sqlite
unix_socket : /var/run/headscale/headscale.sock
unix_socket_permission : "0770"
log :
level : info
format : text
policy :
path : /etc/headscale/acl.json
dns :
2025-11-03 16:51:53 +01:00
base_domain : {{ headscale_base_domain | quote }}
2025-10-22 23:58:38 +02:00
magic_dns : true
search_domains :
2025-11-03 16:51:53 +01:00
- {{ headscale_base_domain | quote }}
2025-10-22 23:58:38 +02:00
nameservers :
global :
- 1.1 .1 .1
- 1.0 .0 .1
owner : root
2025-11-03 16:51:53 +01:00
group : headscale
mode : '0640'
2025-10-22 23:58:38 +02:00
notify : Restart headscale
- name : Test headscale configuration
become : yes
command : headscale configtest
register : headscale_config_test
failed_when : headscale_config_test.rc != 0
- name : Display headscale config test results
debug :
msg : "{{ headscale_config_test.stdout }}"
2025-11-06 23:09:44 +01:00
- name : Ensure headscale data directory has correct ownership before starting service
become : yes
file :
path : /var/lib/headscale
state : directory
owner : headscale
group : headscale
mode : '0750'
recurse : yes
- name : Ensure headscale run directory has correct ownership
become : yes
file :
path : /var/run/headscale
state : directory
owner : headscale
group : headscale
mode : '0770'
2025-10-22 23:58:38 +02:00
- name : Enable and start headscale service
become : yes
systemd :
name : headscale
enabled : yes
state : started
2025-11-06 23:09:44 +01:00
daemon_reload : yes
2025-10-22 23:58:38 +02:00
2025-11-03 16:51:53 +01:00
- name : Wait for headscale unix socket to be ready
become : yes
wait_for :
path : /var/run/headscale/headscale.sock
state : present
timeout : 60
delay : 2
- name : Create headscale namespace if it doesn't exist
become : yes
command : headscale users create {{ headscale_namespace }}
register : create_namespace_result
failed_when : create_namespace_result.rc != 0 and 'already exists' not in create_namespace_result.stderr and 'UNIQUE constraint' not in create_namespace_result.stderr
changed_when : create_namespace_result.rc == 0
2025-10-22 23:58:38 +02:00
- name : Allow HTTPS through UFW
become : yes
ufw :
rule : allow
port : '443'
proto : tcp
- name : Allow HTTP through UFW (for Let's Encrypt)
become : yes
ufw :
rule : allow
port : '80'
proto : tcp
- name : Allow STUN through UFW (for DERP server)
become : yes
ufw :
rule : allow
port : '3478'
proto : udp
ntfy, datum-gateway, headscale: use the caddy_site role
Completes Stage 3. No hand-rolled Caddy plumbing remains anywhere:
`grep sites-enabled` outside roles/ returns nothing, and so does
`grep "systemctl reload caddy"`.
ntfy uses caddy_site_body for its plain-HTTP listener and @httpget redirect.
Verified ok/unchanged against watchtower; the one other changed task is a
pre-existing "Update APT cache".
datum-gateway keeps a whole-Caddyfile validate after the role call. The role
validates its own fragment, but only a whole-file validate catches a conflict
between two sites, and this playbook was the only one that ever had it. Its
two debug tasks that echoed command output are gone with the commands.
headscale is the one that mattered. Its playbook wrote
`reverse_proxy localhost:8080`, but spacey is actually running a /admin*
route in front of Headplane behind Caddy basic auth. Running that playbook
would have deleted the admin route and its auth - a hazard that predates this
work. It now renders the config that is really there, verified ok/unchanged
via --start-at-task (the play cannot reach Caddy in check mode: "Install
headscale package" fails because the .deb is not really downloaded, before
and after this edit alike).
Supporting changes for headscale:
- headscale_ui_password_hash added to infra_secrets.yml and the identical
group_vars/all/vault.yml, read from the live config on spacey. The vault
already had headscale_ui_username (= counterweight, confirmed) and
headscale_ui_password; I did not verify the password is the plaintext of
this hash.
- headplane_port added to headscale_vars.yml.
- The role's handler now sets become: true. Handlers do not inherit become
from the task that notified them, and this play runs become: no.
- The include uses `apply: become: yes`; `become:` on an include_role is
rejected outright.
All 14 site files on all 3 hosts still byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-11 23:37:38 +02:00
- name : Publish headscale through Caddy
ansible.builtin.include_role :
name : caddy_site
# This play is become: no and elevates per task. `apply` is how an
# include_role passes become down to the role's tasks - `become:` on
# the include itself is rejected. The role's handler sets its own.
apply :
become : yes
vars :
caddy_site_name : headscale
caddy_site_domain : "{{ headscale_domain }}"
2026-10-19 22:10:23 +02:00
# The Headscale API authenticates every request itself (Bearer API
# key), so a plain proxy is all it needs.
caddy_site_upstream : "localhost:{{ headscale_port }}"
2025-10-22 23:58:38 +02:00
handlers :
- name : Restart headscale
become : yes
systemd :
name : headscale
state : restarted