forgejo-runner: give CI jobs a Docker daemon of their own

jar-lid-archive's CI needs `docker build` and `docker compose`, and job
containers had no Docker at all: config.yml carried the stock
`container.docker_host: "-"`, which is precisely "find a daemon for the
runner and do not let jobs near it".

A docker:dind sidecar now runs on forgejo-runner-box, published on
127.0.0.1 only, and jobs build against that instead of the host daemon.
Following Forgejo's own admin docs, which make the point that this is
THREE settings and not one:

  forgejo-dind.service        a privileged docker:dind on the host daemon
  container.docker_host       tcp://127.0.0.1:2376 - where the runner
                              CREATES job containers
  runner.envs.DOCKER_HOST +   tcp://forgejo-dind.docker.internal:2375 -
  container.options           how a job, from INSIDE dind, reaches it

The third is the one that is easy to miss and fails identically to
having done nothing: for a tcp:// docker_host the runner mounts nothing
into the job container, so the job still finds no daemon. The two
addresses differ because they are used from different places - the
runner is a process on the host; a job container is inside dind, where
127.0.0.1 is itself. `host-gateway` is expanded by the daemon creating
the container, which is dind, so it lands on dind's own bridge gateway.

Verified on the box, along the exact path a job takes - a container on a
user-defined network, with the same --add-host and DOCKER_HOST the
runner will set: the name resolves to 172.20.0.1, `docker version`
returns the dind server, and `docker build` succeeds. The host daemon
runs forgejo-dind and nothing else.

config.yml had to become a template to do any of this. It was previously
`generate-config` run once on first deploy and never touched again, so
the repo had no say in its contents. Every value was transcribed from
the live file and diffed against it key by key: with dind disabled, 30
of 30 identical. Two deliberate differences - runner.envs drops
A_TEST_ENV_NAME_1/2, upstream's placeholders, which were being injected
into every job here; and the upstream comment blob is gone. Because the
file is managed now it also gets a restart handler, the absence of which
has bitten this repo before.

Things deliberately chosen rather than defaulted into:

- the flag lives in the playbook, not the role defaults, so the role's
  default stays the configuration this host has always run and turning
  it off is one line. Disabling it also tears the units down rather than
  just skipping them.
- dind is a hard Requires= of forgejo-runner, and its unit stays in
  "activating" until the daemon answers. A dead dind is then one red
  unit instead of a green runner failing every job it claims.
- the healthcheck and monitored_services learned about dind for the same
  reason: runner up + dind down is the interesting failure and it was
  previously invisible.
- /var/lib/docker is a named volume, or every image is re-pulled on each
  container recreation; a weekly prune keeps that from growing forever.
- dind's network pool is pinned to 172.20.0.0/14. Docker's stock pool
  reaches into 192.168.x, and act_runner creates a network per job, so a
  job could eventually allocate this LAN's subnet and lose the route to
  every box on it.

Note this runner is SHARED - arbret, arbret-biz, arbretdos and
arbret-ops build here too, far more often, and docker_host is per-runner
not per-repo. Their workflows are unchanged but their next run pulls
images into a cold cache. Job images still need a docker client of their
own: node:{20,22}-bookworm ship none.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ansible/roles/forgejo_runner/README.md

@@ -57,3 +57,100 @@ never ran. It is not a monitoring task — it is the deployment checking its own
 work — and the deprecation banner swept it up by mistake. It is ungated here,
 which means a runner that fails to start now fails the play instead of
 deploying "successfully" in silence.
+
+## Docker-in-Docker for jobs
+
+`forgejo_runner_dind_enabled: true` gives Actions jobs a Docker daemon of their
+own — a `docker:dind` container on this host's daemon, isolated from it — so
+`docker build` and `docker compose` work inside a workflow.
+
+It follows Forgejo's own [Utilizing Docker within
+Actions](https://forgejo.org/docs/latest/admin/actions/docker-access/) →
+Docker-in-Docker, which is **three** parts, not one:
+
+| | what it does | where it applies |
+|---|---|---|
+| `forgejo-dind.service` | a `docker:dind` container, published on `127.0.0.1:2376` only | the host |
+| `container.docker_host` | `tcp://127.0.0.1:2376` — the runner creates job/service/step containers **on dind** | the runner process |
+| `runner.envs.DOCKER_HOST` + `container.options` | `tcp://forgejo-dind.docker.internal:2375`, resolved by `--add-host=…:host-gateway` | inside each job container |
+
+The third row is the one that is easy to miss, and leaving it out fails in a
+way that looks identical to not having configured anything. `docker_host` only
+decides **where the job container is created**; for a `tcp://` value Forgejo
+mounts nothing into that container, so the job still has no daemon. The two
+addresses differ because they are used from different places — the runner is a
+process on the host and reaches dind on loopback; a job container is *inside*
+dind, where `127.0.0.1` is itself. `host-gateway` is expanded by the daemon
+that creates the container, which is dind, so it lands on dind's own bridge
+gateway where its `dockerd` is listening.
+
+### What changes for jobs
+
+- Job containers, service containers and anything a job starts all live on the
+  dind daemon. A `postgres` service and a `docker compose` stack are siblings
+  there, so they reach each other by name — which is what makes end-to-end
+  tests work.
+- **Nothing a workflow does can touch this host's Docker daemon any more.**
+  That is the point of dind over `automount`.
+- dind has its own image store. Every image is pulled again the first time it
+  is needed. It is a named volume (`forgejo-dind-lib`), so it survives the
+  container being recreated, and `forgejo-dind-prune.timer` drops weekly what
+  has gone a week unused.
+- Bind mounts inside a job's own `docker compose` resolve on the **dind**
+  filesystem, not the job container's. A compose file that bind-mounts the
+  checked-out repo will not see it. Named volumes and `build:` contexts are
+  fine. This is inherent to dind, not to this role.
+
+### Things to know before enabling
+
+- **This runner is shared.** `arbret`, `arbret-biz`, `arbretdos` and
+  `arbret-ops` build here too, and `container.docker_host` is per *runner*, not
+  per repo — every one of them moves onto dind. Their workflows do not change;
+  their next run pulls images into a cold cache.
+- **`capacity` is 4 and dind is one daemon.** Concurrent jobs can see and touch
+  each other's containers and images through the Docker CLI. Forgejo's docs
+  call this out. Fine for a single-tenant runner; set `capacity: 1` if
+  untrusted code ever runs here.
+- **The dind container is `--privileged`**, which is why this belongs on a
+  dedicated CI VM and not beside anything that matters. `forgejo-runner-box`
+  runs the runner and nothing else.
+- **The daemon has no TLS and no authentication.** It is published on
+  `127.0.0.1` only, so reaching it means already having a shell on this box —
+  but it does mean any local user is root in a privileged container. Do not
+  move `forgejo_runner_dind_bind_address` off loopback.
+- **The `runner` user is still in the host `docker` group**, left over from
+  when the runner used the host daemon. With dind it no longer needs it. Worth
+  removing, deliberately, as its own change.
+- **`cache.host` is autodetected**, and job containers now reach the runner's
+  cache server across dind's NAT. Autodetect returns this box's LAN address,
+  which is routable from in there. If `actions/cache` starts silently missing,
+  pin `forgejo_runner_cache_host` — that is the first place to look.
+- **Deploy when the runner is idle.** `shutdown_timeout` is 3h, so restarting
+  it mid-job waits for the job.
+
+### Turning it off
+
+Set `forgejo_runner_dind_enabled: false` and re-run. The role stops and
+disables the units, removes them, and puts `container.docker_host` back to
+`"-"` — the exact configuration this host ran before. The image and the
+`forgejo-dind-lib` volume are left alone: they are a cache, they hold no
+secrets, and deleting them on a toggle is not the role's call.
+
+## `config.yml` is templated now
+
+It used to be `forgejo-runner generate-config`, run once on first deploy and
+never touched again — so the repo had no say in anything inside it, including
+the setting this feature needs. It is now `templates/config.yml.j2`.
+
+Every value in it was transcribed from the file live on `forgejo-runner-box`
+and diffed key-by-key against it: with dind disabled, 30 of 30 keys are
+identical. Two deliberate differences:
+
+- `runner.envs` no longer carries `A_TEST_ENV_NAME_1` / `A_TEST_ENV_NAME_2`,
+  upstream's example placeholders, which were being injected into every job.
+- the upstream comment blob is gone; the values are not.
+
+Because the file is managed, a change to it now restarts the runner
+(`handlers/main.yml`). Previously a config change could not happen at all; the
+failure mode where it lands on disk and the running process never picks it up
+is one this repo has hit before.

ansible/roles/forgejo_runner/defaults/main.yml

@@ -11,6 +11,27 @@ forgejo_runner_dir: "/opt/forgejo-runner"
 forgejo_runner_config_path: "{{ forgejo_runner_dir }}/config.yml"
 forgejo_runner_labels: "docker:docker://node:20-bookworm,ubuntu-latest:docker://node:20-bookworm,ubuntu-22.04:docker://node:20-bookworm,ubuntu-24.04:docker://node:20-bookworm"
 
+# --- config.yml contents ----------------------------------------------------
+# These reproduce, exactly, the stock `generate-config` output this host has
+# been running since first deploy. They are variables now only because the file
+# became a template; changing one of them changes the box.
+forgejo_runner_log_level: info
+forgejo_runner_log_job_level: info
+forgejo_runner_capacity: 4
+forgejo_runner_job_timeout: "3h"
+forgejo_runner_shutdown_timeout: "3h"
+forgejo_runner_cache_enabled: true
+# Empty = autodetect this host's outbound address. See the note in the template
+# before changing it; under dind this is the value that decides whether
+# actions/cache works.
+forgejo_runner_cache_host: ""
+forgejo_runner_container_network: ""
+forgejo_runner_valid_volumes: []
+# Extra job environment and extra `docker run` options for job containers.
+# Both MERGE with what dind adds rather than replacing it.
+forgejo_runner_envs: {}
+forgejo_runner_container_options: ""
+
 # The Forgejo instance this runner registers with.
 forgejo_instance_url: "https://forgejo.contrapeso.xyz"
 # forgejo_runner_registration_token comes from the vault.
@@ -39,3 +60,78 @@ healthcheck_push_url: ""
 # Bearer token for the Gatus external endpoint. Required whenever a push URL
 # is set: Gatus rejects an unauthenticated push with 401.
 healthcheck_push_token: ""
+
+# --- Docker-in-Docker for job containers ------------------------------------
+# Off by default. The role default is the SAFE state: `container.docker_host`
+# stays "-", which is what this host has always run, and jobs get no Docker at
+# all. services/forgejo-runner/deploy_forgejo_runner_playbook.yml turns it on.
+#
+# The shape is the one in Forgejo's own admin docs, "Utilizing Docker within
+# Actions" -> Docker-in-Docker, and it is three moving parts, not one:
+#
+#   1. a docker:dind container on the HOST daemon, published on loopback only;
+#   2. container.docker_host, which points the RUNNER PROCESS at that daemon,
+#      so job/service/step containers are created INSIDE dind;
+#   3. runner.envs.DOCKER_HOST plus a container.options --add-host, which is
+#      what lets the JOB reach the same daemon from inside.
+#
+# Step 3 is not optional and not a duplicate of step 2. The two addresses are
+# different because they are used from different places: the runner talks to
+# 127.0.0.1 on the host, and a job container inside dind cannot - its own
+# 127.0.0.1 is itself. Setting only docker_host gets you job containers that
+# run in dind but still cannot find a daemon, which looks exactly like the bug
+# we are fixing. See the README.
+forgejo_runner_dind_enabled: false
+
+forgejo_runner_dind_container: forgejo-dind
+# Already present on forgejo-runner-box, and the tag the project's own workflow
+# refers to. A floating 28.x picks up patch releases on container recreation.
+forgejo_runner_dind_image: "docker:28-dind"
+forgejo_runner_dind_service: forgejo-dind
+
+# Loopback ONLY. This daemon has no TLS and no authentication, so anything that
+# can open the port is root inside a privileged container. 127.0.0.1 keeps that
+# to processes on this box; do not move it to 0.0.0.0 or the LAN address.
+#
+# The number is 2376 because the upstream docs use it. 2376 is conventionally
+# the TLS port and this is PLAINTEXT - the port number promises nothing.
+forgejo_runner_dind_bind_address: "127.0.0.1"
+forgejo_runner_dind_host_port: 2376
+forgejo_runner_dind_container_port: 2375
+
+# The name a JOB container uses to reach the dind daemon. Resolved by an
+# --add-host ... :host-gateway on the job container, which Docker expands in
+# the context of the daemon that CREATES the container - dind - so it lands on
+# dind's own inner bridge gateway, where its dockerd is listening. That is why
+# a name is used here and not an address: the address is only knowable from
+# inside dind, at container-creation time.
+forgejo_runner_dind_internal_host: "forgejo-dind.docker.internal"
+
+# Networks for the containers dind creates. act_runner makes a fresh network
+# per job, so these multiply, and Docker's stock pool reaches into 192.168.x -
+# the same space as this LAN. A job container that allocated 192.168.1.0/24
+# would lose the route to every box on it, including this one. Pinning the pool
+# well clear of the LAN removes that entirely.
+forgejo_runner_dind_address_pool: "172.20.0.0/14"
+forgejo_runner_dind_address_pool_size: 24
+
+# How long the dind unit waits for its daemon to answer before failing. Cold
+# start pulls the image and initialises /var/lib/docker.
+forgejo_runner_dind_ready_timeout: 180
+
+# dind keeps its own /var/lib/docker. Without a volume it is wiped every time
+# the container is recreated, and every image every workflow uses - the
+# playwright image alone is 3.3GB - is pulled again from scratch.
+forgejo_runner_dind_volume: forgejo-dind-lib
+
+# ...which then grows without bound, so it is pruned. `until=168h` keeps
+# anything used in the last week, which is the cache that makes the volume
+# worth having, and drops the rest.
+#
+# Volumes are deliberately NOT pruned: `until` does not apply to them, so
+# --volumes here would be "delete every volume not currently attached", and a
+# workflow's named volume between two jobs is not an obvious thing to destroy
+# on a timer. Watch disk and prune those by hand.
+forgejo_runner_dind_prune_enabled: true
+forgejo_runner_dind_prune_on_calendar: "Sun 04:30"
+forgejo_runner_dind_prune_filter: "until=168h"

ansible/roles/forgejo_runner/handlers/main.yml

@@ -0,0 +1,31 @@
+---
+- name: Reload systemd
+  ansible.builtin.systemd:
+    daemon_reload: yes
+
+- name: Restart dind
+  ansible.builtin.systemd:
+    name: "{{ forgejo_runner_dind_service }}.service"
+    state: restarted
+    daemon_reload: yes
+  # In check mode the unit file was never written, so systemd has never heard
+  # of it and this fails on "Could not find the requested service" - an
+  # artefact of the dry run, not a problem with the run. Skipping it keeps
+  # `--check --diff` able to reach the config.yml diff, which is the part of a
+  # dry run worth reading.
+  when: not ansible_check_mode
+
+# The role had no handlers at all before, because it managed no config: the
+# runner's config.yml was generated once on first deploy and never touched
+# again, so there was nothing a change could need to restart. Now that
+# config.yml is a template, a change to it that did NOT restart the runner
+# would be the worst of both - the file on disk right and the running process
+# still on the old settings. That failure mode has bitten this repo before.
+#
+# Note the runner's shutdown_timeout is 3h: a restart while a job is in flight
+# waits for it, up to systemd's stop timeout. Deploy when the runner is idle.
+- name: Restart forgejo-runner
+  ansible.builtin.systemd:
+    name: forgejo-runner
+    state: restarted
+    daemon_reload: yes

ansible/roles/forgejo_runner/tasks/configure.yml

@@ -1,23 +1,32 @@
 ---
-- name: Check if config already exists
-  stat:
-    path: "{{ forgejo_runner_config_path }}"
-  register: config_stat
-
-- name: Generate default config
-  shell: "{{ forgejo_runner_bin_path }} generate-config > {{ forgejo_runner_config_path }}"
-  args:
-    chdir: "{{ forgejo_runner_dir }}"
-  when: not config_stat.stat.exists
-
-- name: Set config file ownership
-  file:
-    path: "{{ forgejo_runner_config_path }}"
+# config.yml is a TEMPLATE now, not a one-shot `generate-config`.
+#
+# Before this, the file was generated on first deploy and never touched again,
+# so the repo had no say in anything inside it - including container.docker_host,
+# which is the setting that decides whether jobs can use Docker at all. The
+# alternative was patching keys in place with a regex, which in a YAML file
+# with repeated key names under different parents is the kind of thing that
+# works until it silently does not.
+#
+# owner/group/mode are runner:runner 0644, read off the live box, matching what
+# `generate-config` plus the old chown produced. Nothing secret is in here; the
+# registration token lives in .runner.
+- name: Write the forgejo-runner configuration
+  ansible.builtin.template:
+    src: config.yml.j2
+    dest: "{{ forgejo_runner_config_path }}"
     owner: "{{ forgejo_runner_user }}"
     group: "{{ forgejo_runner_user }}"
-  when: not config_stat.stat.exists
+    mode: '0644'
+    # A malformed config is a runner that will not start. Catch it here, while
+    # the old file is still in place, rather than after it has been replaced.
+    validate: "python3 -c \"import yaml,sys; yaml.safe_load(open(sys.argv[1]))\" %s"
+  notify: Restart forgejo-runner
 
-# ── 6. Register runner ─────────────────────────────────────────────
+# ── Register runner ────────────────────────────────────────────────────────
+# Registration is separate from configuration and happens once. The labels are
+# frozen into .runner at this point; changing forgejo_runner_labels later does
+# not move them, which is why the label set is left exactly as it was.
 - name: Check if runner is already registered
   stat:
     path: "{{ forgejo_runner_dir }}/.runner"
@@ -40,4 +49,3 @@
     owner: "{{ forgejo_runner_user }}"
     group: "{{ forgejo_runner_user }}"
   when: not runner_stat.stat.exists
-

ansible/roles/forgejo_runner/tasks/dind.yml

@@ -0,0 +1,140 @@
+---
+# The isolated Docker daemon jobs build against.
+#
+# Runs BEFORE configure.yml and service.yml on purpose: config.yml names this
+# daemon's address, and forgejo-runner.service Requires= this unit. Creating it
+# after either would mean one run where the runner points at nothing.
+
+# ── Disabled: undo it, do not just skip it ─────────────────────────────────
+# A flag that only ever adds is not a flag. If dind is switched off the runner
+# goes back to container.docker_host "-" a few tasks from now, and a dind still
+# running would be a privileged container nothing uses. The volume and the
+# image are deliberately left: they are the image cache, they hold no secrets,
+# and destroying them on a toggle is not this role's call.
+- name: Find the dind unit when dind is disabled
+  ansible.builtin.stat:
+    path: "/etc/systemd/system/{{ forgejo_runner_dind_service }}.service"
+  register: dind_unit_leftover
+  when: not forgejo_runner_dind_enabled
+
+- name: Stop and disable dind
+  ansible.builtin.systemd:
+    name: "{{ item }}"
+    state: stopped
+    enabled: no
+  loop:
+    - "{{ forgejo_runner_dind_service }}.service"
+    - "{{ forgejo_runner_dind_service }}-prune.timer"
+  failed_when: false
+  when:
+    - not forgejo_runner_dind_enabled
+    - dind_unit_leftover.stat.exists | default(false)
+
+- name: Remove the dind units
+  ansible.builtin.file:
+    path: "/etc/systemd/system/{{ item }}"
+    state: absent
+  loop:
+    - "{{ forgejo_runner_dind_service }}.service"
+    - "{{ forgejo_runner_dind_service }}-prune.service"
+    - "{{ forgejo_runner_dind_service }}-prune.timer"
+  when: not forgejo_runner_dind_enabled
+  notify: Reload systemd
+
+# ── Enabled ───────────────────────────────────────────────────────────────
+- name: Pull the dind image
+  ansible.builtin.command: "docker image pull {{ forgejo_runner_dind_image }}"
+  register: dind_pull
+  changed_when: "'Downloaded newer image' in dind_pull.stdout"
+  when: forgejo_runner_dind_enabled
+
+# Created explicitly rather than left to `docker run`, so that the name is
+# declared here and a later `docker volume ls` has an obvious owner.
+- name: Check for the dind image-store volume
+  ansible.builtin.command: "docker volume inspect {{ forgejo_runner_dind_volume }}"
+  register: dind_volume
+  changed_when: false
+  failed_when: false
+  check_mode: false
+  when: forgejo_runner_dind_enabled
+
+- name: Create the dind image-store volume
+  ansible.builtin.command: "docker volume create {{ forgejo_runner_dind_volume }}"
+  when:
+    - forgejo_runner_dind_enabled
+    - dind_volume.rc | default(1) != 0
+
+- name: Create the dind systemd unit
+  ansible.builtin.template:
+    src: forgejo-dind.service.j2
+    dest: "/etc/systemd/system/{{ forgejo_runner_dind_service }}.service"
+    owner: root
+    group: root
+    mode: '0644'
+  when: forgejo_runner_dind_enabled
+  notify:
+    - Reload systemd
+    - Restart dind
+
+- name: Create the dind prune service
+  ansible.builtin.template:
+    src: forgejo-dind-prune.service.j2
+    dest: "/etc/systemd/system/{{ forgejo_runner_dind_service }}-prune.service"
+    owner: root
+    group: root
+    mode: '0644'
+  when: forgejo_runner_dind_enabled and forgejo_runner_dind_prune_enabled
+  notify: Reload systemd
+
+- name: Create the dind prune timer
+  ansible.builtin.template:
+    src: forgejo-dind-prune.timer.j2
+    dest: "/etc/systemd/system/{{ forgejo_runner_dind_service }}-prune.timer"
+    owner: root
+    group: root
+    mode: '0644'
+  when: forgejo_runner_dind_enabled and forgejo_runner_dind_prune_enabled
+  notify: Reload systemd
+
+- name: Apply pending unit changes before starting dind
+  ansible.builtin.meta: flush_handlers
+
+- name: Enable and start dind
+  ansible.builtin.systemd:
+    name: "{{ forgejo_runner_dind_service }}.service"
+    enabled: yes
+    state: started
+    daemon_reload: yes
+  when: forgejo_runner_dind_enabled
+
+- name: Enable and start the dind prune timer
+  ansible.builtin.systemd:
+    name: "{{ forgejo_runner_dind_service }}-prune.timer"
+    enabled: yes
+    state: started
+    daemon_reload: yes
+  when: forgejo_runner_dind_enabled and forgejo_runner_dind_prune_enabled
+
+# The unit's own ExecStartPost already blocks until the daemon answers, so by
+# here this should be instant. It is asserted anyway, from the same address the
+# runner will use, because "the unit is active" and "the daemon on that port
+# answers" are different claims and it is the second one the runner depends on.
+- name: Verify the dind daemon answers on the address the runner will use
+  ansible.builtin.command: >-
+    docker -H tcp://{{ forgejo_runner_dind_bind_address }}:{{ forgejo_runner_dind_host_port }}
+    version --format {{ '{{.Server.Version}}' }}
+  register: dind_version
+  changed_when: false
+  retries: 10
+  delay: 3
+  until: dind_version.rc == 0
+  when: forgejo_runner_dind_enabled and not ansible_check_mode
+
+- name: Show which daemon the jobs will build on
+  ansible.builtin.debug:
+    msg: >-
+      dind daemon {{ dind_version.stdout }} is up on
+      {{ forgejo_runner_dind_bind_address }}:{{ forgejo_runner_dind_host_port }}
+      (host daemon is untouched); jobs reach it as
+      tcp://{{ forgejo_runner_dind_internal_host }}:{{ forgejo_runner_dind_container_port }}
+  when: forgejo_runner_dind_enabled and not ansible_check_mode

ansible/roles/forgejo_runner/tasks/main.yml

@@ -4,6 +4,9 @@
 # how this refactor was verified against the playbook it replaced.
 - ansible.builtin.import_tasks: prerequisites.yml
 - ansible.builtin.import_tasks: install.yml
+# Before configure.yml and service.yml: config.yml names the dind daemon and
+# forgejo-runner.service Requires= its unit, so it has to exist by then.
+- ansible.builtin.import_tasks: dind.yml
 - ansible.builtin.import_tasks: configure.yml
 - ansible.builtin.import_tasks: service.yml
 - ansible.builtin.import_tasks: healthcheck.yml

ansible/roles/forgejo_runner/tasks/service.yml

@@ -6,6 +6,7 @@
     owner: root
     group: root
     mode: '0644'
+  notify: Restart forgejo-runner
 
 - name: Reload systemd
   systemd:
@@ -17,6 +18,12 @@
     enabled: yes
     state: started
 
+# Before the assert below, not at the end of the play. A config change queues a
+# restart; if that restart ran after the verification, the play would assert
+# that the OLD process is healthy and then replace it unverified.
+- name: Apply a pending restart before checking the runner
+  ansible.builtin.meta: flush_handlers
+
 - name: Verify forgejo-runner is active
   command: systemctl is-active forgejo-runner
   register: runner_active

ansible/roles/forgejo_runner/templates/config.yml.j2

@@ -0,0 +1,122 @@
+# Forgejo Runner configuration - MANAGED BY ANSIBLE (roles/forgejo_runner).
+# Local edits are overwritten on the next run. Change the role instead.
+#
+# Until the Docker-in-Docker work this file was NOT managed: configure.yml ran
+# `forgejo-runner generate-config` once, on first deploy, and never touched the
+# result again. Setting container.docker_host meant either patching YAML in
+# place with a regex - the fragile thing this repo has been bitten by before -
+# or templating the file. So it is templated.
+#
+# Every value below was transcribed from the config.yml live on
+# forgejo-runner-box, which was still stock generate-config output, so applying
+# this with dind disabled is a no-op. Two exceptions, both deliberate:
+#
+#   - runner.envs lost A_TEST_ENV_NAME_1 and A_TEST_ENV_NAME_2. They are
+#     upstream's example placeholders and were being injected into every job on
+#     this runner. Nothing can depend on them.
+#   - the upstream comment blob is gone. The values are unchanged.
+
+{#
+  Two values are assembled rather than written straight out, because each has
+  a dind half and a caller-supplied half and they have to merge, not compete.
+#}
+{% set _envs = dict(forgejo_runner_envs) %}
+{% set _opts = [forgejo_runner_container_options] if forgejo_runner_container_options else [] %}
+{% if forgejo_runner_dind_enabled %}
+{%   set _ = _envs.update({'DOCKER_HOST': 'tcp://' ~ forgejo_runner_dind_internal_host ~ ':' ~ forgejo_runner_dind_container_port}) %}
+{%   set _ = _opts.insert(0, '--add-host=' ~ forgejo_runner_dind_internal_host ~ ':host-gateway') %}
+{% endif %}
+log:
+  level: {{ forgejo_runner_log_level }}
+  job_level: {{ forgejo_runner_log_job_level }}
+
+runner:
+  file: .runner
+  # Concurrent tasks. NOTE, with dind enabled: all capacity slots share ONE
+  # Docker daemon, so two jobs running at once can see and touch each other's
+  # containers, images and volumes through the docker CLI. Forgejo's admin docs
+  # call this out. It is the accepted trade for a single shared runner; drop
+  # this to 1 if untrusted code ever runs here.
+  capacity: {{ forgejo_runner_capacity }}
+{% if _envs %}
+  # Environment for every job. DOCKER_HOST is the half of the dind setup that
+  # the JOB uses: the runner exports it into each job, service and step
+  # container, so `docker build` and `docker compose` inside a job find the
+  # dind daemon instead of a socket that is not there.
+  envs:
+{% for key, value in _envs | dictsort %}
+    {{ key }}: {{ value }}
+{% endfor %}
+{% else %}
+  envs: {}
+{% endif %}
+  env_file: .env
+  timeout: {{ forgejo_runner_job_timeout }}
+  # How long a `systemctl stop/restart` waits for in-flight jobs. It is 3h, so
+  # redeploying while a job is running does NOT return promptly - see the
+  # README's note on when to deploy.
+  shutdown_timeout: {{ forgejo_runner_shutdown_timeout }}
+  insecure: false
+  fetch_timeout: 5s
+  fetch_interval: 2s
+  report_interval: 1s
+  # Empty: the labels in .runner, fixed at registration, are used.
+  labels: []
+
+cache:
+  enabled: {{ 'true' if forgejo_runner_cache_enabled else 'false' }}
+  dir: ""
+  # The address job containers use to reach the runner's cache server, NOT the
+  # address it listens on. Empty means autodetect, which picks this host's
+  # outbound address.
+  #
+  # This is the value most likely to need attention under dind. Job containers
+  # now live inside dind and reach the host by NAT, so the address has to be
+  # one that is routable from in there - this box's LAN address is, and that is
+  # what autodetect returns today. If actions/cache starts silently missing
+  # after enabling dind, look here first: pin forgejo_runner_cache_host to
+  # {{ ansible_default_ipv4.address | default('this host LAN address') }}.
+  host: "{{ forgejo_runner_cache_host }}"
+  port: 0
+  proxy_port: 0
+  external_server: ""
+  secret: ""
+  actions_cache_url_override: ""
+
+container:
+  network: "{{ forgejo_runner_container_network }}"
+  enable_ipv6: false
+  # Stays false. The dind CONTAINER is privileged; job containers have no need
+  # to be, because the daemon they talk to is already running elsewhere.
+  privileged: false
+{% if _opts %}
+  # --add-host resolves the dind hostname inside every job container. Docker
+  # expands `host-gateway` against the daemon that CREATES the container -
+  # which is dind - so it lands on dind's own bridge gateway, where its dockerd
+  # is listening. Without this the DOCKER_HOST above names something no job can
+  # resolve, and jobs fail exactly as if nothing had been configured.
+  options: '{{ _opts | join(" ") }}'
+{% else %}
+  options:
+{% endif %}
+  workdir_parent:
+  valid_volumes: {{ forgejo_runner_valid_volumes | to_json }}
+{% if forgejo_runner_dind_enabled %}
+  # Create job, service and step containers on the dind daemon instead of this
+  # host's. Reached over the published loopback port, because the runner is a
+  # plain process on the host, not a container beside dind.
+  #
+  # Consequence worth knowing: NOTHING a workflow does now touches the host's
+  # Docker daemon. Images are pulled fresh into dind the first time each one is
+  # needed, and service containers declared by a job land in dind alongside the
+  # job container, which is what makes `docker compose` reach them by name.
+  docker_host: 'tcp://{{ forgejo_runner_dind_bind_address }}:{{ forgejo_runner_dind_host_port }}'
+{% else %}
+  # "-" - find a daemon for the runner's own use, and do not expose it to jobs.
+  docker_host: '-'
+{% endif %}
+  force_pull: false
+  force_rebuild: false
+
+host:
+  workdir_parent:

ansible/roles/forgejo_runner/templates/forgejo-dind-prune.service.j2

@@ -0,0 +1,16 @@
+# MANAGED BY ANSIBLE - roles/forgejo_runner
+[Unit]
+Description=Prune unused images and containers inside the Forgejo dind daemon
+After={{ forgejo_runner_dind_service }}.service
+Requires={{ forgejo_runner_dind_service }}.service
+
+[Service]
+Type=oneshot
+# Inside the dind container, against dind's own daemon - NOT the host's. The
+# host daemon still holds the dind image itself and is not touched here.
+#
+# Volumes are excluded on purpose; see forgejo_runner_dind_prune_filter in the
+# role defaults for why a timer is the wrong place to delete those.
+ExecStart=/usr/bin/docker exec {{ forgejo_runner_dind_container }} docker system prune --all --force --filter "{{ forgejo_runner_dind_prune_filter }}"
+StandardOutput=journal
+StandardError=journal

ansible/roles/forgejo_runner/templates/forgejo-dind-prune.timer.j2

@@ -0,0 +1,12 @@
+# MANAGED BY ANSIBLE - roles/forgejo_runner
+[Unit]
+Description=Weekly prune of the Forgejo dind image and container store
+
+[Timer]
+OnCalendar={{ forgejo_runner_dind_prune_on_calendar }}
+# The box may be off at 04:30 on a Sunday; catch up rather than skip a week.
+Persistent=true
+RandomizedDelaySec=900
+
+[Install]
+WantedBy=timers.target

ansible/roles/forgejo_runner/templates/forgejo-dind.service.j2

@@ -0,0 +1,52 @@
+# MANAGED BY ANSIBLE - roles/forgejo_runner
+#
+# The isolated Docker daemon that Forgejo Actions jobs use. It runs ON this
+# host's daemon but is a separate daemon: nothing a workflow does here can see
+# or touch the host's containers, images or volumes.
+[Unit]
+Description=Docker-in-Docker daemon for Forgejo Actions jobs
+Documentation=https://forgejo.org/docs/latest/admin/actions/docker-access/
+After=docker.service
+Requires=docker.service
+
+[Service]
+Type=simple
+Restart=always
+RestartSec=10
+# dind pulls its image on first start and initialises /var/lib/docker; neither
+# is fast on a cold volume, and a deadline here would just flap it.
+TimeoutStartSec=0
+
+# A container left behind by a hard kill would make `docker run --name` fail.
+# Leading "-" so a clean start is not an error.
+ExecStartPre=-/usr/bin/docker rm -f {{ forgejo_runner_dind_container }}
+
+ExecStart=/usr/bin/docker run --rm --name {{ forgejo_runner_dind_container }} \
+    --privileged \
+    --publish {{ forgejo_runner_dind_bind_address }}:{{ forgejo_runner_dind_host_port }}:{{ forgejo_runner_dind_container_port }} \
+    --volume {{ forgejo_runner_dind_volume }}:/var/lib/docker \
+    --env DOCKER_TLS_CERTDIR= \
+    {{ forgejo_runner_dind_image }} \
+    dockerd \
+    --host=tcp://0.0.0.0:{{ forgejo_runner_dind_container_port }} \
+    --host=unix:///var/run/docker.sock \
+    --default-address-pool base={{ forgejo_runner_dind_address_pool }},size={{ forgejo_runner_dind_address_pool_size }}
+
+# Hold the unit in "activating" until the daemon actually answers, so that
+# forgejo-runner.service - which Requires= and After= this one - is never
+# started against a socket that is not up yet. The runner does not retry: with
+# an explicit container.docker_host it fails the connection and the job with it.
+#
+# Bounded by `timeout` rather than a counting loop, so a dind that never comes
+# up FAILS this unit instead of hanging the boot. That is also the thing that
+# catches a bad dockerd flag: the container exits, this never answers, and the
+# unit goes red rather than the runner quietly losing Docker.
+ExecStartPost=/usr/bin/timeout {{ forgejo_runner_dind_ready_timeout }} /bin/sh -c 'until /usr/bin/docker -H tcp://{{ forgejo_runner_dind_bind_address }}:{{ forgejo_runner_dind_host_port }} version >/dev/null 2>&1; do sleep 1; done'
+
+ExecStop=/usr/bin/docker stop --time 30 {{ forgejo_runner_dind_container }}
+
+StandardOutput=journal
+StandardError=journal
+
+[Install]
+WantedBy=multi-user.target

ansible/roles/forgejo_runner/templates/forgejo-runner.service.j2

@@ -1,8 +1,23 @@
+# MANAGED BY ANSIBLE - roles/forgejo_runner
 [Unit]
 Description=Forgejo Runner
 Documentation=https://forgejo.org/docs/latest/admin/actions/
+{% if forgejo_runner_dind_enabled %}
+# forgejo-dind is a hard dependency, not a nicety. With an explicit
+# container.docker_host the runner does not fall back to anything: if that
+# daemon is not answering, every job it picks up fails to create its container.
+# Requires= also means a stopped dind stops the runner, so the failure is one
+# red unit rather than a runner that looks healthy and fails every job.
+#
+# The dind unit stays in "activating" until its daemon answers, so After= here
+# really does mean "after Docker is usable", not just "after the process
+# forked".
+After=docker.service {{ forgejo_runner_dind_service }}.service
+Requires=docker.service {{ forgejo_runner_dind_service }}.service
+{% else %}
 After=docker.service
 Requires=docker.service
+{% endif %}
 
 [Service]
 Type=simple

ansible/roles/forgejo_runner/templates/healthcheck.sh.j2

@@ -38,10 +38,34 @@ main() {
         report false "forgejo-runner is not active"
         exit 1
     fi
+{% if forgejo_runner_dind_enabled %}
+
+    # An active runner is no longer the whole answer. With container.docker_host
+    # pointing at dind, a dead dind means the runner keeps claiming jobs and
+    # fails every one of them at container creation - green unit, red CI. So
+    # check the daemon jobs actually build on, not just that its unit is up:
+    # the unit can be active while dockerd inside it has died.
+    if ! systemctl is-active --quiet {{ forgejo_runner_dind_service }}; then
+        log_message "ERROR: {{ forgejo_runner_dind_service }} is not active"
+        report false "{{ forgejo_runner_dind_service }} is not active"
+        exit 1
+    fi
+
+    if ! docker -H tcp://{{ forgejo_runner_dind_bind_address }}:{{ forgejo_runner_dind_host_port }} version >/dev/null 2>&1; then
+        log_message "ERROR: dind daemon not answering on {{ forgejo_runner_dind_bind_address }}:{{ forgejo_runner_dind_host_port }}"
+        report false "dind daemon not answering"
+        exit 1
+    fi
+
+    log_message "forgejo-runner is active and the dind daemon answers"
+    report true "active"
+    exit 0
+{% else %}
 
     log_message "forgejo-runner is active"
     report true "active"
     exit 0
+{% endif %}
 }
 
 main