diff --git a/ansible/host_vars/forgejo_runner_local/main.yml b/ansible/host_vars/forgejo_runner_local/main.yml index 6e852ce..1fae71b 100644 --- a/ansible/host_vars/forgejo_runner_local/main.yml +++ b/ansible/host_vars/forgejo_runner_local/main.yml @@ -10,3 +10,8 @@ # supervision and would be noise. monitored_services: - forgejo-runner + # The isolated Docker daemon that Actions jobs build on. Listed separately + # from forgejo-runner because they fail separately and the interesting case + # is the asymmetric one: dind down, runner up, every job failing while the + # runner's own unit stays green. + - forgejo-dind diff --git a/ansible/roles/forgejo_runner/README.md b/ansible/roles/forgejo_runner/README.md index e7f761b..6765e7f 100644 --- a/ansible/roles/forgejo_runner/README.md +++ b/ansible/roles/forgejo_runner/README.md @@ -57,3 +57,100 @@ never ran. It is not a monitoring task — it is the deployment checking its own work — and the deprecation banner swept it up by mistake. It is ungated here, which means a runner that fails to start now fails the play instead of deploying "successfully" in silence. + +## Docker-in-Docker for jobs + +`forgejo_runner_dind_enabled: true` gives Actions jobs a Docker daemon of their +own — a `docker:dind` container on this host's daemon, isolated from it — so +`docker build` and `docker compose` work inside a workflow. + +It follows Forgejo's own [Utilizing Docker within +Actions](https://forgejo.org/docs/latest/admin/actions/docker-access/) → +Docker-in-Docker, which is **three** parts, not one: + +| | what it does | where it applies | +|---|---|---| +| `forgejo-dind.service` | a `docker:dind` container, published on `127.0.0.1:2376` only | the host | +| `container.docker_host` | `tcp://127.0.0.1:2376` — the runner creates job/service/step containers **on dind** | the runner process | +| `runner.envs.DOCKER_HOST` + `container.options` | `tcp://forgejo-dind.docker.internal:2375`, resolved by `--add-host=…:host-gateway` | inside each job container | + +The third row is the one that is easy to miss, and leaving it out fails in a +way that looks identical to not having configured anything. `docker_host` only +decides **where the job container is created**; for a `tcp://` value Forgejo +mounts nothing into that container, so the job still has no daemon. The two +addresses differ because they are used from different places — the runner is a +process on the host and reaches dind on loopback; a job container is *inside* +dind, where `127.0.0.1` is itself. `host-gateway` is expanded by the daemon +that creates the container, which is dind, so it lands on dind's own bridge +gateway where its `dockerd` is listening. + +### What changes for jobs + +- Job containers, service containers and anything a job starts all live on the + dind daemon. A `postgres` service and a `docker compose` stack are siblings + there, so they reach each other by name — which is what makes end-to-end + tests work. +- **Nothing a workflow does can touch this host's Docker daemon any more.** + That is the point of dind over `automount`. +- dind has its own image store. Every image is pulled again the first time it + is needed. It is a named volume (`forgejo-dind-lib`), so it survives the + container being recreated, and `forgejo-dind-prune.timer` drops weekly what + has gone a week unused. +- Bind mounts inside a job's own `docker compose` resolve on the **dind** + filesystem, not the job container's. A compose file that bind-mounts the + checked-out repo will not see it. Named volumes and `build:` contexts are + fine. This is inherent to dind, not to this role. + +### Things to know before enabling + +- **This runner is shared.** `arbret`, `arbret-biz`, `arbretdos` and + `arbret-ops` build here too, and `container.docker_host` is per *runner*, not + per repo — every one of them moves onto dind. Their workflows do not change; + their next run pulls images into a cold cache. +- **`capacity` is 4 and dind is one daemon.** Concurrent jobs can see and touch + each other's containers and images through the Docker CLI. Forgejo's docs + call this out. Fine for a single-tenant runner; set `capacity: 1` if + untrusted code ever runs here. +- **The dind container is `--privileged`**, which is why this belongs on a + dedicated CI VM and not beside anything that matters. `forgejo-runner-box` + runs the runner and nothing else. +- **The daemon has no TLS and no authentication.** It is published on + `127.0.0.1` only, so reaching it means already having a shell on this box — + but it does mean any local user is root in a privileged container. Do not + move `forgejo_runner_dind_bind_address` off loopback. +- **The `runner` user is still in the host `docker` group**, left over from + when the runner used the host daemon. With dind it no longer needs it. Worth + removing, deliberately, as its own change. +- **`cache.host` is autodetected**, and job containers now reach the runner's + cache server across dind's NAT. Autodetect returns this box's LAN address, + which is routable from in there. If `actions/cache` starts silently missing, + pin `forgejo_runner_cache_host` — that is the first place to look. +- **Deploy when the runner is idle.** `shutdown_timeout` is 3h, so restarting + it mid-job waits for the job. + +### Turning it off + +Set `forgejo_runner_dind_enabled: false` and re-run. The role stops and +disables the units, removes them, and puts `container.docker_host` back to +`"-"` — the exact configuration this host ran before. The image and the +`forgejo-dind-lib` volume are left alone: they are a cache, they hold no +secrets, and deleting them on a toggle is not the role's call. + +## `config.yml` is templated now + +It used to be `forgejo-runner generate-config`, run once on first deploy and +never touched again — so the repo had no say in anything inside it, including +the setting this feature needs. It is now `templates/config.yml.j2`. + +Every value in it was transcribed from the file live on `forgejo-runner-box` +and diffed key-by-key against it: with dind disabled, 30 of 30 keys are +identical. Two deliberate differences: + +- `runner.envs` no longer carries `A_TEST_ENV_NAME_1` / `A_TEST_ENV_NAME_2`, + upstream's example placeholders, which were being injected into every job. +- the upstream comment blob is gone; the values are not. + +Because the file is managed, a change to it now restarts the runner +(`handlers/main.yml`). Previously a config change could not happen at all; the +failure mode where it lands on disk and the running process never picks it up +is one this repo has hit before. diff --git a/ansible/roles/forgejo_runner/defaults/main.yml b/ansible/roles/forgejo_runner/defaults/main.yml index aba6b61..36f4bc1 100644 --- a/ansible/roles/forgejo_runner/defaults/main.yml +++ b/ansible/roles/forgejo_runner/defaults/main.yml @@ -11,6 +11,27 @@ forgejo_runner_dir: "/opt/forgejo-runner" forgejo_runner_config_path: "{{ forgejo_runner_dir }}/config.yml" forgejo_runner_labels: "docker:docker://node:20-bookworm,ubuntu-latest:docker://node:20-bookworm,ubuntu-22.04:docker://node:20-bookworm,ubuntu-24.04:docker://node:20-bookworm" +# --- config.yml contents ---------------------------------------------------- +# These reproduce, exactly, the stock `generate-config` output this host has +# been running since first deploy. They are variables now only because the file +# became a template; changing one of them changes the box. +forgejo_runner_log_level: info +forgejo_runner_log_job_level: info +forgejo_runner_capacity: 4 +forgejo_runner_job_timeout: "3h" +forgejo_runner_shutdown_timeout: "3h" +forgejo_runner_cache_enabled: true +# Empty = autodetect this host's outbound address. See the note in the template +# before changing it; under dind this is the value that decides whether +# actions/cache works. +forgejo_runner_cache_host: "" +forgejo_runner_container_network: "" +forgejo_runner_valid_volumes: [] +# Extra job environment and extra `docker run` options for job containers. +# Both MERGE with what dind adds rather than replacing it. +forgejo_runner_envs: {} +forgejo_runner_container_options: "" + # The Forgejo instance this runner registers with. forgejo_instance_url: "https://forgejo.contrapeso.xyz" # forgejo_runner_registration_token comes from the vault. @@ -39,3 +60,78 @@ healthcheck_push_url: "" # Bearer token for the Gatus external endpoint. Required whenever a push URL # is set: Gatus rejects an unauthenticated push with 401. healthcheck_push_token: "" + +# --- Docker-in-Docker for job containers ------------------------------------ +# Off by default. The role default is the SAFE state: `container.docker_host` +# stays "-", which is what this host has always run, and jobs get no Docker at +# all. services/forgejo-runner/deploy_forgejo_runner_playbook.yml turns it on. +# +# The shape is the one in Forgejo's own admin docs, "Utilizing Docker within +# Actions" -> Docker-in-Docker, and it is three moving parts, not one: +# +# 1. a docker:dind container on the HOST daemon, published on loopback only; +# 2. container.docker_host, which points the RUNNER PROCESS at that daemon, +# so job/service/step containers are created INSIDE dind; +# 3. runner.envs.DOCKER_HOST plus a container.options --add-host, which is +# what lets the JOB reach the same daemon from inside. +# +# Step 3 is not optional and not a duplicate of step 2. The two addresses are +# different because they are used from different places: the runner talks to +# 127.0.0.1 on the host, and a job container inside dind cannot - its own +# 127.0.0.1 is itself. Setting only docker_host gets you job containers that +# run in dind but still cannot find a daemon, which looks exactly like the bug +# we are fixing. See the README. +forgejo_runner_dind_enabled: false + +forgejo_runner_dind_container: forgejo-dind +# Already present on forgejo-runner-box, and the tag the project's own workflow +# refers to. A floating 28.x picks up patch releases on container recreation. +forgejo_runner_dind_image: "docker:28-dind" +forgejo_runner_dind_service: forgejo-dind + +# Loopback ONLY. This daemon has no TLS and no authentication, so anything that +# can open the port is root inside a privileged container. 127.0.0.1 keeps that +# to processes on this box; do not move it to 0.0.0.0 or the LAN address. +# +# The number is 2376 because the upstream docs use it. 2376 is conventionally +# the TLS port and this is PLAINTEXT - the port number promises nothing. +forgejo_runner_dind_bind_address: "127.0.0.1" +forgejo_runner_dind_host_port: 2376 +forgejo_runner_dind_container_port: 2375 + +# The name a JOB container uses to reach the dind daemon. Resolved by an +# --add-host ... :host-gateway on the job container, which Docker expands in +# the context of the daemon that CREATES the container - dind - so it lands on +# dind's own inner bridge gateway, where its dockerd is listening. That is why +# a name is used here and not an address: the address is only knowable from +# inside dind, at container-creation time. +forgejo_runner_dind_internal_host: "forgejo-dind.docker.internal" + +# Networks for the containers dind creates. act_runner makes a fresh network +# per job, so these multiply, and Docker's stock pool reaches into 192.168.x - +# the same space as this LAN. A job container that allocated 192.168.1.0/24 +# would lose the route to every box on it, including this one. Pinning the pool +# well clear of the LAN removes that entirely. +forgejo_runner_dind_address_pool: "172.20.0.0/14" +forgejo_runner_dind_address_pool_size: 24 + +# How long the dind unit waits for its daemon to answer before failing. Cold +# start pulls the image and initialises /var/lib/docker. +forgejo_runner_dind_ready_timeout: 180 + +# dind keeps its own /var/lib/docker. Without a volume it is wiped every time +# the container is recreated, and every image every workflow uses - the +# playwright image alone is 3.3GB - is pulled again from scratch. +forgejo_runner_dind_volume: forgejo-dind-lib + +# ...which then grows without bound, so it is pruned. `until=168h` keeps +# anything used in the last week, which is the cache that makes the volume +# worth having, and drops the rest. +# +# Volumes are deliberately NOT pruned: `until` does not apply to them, so +# --volumes here would be "delete every volume not currently attached", and a +# workflow's named volume between two jobs is not an obvious thing to destroy +# on a timer. Watch disk and prune those by hand. +forgejo_runner_dind_prune_enabled: true +forgejo_runner_dind_prune_on_calendar: "Sun 04:30" +forgejo_runner_dind_prune_filter: "until=168h" diff --git a/ansible/roles/forgejo_runner/handlers/main.yml b/ansible/roles/forgejo_runner/handlers/main.yml new file mode 100644 index 0000000..e367495 --- /dev/null +++ b/ansible/roles/forgejo_runner/handlers/main.yml @@ -0,0 +1,31 @@ +--- +- name: Reload systemd + ansible.builtin.systemd: + daemon_reload: yes + +- name: Restart dind + ansible.builtin.systemd: + name: "{{ forgejo_runner_dind_service }}.service" + state: restarted + daemon_reload: yes + # In check mode the unit file was never written, so systemd has never heard + # of it and this fails on "Could not find the requested service" - an + # artefact of the dry run, not a problem with the run. Skipping it keeps + # `--check --diff` able to reach the config.yml diff, which is the part of a + # dry run worth reading. + when: not ansible_check_mode + +# The role had no handlers at all before, because it managed no config: the +# runner's config.yml was generated once on first deploy and never touched +# again, so there was nothing a change could need to restart. Now that +# config.yml is a template, a change to it that did NOT restart the runner +# would be the worst of both - the file on disk right and the running process +# still on the old settings. That failure mode has bitten this repo before. +# +# Note the runner's shutdown_timeout is 3h: a restart while a job is in flight +# waits for it, up to systemd's stop timeout. Deploy when the runner is idle. +- name: Restart forgejo-runner + ansible.builtin.systemd: + name: forgejo-runner + state: restarted + daemon_reload: yes diff --git a/ansible/roles/forgejo_runner/tasks/configure.yml b/ansible/roles/forgejo_runner/tasks/configure.yml index ab96d52..d7e7944 100644 --- a/ansible/roles/forgejo_runner/tasks/configure.yml +++ b/ansible/roles/forgejo_runner/tasks/configure.yml @@ -1,23 +1,32 @@ --- -- name: Check if config already exists - stat: - path: "{{ forgejo_runner_config_path }}" - register: config_stat - -- name: Generate default config - shell: "{{ forgejo_runner_bin_path }} generate-config > {{ forgejo_runner_config_path }}" - args: - chdir: "{{ forgejo_runner_dir }}" - when: not config_stat.stat.exists - -- name: Set config file ownership - file: - path: "{{ forgejo_runner_config_path }}" +# config.yml is a TEMPLATE now, not a one-shot `generate-config`. +# +# Before this, the file was generated on first deploy and never touched again, +# so the repo had no say in anything inside it - including container.docker_host, +# which is the setting that decides whether jobs can use Docker at all. The +# alternative was patching keys in place with a regex, which in a YAML file +# with repeated key names under different parents is the kind of thing that +# works until it silently does not. +# +# owner/group/mode are runner:runner 0644, read off the live box, matching what +# `generate-config` plus the old chown produced. Nothing secret is in here; the +# registration token lives in .runner. +- name: Write the forgejo-runner configuration + ansible.builtin.template: + src: config.yml.j2 + dest: "{{ forgejo_runner_config_path }}" owner: "{{ forgejo_runner_user }}" group: "{{ forgejo_runner_user }}" - when: not config_stat.stat.exists + mode: '0644' + # A malformed config is a runner that will not start. Catch it here, while + # the old file is still in place, rather than after it has been replaced. + validate: "python3 -c \"import yaml,sys; yaml.safe_load(open(sys.argv[1]))\" %s" + notify: Restart forgejo-runner -# ── 6. Register runner ───────────────────────────────────────────── +# ── Register runner ──────────────────────────────────────────────────────── +# Registration is separate from configuration and happens once. The labels are +# frozen into .runner at this point; changing forgejo_runner_labels later does +# not move them, which is why the label set is left exactly as it was. - name: Check if runner is already registered stat: path: "{{ forgejo_runner_dir }}/.runner" @@ -40,4 +49,3 @@ owner: "{{ forgejo_runner_user }}" group: "{{ forgejo_runner_user }}" when: not runner_stat.stat.exists - diff --git a/ansible/roles/forgejo_runner/tasks/dind.yml b/ansible/roles/forgejo_runner/tasks/dind.yml new file mode 100644 index 0000000..f14c070 --- /dev/null +++ b/ansible/roles/forgejo_runner/tasks/dind.yml @@ -0,0 +1,140 @@ +--- +# The isolated Docker daemon jobs build against. +# +# Runs BEFORE configure.yml and service.yml on purpose: config.yml names this +# daemon's address, and forgejo-runner.service Requires= this unit. Creating it +# after either would mean one run where the runner points at nothing. + +# ── Disabled: undo it, do not just skip it ───────────────────────────────── +# A flag that only ever adds is not a flag. If dind is switched off the runner +# goes back to container.docker_host "-" a few tasks from now, and a dind still +# running would be a privileged container nothing uses. The volume and the +# image are deliberately left: they are the image cache, they hold no secrets, +# and destroying them on a toggle is not this role's call. +- name: Find the dind unit when dind is disabled + ansible.builtin.stat: + path: "/etc/systemd/system/{{ forgejo_runner_dind_service }}.service" + register: dind_unit_leftover + when: not forgejo_runner_dind_enabled + +- name: Stop and disable dind + ansible.builtin.systemd: + name: "{{ item }}" + state: stopped + enabled: no + loop: + - "{{ forgejo_runner_dind_service }}.service" + - "{{ forgejo_runner_dind_service }}-prune.timer" + failed_when: false + when: + - not forgejo_runner_dind_enabled + - dind_unit_leftover.stat.exists | default(false) + +- name: Remove the dind units + ansible.builtin.file: + path: "/etc/systemd/system/{{ item }}" + state: absent + loop: + - "{{ forgejo_runner_dind_service }}.service" + - "{{ forgejo_runner_dind_service }}-prune.service" + - "{{ forgejo_runner_dind_service }}-prune.timer" + when: not forgejo_runner_dind_enabled + notify: Reload systemd + +# ── Enabled ─────────────────────────────────────────────────────────────── +- name: Pull the dind image + ansible.builtin.command: "docker image pull {{ forgejo_runner_dind_image }}" + register: dind_pull + changed_when: "'Downloaded newer image' in dind_pull.stdout" + when: forgejo_runner_dind_enabled + +# Created explicitly rather than left to `docker run`, so that the name is +# declared here and a later `docker volume ls` has an obvious owner. +- name: Check for the dind image-store volume + ansible.builtin.command: "docker volume inspect {{ forgejo_runner_dind_volume }}" + register: dind_volume + changed_when: false + failed_when: false + check_mode: false + when: forgejo_runner_dind_enabled + +- name: Create the dind image-store volume + ansible.builtin.command: "docker volume create {{ forgejo_runner_dind_volume }}" + when: + - forgejo_runner_dind_enabled + - dind_volume.rc | default(1) != 0 + +- name: Create the dind systemd unit + ansible.builtin.template: + src: forgejo-dind.service.j2 + dest: "/etc/systemd/system/{{ forgejo_runner_dind_service }}.service" + owner: root + group: root + mode: '0644' + when: forgejo_runner_dind_enabled + notify: + - Reload systemd + - Restart dind + +- name: Create the dind prune service + ansible.builtin.template: + src: forgejo-dind-prune.service.j2 + dest: "/etc/systemd/system/{{ forgejo_runner_dind_service }}-prune.service" + owner: root + group: root + mode: '0644' + when: forgejo_runner_dind_enabled and forgejo_runner_dind_prune_enabled + notify: Reload systemd + +- name: Create the dind prune timer + ansible.builtin.template: + src: forgejo-dind-prune.timer.j2 + dest: "/etc/systemd/system/{{ forgejo_runner_dind_service }}-prune.timer" + owner: root + group: root + mode: '0644' + when: forgejo_runner_dind_enabled and forgejo_runner_dind_prune_enabled + notify: Reload systemd + +- name: Apply pending unit changes before starting dind + ansible.builtin.meta: flush_handlers + +- name: Enable and start dind + ansible.builtin.systemd: + name: "{{ forgejo_runner_dind_service }}.service" + enabled: yes + state: started + daemon_reload: yes + when: forgejo_runner_dind_enabled + +- name: Enable and start the dind prune timer + ansible.builtin.systemd: + name: "{{ forgejo_runner_dind_service }}-prune.timer" + enabled: yes + state: started + daemon_reload: yes + when: forgejo_runner_dind_enabled and forgejo_runner_dind_prune_enabled + +# The unit's own ExecStartPost already blocks until the daemon answers, so by +# here this should be instant. It is asserted anyway, from the same address the +# runner will use, because "the unit is active" and "the daemon on that port +# answers" are different claims and it is the second one the runner depends on. +- name: Verify the dind daemon answers on the address the runner will use + ansible.builtin.command: >- + docker -H tcp://{{ forgejo_runner_dind_bind_address }}:{{ forgejo_runner_dind_host_port }} + version --format {{ '{{.Server.Version}}' }} + register: dind_version + changed_when: false + retries: 10 + delay: 3 + until: dind_version.rc == 0 + when: forgejo_runner_dind_enabled and not ansible_check_mode + +- name: Show which daemon the jobs will build on + ansible.builtin.debug: + msg: >- + dind daemon {{ dind_version.stdout }} is up on + {{ forgejo_runner_dind_bind_address }}:{{ forgejo_runner_dind_host_port }} + (host daemon is untouched); jobs reach it as + tcp://{{ forgejo_runner_dind_internal_host }}:{{ forgejo_runner_dind_container_port }} + when: forgejo_runner_dind_enabled and not ansible_check_mode diff --git a/ansible/roles/forgejo_runner/tasks/main.yml b/ansible/roles/forgejo_runner/tasks/main.yml index aa71001..5023e59 100644 --- a/ansible/roles/forgejo_runner/tasks/main.yml +++ b/ansible/roles/forgejo_runner/tasks/main.yml @@ -4,6 +4,9 @@ # how this refactor was verified against the playbook it replaced. - ansible.builtin.import_tasks: prerequisites.yml - ansible.builtin.import_tasks: install.yml +# Before configure.yml and service.yml: config.yml names the dind daemon and +# forgejo-runner.service Requires= its unit, so it has to exist by then. +- ansible.builtin.import_tasks: dind.yml - ansible.builtin.import_tasks: configure.yml - ansible.builtin.import_tasks: service.yml - ansible.builtin.import_tasks: healthcheck.yml diff --git a/ansible/roles/forgejo_runner/tasks/service.yml b/ansible/roles/forgejo_runner/tasks/service.yml index f971e52..11e8784 100644 --- a/ansible/roles/forgejo_runner/tasks/service.yml +++ b/ansible/roles/forgejo_runner/tasks/service.yml @@ -6,6 +6,7 @@ owner: root group: root mode: '0644' + notify: Restart forgejo-runner - name: Reload systemd systemd: @@ -17,6 +18,12 @@ enabled: yes state: started +# Before the assert below, not at the end of the play. A config change queues a +# restart; if that restart ran after the verification, the play would assert +# that the OLD process is healthy and then replace it unverified. +- name: Apply a pending restart before checking the runner + ansible.builtin.meta: flush_handlers + - name: Verify forgejo-runner is active command: systemctl is-active forgejo-runner register: runner_active diff --git a/ansible/roles/forgejo_runner/templates/config.yml.j2 b/ansible/roles/forgejo_runner/templates/config.yml.j2 new file mode 100644 index 0000000..b0a5c1e --- /dev/null +++ b/ansible/roles/forgejo_runner/templates/config.yml.j2 @@ -0,0 +1,122 @@ +# Forgejo Runner configuration - MANAGED BY ANSIBLE (roles/forgejo_runner). +# Local edits are overwritten on the next run. Change the role instead. +# +# Until the Docker-in-Docker work this file was NOT managed: configure.yml ran +# `forgejo-runner generate-config` once, on first deploy, and never touched the +# result again. Setting container.docker_host meant either patching YAML in +# place with a regex - the fragile thing this repo has been bitten by before - +# or templating the file. So it is templated. +# +# Every value below was transcribed from the config.yml live on +# forgejo-runner-box, which was still stock generate-config output, so applying +# this with dind disabled is a no-op. Two exceptions, both deliberate: +# +# - runner.envs lost A_TEST_ENV_NAME_1 and A_TEST_ENV_NAME_2. They are +# upstream's example placeholders and were being injected into every job on +# this runner. Nothing can depend on them. +# - the upstream comment blob is gone. The values are unchanged. + +{# + Two values are assembled rather than written straight out, because each has + a dind half and a caller-supplied half and they have to merge, not compete. +#} +{% set _envs = dict(forgejo_runner_envs) %} +{% set _opts = [forgejo_runner_container_options] if forgejo_runner_container_options else [] %} +{% if forgejo_runner_dind_enabled %} +{% set _ = _envs.update({'DOCKER_HOST': 'tcp://' ~ forgejo_runner_dind_internal_host ~ ':' ~ forgejo_runner_dind_container_port}) %} +{% set _ = _opts.insert(0, '--add-host=' ~ forgejo_runner_dind_internal_host ~ ':host-gateway') %} +{% endif %} +log: + level: {{ forgejo_runner_log_level }} + job_level: {{ forgejo_runner_log_job_level }} + +runner: + file: .runner + # Concurrent tasks. NOTE, with dind enabled: all capacity slots share ONE + # Docker daemon, so two jobs running at once can see and touch each other's + # containers, images and volumes through the docker CLI. Forgejo's admin docs + # call this out. It is the accepted trade for a single shared runner; drop + # this to 1 if untrusted code ever runs here. + capacity: {{ forgejo_runner_capacity }} +{% if _envs %} + # Environment for every job. DOCKER_HOST is the half of the dind setup that + # the JOB uses: the runner exports it into each job, service and step + # container, so `docker build` and `docker compose` inside a job find the + # dind daemon instead of a socket that is not there. + envs: +{% for key, value in _envs | dictsort %} + {{ key }}: {{ value }} +{% endfor %} +{% else %} + envs: {} +{% endif %} + env_file: .env + timeout: {{ forgejo_runner_job_timeout }} + # How long a `systemctl stop/restart` waits for in-flight jobs. It is 3h, so + # redeploying while a job is running does NOT return promptly - see the + # README's note on when to deploy. + shutdown_timeout: {{ forgejo_runner_shutdown_timeout }} + insecure: false + fetch_timeout: 5s + fetch_interval: 2s + report_interval: 1s + # Empty: the labels in .runner, fixed at registration, are used. + labels: [] + +cache: + enabled: {{ 'true' if forgejo_runner_cache_enabled else 'false' }} + dir: "" + # The address job containers use to reach the runner's cache server, NOT the + # address it listens on. Empty means autodetect, which picks this host's + # outbound address. + # + # This is the value most likely to need attention under dind. Job containers + # now live inside dind and reach the host by NAT, so the address has to be + # one that is routable from in there - this box's LAN address is, and that is + # what autodetect returns today. If actions/cache starts silently missing + # after enabling dind, look here first: pin forgejo_runner_cache_host to + # {{ ansible_default_ipv4.address | default('this host LAN address') }}. + host: "{{ forgejo_runner_cache_host }}" + port: 0 + proxy_port: 0 + external_server: "" + secret: "" + actions_cache_url_override: "" + +container: + network: "{{ forgejo_runner_container_network }}" + enable_ipv6: false + # Stays false. The dind CONTAINER is privileged; job containers have no need + # to be, because the daemon they talk to is already running elsewhere. + privileged: false +{% if _opts %} + # --add-host resolves the dind hostname inside every job container. Docker + # expands `host-gateway` against the daemon that CREATES the container - + # which is dind - so it lands on dind's own bridge gateway, where its dockerd + # is listening. Without this the DOCKER_HOST above names something no job can + # resolve, and jobs fail exactly as if nothing had been configured. + options: '{{ _opts | join(" ") }}' +{% else %} + options: +{% endif %} + workdir_parent: + valid_volumes: {{ forgejo_runner_valid_volumes | to_json }} +{% if forgejo_runner_dind_enabled %} + # Create job, service and step containers on the dind daemon instead of this + # host's. Reached over the published loopback port, because the runner is a + # plain process on the host, not a container beside dind. + # + # Consequence worth knowing: NOTHING a workflow does now touches the host's + # Docker daemon. Images are pulled fresh into dind the first time each one is + # needed, and service containers declared by a job land in dind alongside the + # job container, which is what makes `docker compose` reach them by name. + docker_host: 'tcp://{{ forgejo_runner_dind_bind_address }}:{{ forgejo_runner_dind_host_port }}' +{% else %} + # "-" - find a daemon for the runner's own use, and do not expose it to jobs. + docker_host: '-' +{% endif %} + force_pull: false + force_rebuild: false + +host: + workdir_parent: diff --git a/ansible/roles/forgejo_runner/templates/forgejo-dind-prune.service.j2 b/ansible/roles/forgejo_runner/templates/forgejo-dind-prune.service.j2 new file mode 100644 index 0000000..571466e --- /dev/null +++ b/ansible/roles/forgejo_runner/templates/forgejo-dind-prune.service.j2 @@ -0,0 +1,16 @@ +# MANAGED BY ANSIBLE - roles/forgejo_runner +[Unit] +Description=Prune unused images and containers inside the Forgejo dind daemon +After={{ forgejo_runner_dind_service }}.service +Requires={{ forgejo_runner_dind_service }}.service + +[Service] +Type=oneshot +# Inside the dind container, against dind's own daemon - NOT the host's. The +# host daemon still holds the dind image itself and is not touched here. +# +# Volumes are excluded on purpose; see forgejo_runner_dind_prune_filter in the +# role defaults for why a timer is the wrong place to delete those. +ExecStart=/usr/bin/docker exec {{ forgejo_runner_dind_container }} docker system prune --all --force --filter "{{ forgejo_runner_dind_prune_filter }}" +StandardOutput=journal +StandardError=journal diff --git a/ansible/roles/forgejo_runner/templates/forgejo-dind-prune.timer.j2 b/ansible/roles/forgejo_runner/templates/forgejo-dind-prune.timer.j2 new file mode 100644 index 0000000..a2c2a7b --- /dev/null +++ b/ansible/roles/forgejo_runner/templates/forgejo-dind-prune.timer.j2 @@ -0,0 +1,12 @@ +# MANAGED BY ANSIBLE - roles/forgejo_runner +[Unit] +Description=Weekly prune of the Forgejo dind image and container store + +[Timer] +OnCalendar={{ forgejo_runner_dind_prune_on_calendar }} +# The box may be off at 04:30 on a Sunday; catch up rather than skip a week. +Persistent=true +RandomizedDelaySec=900 + +[Install] +WantedBy=timers.target diff --git a/ansible/roles/forgejo_runner/templates/forgejo-dind.service.j2 b/ansible/roles/forgejo_runner/templates/forgejo-dind.service.j2 new file mode 100644 index 0000000..2caf419 --- /dev/null +++ b/ansible/roles/forgejo_runner/templates/forgejo-dind.service.j2 @@ -0,0 +1,52 @@ +# MANAGED BY ANSIBLE - roles/forgejo_runner +# +# The isolated Docker daemon that Forgejo Actions jobs use. It runs ON this +# host's daemon but is a separate daemon: nothing a workflow does here can see +# or touch the host's containers, images or volumes. +[Unit] +Description=Docker-in-Docker daemon for Forgejo Actions jobs +Documentation=https://forgejo.org/docs/latest/admin/actions/docker-access/ +After=docker.service +Requires=docker.service + +[Service] +Type=simple +Restart=always +RestartSec=10 +# dind pulls its image on first start and initialises /var/lib/docker; neither +# is fast on a cold volume, and a deadline here would just flap it. +TimeoutStartSec=0 + +# A container left behind by a hard kill would make `docker run --name` fail. +# Leading "-" so a clean start is not an error. +ExecStartPre=-/usr/bin/docker rm -f {{ forgejo_runner_dind_container }} + +ExecStart=/usr/bin/docker run --rm --name {{ forgejo_runner_dind_container }} \ + --privileged \ + --publish {{ forgejo_runner_dind_bind_address }}:{{ forgejo_runner_dind_host_port }}:{{ forgejo_runner_dind_container_port }} \ + --volume {{ forgejo_runner_dind_volume }}:/var/lib/docker \ + --env DOCKER_TLS_CERTDIR= \ + {{ forgejo_runner_dind_image }} \ + dockerd \ + --host=tcp://0.0.0.0:{{ forgejo_runner_dind_container_port }} \ + --host=unix:///var/run/docker.sock \ + --default-address-pool base={{ forgejo_runner_dind_address_pool }},size={{ forgejo_runner_dind_address_pool_size }} + +# Hold the unit in "activating" until the daemon actually answers, so that +# forgejo-runner.service - which Requires= and After= this one - is never +# started against a socket that is not up yet. The runner does not retry: with +# an explicit container.docker_host it fails the connection and the job with it. +# +# Bounded by `timeout` rather than a counting loop, so a dind that never comes +# up FAILS this unit instead of hanging the boot. That is also the thing that +# catches a bad dockerd flag: the container exits, this never answers, and the +# unit goes red rather than the runner quietly losing Docker. +ExecStartPost=/usr/bin/timeout {{ forgejo_runner_dind_ready_timeout }} /bin/sh -c 'until /usr/bin/docker -H tcp://{{ forgejo_runner_dind_bind_address }}:{{ forgejo_runner_dind_host_port }} version >/dev/null 2>&1; do sleep 1; done' + +ExecStop=/usr/bin/docker stop --time 30 {{ forgejo_runner_dind_container }} + +StandardOutput=journal +StandardError=journal + +[Install] +WantedBy=multi-user.target diff --git a/ansible/roles/forgejo_runner/templates/forgejo-runner.service.j2 b/ansible/roles/forgejo_runner/templates/forgejo-runner.service.j2 index d3db25d..6c3e468 100644 --- a/ansible/roles/forgejo_runner/templates/forgejo-runner.service.j2 +++ b/ansible/roles/forgejo_runner/templates/forgejo-runner.service.j2 @@ -1,8 +1,23 @@ +# MANAGED BY ANSIBLE - roles/forgejo_runner [Unit] Description=Forgejo Runner Documentation=https://forgejo.org/docs/latest/admin/actions/ +{% if forgejo_runner_dind_enabled %} +# forgejo-dind is a hard dependency, not a nicety. With an explicit +# container.docker_host the runner does not fall back to anything: if that +# daemon is not answering, every job it picks up fails to create its container. +# Requires= also means a stopped dind stops the runner, so the failure is one +# red unit rather than a runner that looks healthy and fails every job. +# +# The dind unit stays in "activating" until its daemon answers, so After= here +# really does mean "after Docker is usable", not just "after the process +# forked". +After=docker.service {{ forgejo_runner_dind_service }}.service +Requires=docker.service {{ forgejo_runner_dind_service }}.service +{% else %} After=docker.service Requires=docker.service +{% endif %} [Service] Type=simple diff --git a/ansible/roles/forgejo_runner/templates/healthcheck.sh.j2 b/ansible/roles/forgejo_runner/templates/healthcheck.sh.j2 index 9aa5ac1..181714c 100644 --- a/ansible/roles/forgejo_runner/templates/healthcheck.sh.j2 +++ b/ansible/roles/forgejo_runner/templates/healthcheck.sh.j2 @@ -38,10 +38,34 @@ main() { report false "forgejo-runner is not active" exit 1 fi +{% if forgejo_runner_dind_enabled %} + + # An active runner is no longer the whole answer. With container.docker_host + # pointing at dind, a dead dind means the runner keeps claiming jobs and + # fails every one of them at container creation - green unit, red CI. So + # check the daemon jobs actually build on, not just that its unit is up: + # the unit can be active while dockerd inside it has died. + if ! systemctl is-active --quiet {{ forgejo_runner_dind_service }}; then + log_message "ERROR: {{ forgejo_runner_dind_service }} is not active" + report false "{{ forgejo_runner_dind_service }} is not active" + exit 1 + fi + + if ! docker -H tcp://{{ forgejo_runner_dind_bind_address }}:{{ forgejo_runner_dind_host_port }} version >/dev/null 2>&1; then + log_message "ERROR: dind daemon not answering on {{ forgejo_runner_dind_bind_address }}:{{ forgejo_runner_dind_host_port }}" + report false "dind daemon not answering" + exit 1 + fi + + log_message "forgejo-runner is active and the dind daemon answers" + report true "active" + exit 0 +{% else %} log_message "forgejo-runner is active" report true "active" exit 0 +{% endif %} } main diff --git a/ansible/services/forgejo-runner/SETUP.md b/ansible/services/forgejo-runner/SETUP.md index 5cb2a3f..38f6248 100644 --- a/ansible/services/forgejo-runner/SETUP.md +++ b/ansible/services/forgejo-runner/SETUP.md @@ -28,3 +28,36 @@ ansible-playbook ansible/services/forgejo-runner/deploy_forgejo_runner_playbook. 3. ~~In Uptime Kuma: the `forgejo-runner-healthcheck` push monitor should be receiving pings~~ *(Uptime Kuma was decommissioned 2026-09-11 — this check no longer applies. The healthcheck timer still runs on the host but pushes nowhere. See `archive/uptime_kuma/`.)* + +## Docker in CI jobs + +Jobs get their own Docker daemon (a `docker:dind` sidecar), not this host's. +Enabled by `forgejo_runner_dind_enabled` in the deploy playbook; the mechanism, +the trade-offs and how to turn it off are in `ansible/roles/forgejo_runner/README.md`. + +A workflow needs no special configuration — `DOCKER_HOST` is already set in +every job container: + +```yaml +jobs: + build: + runs-on: docker + steps: + - uses: actions/checkout@v7 + - run: docker build -t my-app . + - run: docker compose up -d --wait +``` + +Verifying after a deploy: + +1. `systemctl status forgejo-dind` — active, and `docker -H tcp://127.0.0.1:2376 version` + answers from the box. +2. `docker ps` on the host shows `forgejo-dind` and, during a run, **no** job + containers: they are inside dind now. `docker exec forgejo-dind docker ps` + shows those. +3. A workflow step running `docker version` reports the dind daemon. + +One caveat worth knowing before writing an end-to-end job: a `docker compose` +bind mount resolves on the dind filesystem, not in the job container, so a +compose file that mounts the checked-out repo will not see it. Named volumes +and `build:` contexts work normally. diff --git a/ansible/services/forgejo-runner/deploy_forgejo_runner_playbook.yml b/ansible/services/forgejo-runner/deploy_forgejo_runner_playbook.yml index 04081ff..52c123d 100644 --- a/ansible/services/forgejo-runner/deploy_forgejo_runner_playbook.yml +++ b/ansible/services/forgejo-runner/deploy_forgejo_runner_playbook.yml @@ -9,5 +9,26 @@ # replaces it sets the same variable. healthcheck_push_url: "https://{{ subdomains.gatus }}.{{ root_domain }}/api/v1/endpoints/probe_forgejo-runner/external" healthcheck_push_token: "{{ gatus_push_tokens[inventory_hostname] }}" + + # Give Actions jobs a Docker daemon of their own. + # + # Requested for counterweight/jar-lid-archive, which needs `docker build` + # and `docker compose` in CI. Enabled here rather than in the role defaults + # so that the default stays the safe one and this is a single, visible, + # revertible line. + # + # forgejo-runner-box is the right host for it: a dedicated CI VM on nodito + # that runs the runner and nothing else, which is what "not production" + # asks for — a privileged container is not something to put next to a + # service anyone depends on. + # + # Read before flipping this: the runner is SHARED. arbret, arbret-biz, + # arbretdos and arbret-ops all build here, and far more often than + # jar-lid-archive does. container.docker_host is a per-runner setting, not + # a per-repo one, so this moves every one of those repos' job containers + # off the host daemon and into dind. Nothing about their workflows changes, + # but their first run after this pulls its images again from an empty + # cache. See the role README for what to watch. + forgejo_runner_dind_enabled: true roles: - forgejo_runner