From f9e123dfcbc61295b9a9ab3256ccb6b139246679 Mon Sep 17 00:00:00 2001 From: Niek Otten Date: Thu, 25 Jun 2026 14:00:51 +0200 Subject: [PATCH] docs(infra): tighten gitea-actions-gotchas, add local compose (refs #30) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Restructure for scannability: a shared root-cause intro, a quick-reference table (gotcha → fix → where), and consistent Symptom/Why/Fix sections with tighter prose. Documents infra/docker-compose.local.yml as the no-make/Windows path and drops the now-stale "no bind mounts remain" line (the local compose uses them, which is fine locally). Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/runbooks/gitea-actions-gotchas.md | 162 +++++++++++-------------- 1 file changed, 74 insertions(+), 88 deletions(-) diff --git a/docs/runbooks/gitea-actions-gotchas.md b/docs/runbooks/gitea-actions-gotchas.md index 52a88cc..52da86d 100644 --- a/docs/runbooks/gitea-actions-gotchas.md +++ b/docs/runbooks/gitea-actions-gotchas.md @@ -1,114 +1,100 @@ # Gitea Actions gotchas -Known differences between Gitea Actions (our CI) and a plain local run, and the -workarounds we adopted. Referenced by `CLAUDE.md` §8.7 and §15. +How our CI (Gitea Actions on the hosted **`ubuntu-latest`** runner) differs from a +local run, and the workarounds in this repo. Referenced by `CLAUDE.md` §8.7/§15. -## Bind mounts don't reach Compose services on the hosted runner +**One root cause sits under most of this:** the runner executes the job **inside a +container**, so when a step runs `docker compose up`, Compose starts the stack as +**sibling containers** on the host's daemon. Anything that assumes the job and +those containers share a filesystem — or a `localhost` — breaks. -**Symptom.** `make smoke` is green locally but the `compose-smoke` CI job fails -with the OpenZaak init container exiting 1: +| Gotcha | Fix | Lives in | +|---|---|---| +| Bind-mounted config arrives empty | `docker cp` config into external volumes | `infra/seed-config.sh` | +| `docker compose up --wait` is unsupported / flaky | poll health with `docker inspect` | `infra/wait-healthy.sh` | +| `pg_isready` passes before PostGIS is ready | add a `PostGIS_Version()` probe | the db healthchecks | + +--- + +## 1. Bind mounts don't reach the containers + +**Symptom** — green locally, but `compose-smoke` fails with: ``` oz-init-1 | CommandError: Yaml file `/app/setup_configuration/data.yaml` does not exist. ``` -Migrations run fine; only the step that reads a **mounted** file fails. The same -class of failure hits any service that bind-mounts a workspace path — -`nrc-init` (its `data.yaml`), `flowable-init` (the BPMN), `keycloak` (the realm -import dir). +Migrations run fine; only the step that reads a *mounted* file fails. The same +trap hits `nrc-init`, `flowable-init`, and `keycloak`. -**Cause.** The `ubuntu-latest` runner executes the whole job **inside a -container** (`docker.gitea.com/runner-images:ubuntu-latest`). When the job then -runs `docker compose ... up`, Compose talks to the host's Docker daemon and -starts the stack as **sibling containers**. A relative bind mount such as +**Why** — a relative bind mount like `./openzaak/setup_configuration:/app/...` is +resolved by Compose to a path *inside the job container* +(`/workspace/.../setup_configuration`). The daemon then looks for that path on +*its own host*, doesn't find it, and mounts an **empty directory**. (It works on a +runner that executes jobs on the host — which is why moving to `ubuntu-latest` +exposed it.) -```yaml -volumes: - - ./openzaak/setup_configuration:/app/setup_configuration:ro -``` +**Fix** — use the upstream images verbatim (no build) and stream config into +**external named volumes** with `docker cp`, which copies over the Docker API and +so works wherever the daemon runs. `infra/seed-config.sh` creates each volume, +mounts it in a throwaway helper, and copies the files in: -is resolved by Compose to an absolute path **inside the job container** -(`/workspace/eho/register-referentie/infra/openzaak/setup_configuration`). The -daemon then looks for that path on **its own host**, doesn't find it, and -auto-creates an **empty directory** to mount. The container starts with an empty -mount point, so the file appears "missing". - -This is the classic Docker-in-Docker / sibling-container bind-mount trap. It does -not happen on a runner that executes jobs directly on the host (the previous -self-hosted `respellion-linux` setup), which is why switching to `ubuntu-latest` -exposed it. - -**Fix: the upstream images are used verbatim (no build); config is streamed into -external named volumes with `docker cp`.** `infra/seed-config.sh` creates a fixed- -name volume per asset, runs a throwaway helper container that mounts it, and -`docker cp`s the files in. `docker cp` streams bytes over the Docker API, so it -works no matter where the daemon runs (including Docker-in-Docker). The services -then mount those volumes: - -| Asset | External volume | Mounted by → at | +| Asset | Volume | Mounted at | |---|---|---| -| OpenZaak `setup_configuration/data.yaml` | `rr-oz-config` | `oz-init` → `/app/setup_configuration` | -| Keycloak realm exports | `rr-kc-realms` | `keycloak` → `/opt/keycloak/data/import` | -| `workflows/registratie.bpmn` | `rr-fl-bpmn` | `flowable-init` → `/work` | +| OpenZaak `data.yaml` | `rr-oz-config` | `oz-init:/app/setup_configuration` | +| Keycloak realms | `rr-kc-realms` | `keycloak:/opt/keycloak/data/import` | +| `registratie.bpmn` | `rr-fl-bpmn` | `flowable-init:/work` | -The volumes are declared `external: true` with fixed `name:`s so they resolve -identically under docker compose and podman-compose. The seed step (`make` runs -it before every `up`) recreates them fresh each time; `make down` / the -per-service `*-down` targets remove them. Open Notificaties needs no config at all -— `nrc-init` runs migrations only. +The volumes are `external: true` with fixed names, so they resolve identically +under docker compose and podman-compose. `make` seeds before every `up`; `make +down` removes them. (Open Notificaties needs nothing — `nrc-init` migrates only.) -**Two hard constraints drove this design:** +**Consequence — bare `docker compose up` can't self-seed external volumes:** -- Use plain `docker volume create` / `docker run` / `docker cp` — **not** - `docker compose create`, which **podman-compose** (the local dev runtime) does - not implement. -- `docker cp` rather than a bind mount of the source dir, because that bind mount - is exactly what fails on the containerized runner. +- **CI / Linux / macOS:** `make up` or `make smoke` (seed, then start). +- **No-make / Windows:** `infra/docker-compose.local.yml` — a twin stack that + **bind-mounts** the config instead. Bind mounts are fine *locally* because a + local daemon can see your working directory, so + `docker compose -f infra/docker-compose.local.yml up -d` just works. -**Consequence: bare `docker compose up` no longer self-seeds** — the `external` -volumes must be populated first, so use **`make up`** (or `make -up`), which -seeds then starts. CI uses `make smoke`, which does the same. +**Why not the obvious alternatives** -**Why not the alternatives.** +- *Bake config into an image* (incl. an inline Dockerfile) — `docker compose up` + would then work unaided, but it's a build; we wanted the upstream images as-is. +- *Compose `configs:` with inline `content`* — Compose writes a client-side temp + file and bind-mounts it, hitting the exact same problem. +- *A host-executing runner* — bind mounts would work with zero seeding, but it + reintroduces a self-hosted runner and undoes the move to `ubuntu-latest`. -- *Bake into a (possibly inline) image* — clean and portable, but it's a build; - rejected here because the goal was to use the upstream images verbatim. -- *Compose `configs:` with inline `content`* — Compose materialises these as a - temp file on the **client** side and bind-mounts it → same daemon-can't-see-it - problem. -- *A self-hosted runner that runs jobs on the host* — bind mounts would then work - with zero seeding, but it reintroduces a bespoke runner and undoes the move to - the hosted `ubuntu-latest` label. +--- -No bind mounts of these config files remain, so the SELinux `:z`/`:Z` relabel flag -is no longer needed anywhere in `infra/` (named volumes don't need relabeling). +## 2. Readiness: poll health, don't use `--wait` -## Readiness: a portable health poll, not `docker compose up --wait` +`docker compose up --wait` looks ideal but fails us three ways: -The smoke does **not** use `docker compose up --wait`, for three reasons: +- **podman-compose doesn't implement it** (`unrecognized arguments: --wait`) — so + it would break local dev. +- A project-wide `--wait` **treats a one-shot exiting `0` as a failure** unless + something `depends_on` it with `service_completed_successfully`. `flowable-init` + deploys the BPMN and exits with no dependant, so `--wait` fails the moment it + does — last line `container infra-flowable-init-1 exited (0)`. +- The containerized runner **can't reach published host ports**, so an external + `curl localhost:8080/health` can't work either. -- **podman-compose doesn't implement `--wait`** (`unrecognized arguments: - --wait`), so it would break local dev. -- A whole-project `--wait` **fails when a one-shot with no - `service_completed_successfully` dependant exits** — `flowable-init` deploys - the BPMN and exits 0, which `--wait` treats as the project failing (symptom: - last compose line `container infra-flowable-init-1 exited (0)`). -- The containerized CI runner **can't reach published host ports**, so an - external `curl localhost:8080/health` doesn't work either. +**Fix** — `infra/wait-healthy.sh` polls each durable service (`openzaak nrc-web +acl bff`, listed as `WAIT_SVCS` in the `Makefile`) with `docker ps` + `docker +inspect '{{.State.Health.Status}}'` until it reports `healthy`. It uses only +primitives both runtimes support, reads the **in-container** healthcheck (no host +port needed), and ignores the one-shots (they only need to have run). +`WAIT_TIMEOUT` defaults to 420 s — enough for the cold OpenZaak migrate (~90 s) +plus app start. -**Fix.** `infra/wait-healthy.sh` polls each durable, health-checked service -(`openzaak nrc-web acl bff` — `WAIT_SVCS` in the `Makefile`) with `docker ps` + -`docker inspect '{{.State.Health.Status}}'`, waiting for `healthy`. That uses -only primitives both docker compose and podman-compose support, reads the -in-container healthcheck (no host port needed), and ignores the one-shots (they -only need to have run). `WAIT_TIMEOUT` (default 420 s) covers the cold -OpenZaak migrate (~90 s) plus app start. +--- -## PostGIS readiness vs. `pg_isready` +## 3. `pg_isready` passes before PostGIS is ready -`pg_isready` reports the server is accepting connections as soon as the TCP port -is open — **before** the `postgis/postgis` image has finished running its -`CREATE EXTENSION postgis` init scripts. An init container that starts migrating -in that window can fail on a missing PostGIS. The db healthchecks therefore add a -`SELECT PostGIS_Version()` probe so dependents wait for the extension, not just -the port. +`pg_isready` succeeds as soon as the TCP port is open — *before* the +`postgis/postgis` image has finished running `CREATE EXTENSION postgis`. An init +container that starts migrating in that window can fail on a missing PostGIS. So +the db healthchecks add a `SELECT PostGIS_Version()` probe, making dependents wait +for the extension, not just the port.