ci(infra): Gitea Actions CI pipeline + full-stack compose smoke (closes #30) #50
@@ -1,114 +1,100 @@
|
|||||||
# Gitea Actions gotchas
|
# Gitea Actions gotchas
|
||||||
|
|
||||||
Known differences between Gitea Actions (our CI) and a plain local run, and the
|
How our CI (Gitea Actions on the hosted **`ubuntu-latest`** runner) differs from a
|
||||||
workarounds we adopted. Referenced by `CLAUDE.md` §8.7 and §15.
|
local run, and the workarounds in this repo. Referenced by `CLAUDE.md` §8.7/§15.
|
||||||
|
|
||||||
## Bind mounts don't reach Compose services on the hosted runner
|
**One root cause sits under most of this:** the runner executes the job **inside a
|
||||||
|
container**, so when a step runs `docker compose up`, Compose starts the stack as
|
||||||
|
**sibling containers** on the host's daemon. Anything that assumes the job and
|
||||||
|
those containers share a filesystem — or a `localhost` — breaks.
|
||||||
|
|
||||||
**Symptom.** `make smoke` is green locally but the `compose-smoke` CI job fails
|
| Gotcha | Fix | Lives in |
|
||||||
with the OpenZaak init container exiting 1:
|
|---|---|---|
|
||||||
|
| Bind-mounted config arrives empty | `docker cp` config into external volumes | `infra/seed-config.sh` |
|
||||||
|
| `docker compose up --wait` is unsupported / flaky | poll health with `docker inspect` | `infra/wait-healthy.sh` |
|
||||||
|
| `pg_isready` passes before PostGIS is ready | add a `PostGIS_Version()` probe | the db healthchecks |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Bind mounts don't reach the containers
|
||||||
|
|
||||||
|
**Symptom** — green locally, but `compose-smoke` fails with:
|
||||||
|
|
||||||
```
|
```
|
||||||
oz-init-1 | CommandError: Yaml file `/app/setup_configuration/data.yaml` does not exist.
|
oz-init-1 | CommandError: Yaml file `/app/setup_configuration/data.yaml` does not exist.
|
||||||
```
|
```
|
||||||
|
|
||||||
Migrations run fine; only the step that reads a **mounted** file fails. The same
|
Migrations run fine; only the step that reads a *mounted* file fails. The same
|
||||||
class of failure hits any service that bind-mounts a workspace path —
|
trap hits `nrc-init`, `flowable-init`, and `keycloak`.
|
||||||
`nrc-init` (its `data.yaml`), `flowable-init` (the BPMN), `keycloak` (the realm
|
|
||||||
import dir).
|
|
||||||
|
|
||||||
**Cause.** The `ubuntu-latest` runner executes the whole job **inside a
|
**Why** — a relative bind mount like `./openzaak/setup_configuration:/app/...` is
|
||||||
container** (`docker.gitea.com/runner-images:ubuntu-latest`). When the job then
|
resolved by Compose to a path *inside the job container*
|
||||||
runs `docker compose ... up`, Compose talks to the host's Docker daemon and
|
(`/workspace/.../setup_configuration`). The daemon then looks for that path on
|
||||||
starts the stack as **sibling containers**. A relative bind mount such as
|
*its own host*, doesn't find it, and mounts an **empty directory**. (It works on a
|
||||||
|
runner that executes jobs on the host — which is why moving to `ubuntu-latest`
|
||||||
|
exposed it.)
|
||||||
|
|
||||||
```yaml
|
**Fix** — use the upstream images verbatim (no build) and stream config into
|
||||||
volumes:
|
**external named volumes** with `docker cp`, which copies over the Docker API and
|
||||||
- ./openzaak/setup_configuration:/app/setup_configuration:ro
|
so works wherever the daemon runs. `infra/seed-config.sh` creates each volume,
|
||||||
```
|
mounts it in a throwaway helper, and copies the files in:
|
||||||
|
|
||||||
is resolved by Compose to an absolute path **inside the job container**
|
| Asset | Volume | Mounted at |
|
||||||
(`/workspace/eho/register-referentie/infra/openzaak/setup_configuration`). The
|
|
||||||
daemon then looks for that path on **its own host**, doesn't find it, and
|
|
||||||
auto-creates an **empty directory** to mount. The container starts with an empty
|
|
||||||
mount point, so the file appears "missing".
|
|
||||||
|
|
||||||
This is the classic Docker-in-Docker / sibling-container bind-mount trap. It does
|
|
||||||
not happen on a runner that executes jobs directly on the host (the previous
|
|
||||||
self-hosted `respellion-linux` setup), which is why switching to `ubuntu-latest`
|
|
||||||
exposed it.
|
|
||||||
|
|
||||||
**Fix: the upstream images are used verbatim (no build); config is streamed into
|
|
||||||
external named volumes with `docker cp`.** `infra/seed-config.sh` creates a fixed-
|
|
||||||
name volume per asset, runs a throwaway helper container that mounts it, and
|
|
||||||
`docker cp`s the files in. `docker cp` streams bytes over the Docker API, so it
|
|
||||||
works no matter where the daemon runs (including Docker-in-Docker). The services
|
|
||||||
then mount those volumes:
|
|
||||||
|
|
||||||
| Asset | External volume | Mounted by → at |
|
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| OpenZaak `setup_configuration/data.yaml` | `rr-oz-config` | `oz-init` → `/app/setup_configuration` |
|
| OpenZaak `data.yaml` | `rr-oz-config` | `oz-init:/app/setup_configuration` |
|
||||||
| Keycloak realm exports | `rr-kc-realms` | `keycloak` → `/opt/keycloak/data/import` |
|
| Keycloak realms | `rr-kc-realms` | `keycloak:/opt/keycloak/data/import` |
|
||||||
| `workflows/registratie.bpmn` | `rr-fl-bpmn` | `flowable-init` → `/work` |
|
| `registratie.bpmn` | `rr-fl-bpmn` | `flowable-init:/work` |
|
||||||
|
|
||||||
The volumes are declared `external: true` with fixed `name:`s so they resolve
|
The volumes are `external: true` with fixed names, so they resolve identically
|
||||||
identically under docker compose and podman-compose. The seed step (`make` runs
|
under docker compose and podman-compose. `make` seeds before every `up`; `make
|
||||||
it before every `up`) recreates them fresh each time; `make down` / the
|
down` removes them. (Open Notificaties needs nothing — `nrc-init` migrates only.)
|
||||||
per-service `*-down` targets remove them. Open Notificaties needs no config at all
|
|
||||||
— `nrc-init` runs migrations only.
|
|
||||||
|
|
||||||
**Two hard constraints drove this design:**
|
**Consequence — bare `docker compose up` can't self-seed external volumes:**
|
||||||
|
|
||||||
- Use plain `docker volume create` / `docker run` / `docker cp` — **not**
|
- **CI / Linux / macOS:** `make up` or `make smoke` (seed, then start).
|
||||||
`docker compose create`, which **podman-compose** (the local dev runtime) does
|
- **No-make / Windows:** `infra/docker-compose.local.yml` — a twin stack that
|
||||||
not implement.
|
**bind-mounts** the config instead. Bind mounts are fine *locally* because a
|
||||||
- `docker cp` rather than a bind mount of the source dir, because that bind mount
|
local daemon can see your working directory, so
|
||||||
is exactly what fails on the containerized runner.
|
`docker compose -f infra/docker-compose.local.yml up -d` just works.
|
||||||
|
|
||||||
**Consequence: bare `docker compose up` no longer self-seeds** — the `external`
|
**Why not the obvious alternatives**
|
||||||
volumes must be populated first, so use **`make up`** (or `make <svc>-up`), which
|
|
||||||
seeds then starts. CI uses `make smoke`, which does the same.
|
|
||||||
|
|
||||||
**Why not the alternatives.**
|
- *Bake config into an image* (incl. an inline Dockerfile) — `docker compose up`
|
||||||
|
would then work unaided, but it's a build; we wanted the upstream images as-is.
|
||||||
|
- *Compose `configs:` with inline `content`* — Compose writes a client-side temp
|
||||||
|
file and bind-mounts it, hitting the exact same problem.
|
||||||
|
- *A host-executing runner* — bind mounts would work with zero seeding, but it
|
||||||
|
reintroduces a self-hosted runner and undoes the move to `ubuntu-latest`.
|
||||||
|
|
||||||
- *Bake into a (possibly inline) image* — clean and portable, but it's a build;
|
---
|
||||||
rejected here because the goal was to use the upstream images verbatim.
|
|
||||||
- *Compose `configs:` with inline `content`* — Compose materialises these as a
|
|
||||||
temp file on the **client** side and bind-mounts it → same daemon-can't-see-it
|
|
||||||
problem.
|
|
||||||
- *A self-hosted runner that runs jobs on the host* — bind mounts would then work
|
|
||||||
with zero seeding, but it reintroduces a bespoke runner and undoes the move to
|
|
||||||
the hosted `ubuntu-latest` label.
|
|
||||||
|
|
||||||
No bind mounts of these config files remain, so the SELinux `:z`/`:Z` relabel flag
|
## 2. Readiness: poll health, don't use `--wait`
|
||||||
is no longer needed anywhere in `infra/` (named volumes don't need relabeling).
|
|
||||||
|
|
||||||
## Readiness: a portable health poll, not `docker compose up --wait`
|
`docker compose up --wait` looks ideal but fails us three ways:
|
||||||
|
|
||||||
The smoke does **not** use `docker compose up --wait`, for three reasons:
|
- **podman-compose doesn't implement it** (`unrecognized arguments: --wait`) — so
|
||||||
|
it would break local dev.
|
||||||
|
- A project-wide `--wait` **treats a one-shot exiting `0` as a failure** unless
|
||||||
|
something `depends_on` it with `service_completed_successfully`. `flowable-init`
|
||||||
|
deploys the BPMN and exits with no dependant, so `--wait` fails the moment it
|
||||||
|
does — last line `container infra-flowable-init-1 exited (0)`.
|
||||||
|
- The containerized runner **can't reach published host ports**, so an external
|
||||||
|
`curl localhost:8080/health` can't work either.
|
||||||
|
|
||||||
- **podman-compose doesn't implement `--wait`** (`unrecognized arguments:
|
**Fix** — `infra/wait-healthy.sh` polls each durable service (`openzaak nrc-web
|
||||||
--wait`), so it would break local dev.
|
acl bff`, listed as `WAIT_SVCS` in the `Makefile`) with `docker ps` + `docker
|
||||||
- A whole-project `--wait` **fails when a one-shot with no
|
inspect '{{.State.Health.Status}}'` until it reports `healthy`. It uses only
|
||||||
`service_completed_successfully` dependant exits** — `flowable-init` deploys
|
primitives both runtimes support, reads the **in-container** healthcheck (no host
|
||||||
the BPMN and exits 0, which `--wait` treats as the project failing (symptom:
|
port needed), and ignores the one-shots (they only need to have run).
|
||||||
last compose line `container infra-flowable-init-1 exited (0)`).
|
`WAIT_TIMEOUT` defaults to 420 s — enough for the cold OpenZaak migrate (~90 s)
|
||||||
- The containerized CI runner **can't reach published host ports**, so an
|
plus app start.
|
||||||
external `curl localhost:8080/health` doesn't work either.
|
|
||||||
|
|
||||||
**Fix.** `infra/wait-healthy.sh` polls each durable, health-checked service
|
---
|
||||||
(`openzaak nrc-web acl bff` — `WAIT_SVCS` in the `Makefile`) with `docker ps` +
|
|
||||||
`docker inspect '{{.State.Health.Status}}'`, waiting for `healthy`. That uses
|
|
||||||
only primitives both docker compose and podman-compose support, reads the
|
|
||||||
in-container healthcheck (no host port needed), and ignores the one-shots (they
|
|
||||||
only need to have run). `WAIT_TIMEOUT` (default 420 s) covers the cold
|
|
||||||
OpenZaak migrate (~90 s) plus app start.
|
|
||||||
|
|
||||||
## PostGIS readiness vs. `pg_isready`
|
## 3. `pg_isready` passes before PostGIS is ready
|
||||||
|
|
||||||
`pg_isready` reports the server is accepting connections as soon as the TCP port
|
`pg_isready` succeeds as soon as the TCP port is open — *before* the
|
||||||
is open — **before** the `postgis/postgis` image has finished running its
|
`postgis/postgis` image has finished running `CREATE EXTENSION postgis`. An init
|
||||||
`CREATE EXTENSION postgis` init scripts. An init container that starts migrating
|
container that starts migrating in that window can fail on a missing PostGIS. So
|
||||||
in that window can fail on a missing PostGIS. The db healthchecks therefore add a
|
the db healthchecks add a `SELECT PostGIS_Version()` probe, making dependents wait
|
||||||
`SELECT PostGIS_Version()` probe so dependents wait for the extension, not just
|
for the extension, not just the port.
|
||||||
the port.
|
|
||||||
|
|||||||
Reference in New Issue
Block a user