## What & why Two changes, made and verified together on a real cluster. **S-24 / #25 — a Helm chart for the platform.** One chart, `infra/helm/big-reference`, whose `values.yaml` is a near-literal transcription of `infra/docker-compose.yml`, rendered by three generic templates (Deployment, Job, Service) over a `workloads` map. Adding a service is a values edit. `make k8s-lint` renders and schema-checks the whole stack without a cluster. The issue asked for a *sketch*; this is deployed and verified end to end (see below), which is more than it asked for — the part it asked for that is **not** here is the production-posture write-up (HA, secrets, backup), see Known gaps. **#166 — Caddy replaces nginx in the portals.** nginx resolves a variable `proxy_pass` upstream itself, using only the `resolver` directive and never `/etc/resolv.conf`'s search domains. That had cost two workarounds in one script: rewriting the resolver address for rootless podman, and injecting a full FQDN so the bare `bff` name could resolve on Kubernetes. Caddy dials per request through the system resolver, so `reverse_proxy bff:8080` works on every engine unchanged; `apps/portal-nginx-resolver.sh` and the chart's `BFF_HOST` env are deleted. Closes #25 Closes #166 ## Definition of Done - [x] Linked Gitea issue (above). - [x] Failing test committed before the implementation — twice: the Caddyfile contract test before the Caddyfiles, `make k8s-lint` before the chart. - [x] Implementation makes the test pass. - [x] Conventional Commits referencing the issues (`refs #25` / `refs #166`). - [ ] CI green — awaiting the run on this PR (`make k8s-lint`, `dotnet format` and the new unit self-check pass locally; the compose e2e and mutation lanes are CI's). - [ ] `docker compose up` from a fresh clone reaches green health checks within 3 minutes — the portal images were rebuilt and verified standalone, but a full `make up` run has not been done on this branch. Please confirm in review or let CI's smoke test speak. - [x] Docs updated — `docs/runbooks/kubernetes-talos.md` (new), `frontend-decisions.md`, `demo-script.md`, and the docs that named nginx. - [x] ADR added — ADR-0033 (chart) and ADR-0034 (Caddy). - [ ] Demo note in `docs/demo-script.md` — not added: the deployment target is not a user-visible slice, and the Caddy swap is invisible to the demo script beyond the wording fix included here. ## How it was verified Brought up from scratch on a single-node Talos v1.14.0 VM (6 vCPU / 10 GB, virtio disk) under virt-manager: **29 pods ready and four bootstrap Jobs complete in under three minutes, zero restarts**, using ~4.4 GB of the VM's 10 GB. - Full Common Ground path: portal Caddy → BFF → domain → Flowable → ACL → OpenZaak + Objecten → NRC → event-subscriber → projection → public register (`INGEDIEND`, reference matching the submitted registration). - Werkbak read with an MFA'd medewerker token → 200. - The browser flow driven with Playwright against `http://localhost:30140`: secure context, `crypto.subtle` present, Keycloak form reached, login completed, **no console errors**. - Routing checked against a stub BFF: SPA fallback serves deep links, each portal proxies its own groups, and a portal does *not* proxy a neighbour's group. ## Notes for reviewers Three bugs this shook out, each fixed at the cause rather than the symptom: 1. **`command` vs `args`.** Compose's `command:` replaces the image CMD; Kubernetes' replaces the ENTRYPOINT. Transcribing one to the other broke every upstream image that relies on its entrypoint — postgres refused to run as root, Keycloak tried to exec `start-dev`. The chart now `fail`s at render time on `command`. 2. **Concurrent migrations.** Both `/setup_configuration.sh` and `/start.sh` run `manage.py migrate`; compose serialises them with `depends_on`, Kubernetes has no such edge, so the init Job and its web pod raced (`relation "zgw_consumers_service" already exists`). The four Django services now do both steps in order in the web pod — which also deletes four workloads. 3. **`emptyDir` databases are wiped by any pod-template change.** `make k8s-reseed` now also restarts `event-subscriber` and `projection-api`, which create the projection schema on start and otherwise keep writing to a schema-less database. Known gaps / follow-ups: - **Secrets.** `values.yaml` carries the dev credentials in plain text (`admin/admin`, the ZGW client secret, the two Objecten tokens) and the chart has no `Secret` objects. Fine for a laptop demo, and exactly what #25's "production posture" ADR should address — I suggest a follow-up issue rather than stretching this PR. - **No CI gate for the chart yet.** `make k8s-lint` exists but is not wired into `.gitea/workflows/ci.yaml`, and nothing enforces that the chart and the compose file stay in step. Worth a small follow-up. - **This is two slices in one PR.** They were built and verified together and the diff is entangled (the chart was written against Caddy from the start), so splitting now would mean re-creating an nginx-shaped chart to throw away. Happy to split if you'd rather. - **Rebased onto #161** (merged as #165) rather than merged, to keep the history linear. One conflict, in the `unit:` target where both branches add a self-check line — resolved by keeping both. #161's `infra/host-browser.yml` arrived with `/usr/share/nginx/html/config.json` and is fixed to `/usr/share/caddy/` inside the `feat(portals)` commit, so no commit on this branch leaves that overlay pointing at a path the images no longer have.Reviewed-on: #167
This commit was merged in pull request #167.
This commit is contained in:
@@ -18,7 +18,7 @@ Three forces shape the mechanism:
|
||||
|
||||
- **Nothing notifies anyone.** The trigger lives in Flowable. The domain does not publish
|
||||
task events, and there is no bus between the domain and the BFF.
|
||||
- **The BFF is stateless** and sits behind each portal's nginx.
|
||||
- **The BFF is stateless** and sits behind each portal's reverse proxy.
|
||||
- **This is the repo's first live-updating view**, so the choice sets a precedent.
|
||||
|
||||
## Decision
|
||||
@@ -40,7 +40,7 @@ werkbak is readable at all.
|
||||
Neither buys freshness here, because **nothing notifies the BFF either**:
|
||||
|
||||
- **SSE** (`text/event-stream`) would mean a new streaming endpoint whose handler polls the
|
||||
domain and forwards diffs — the same latency, plus connection lifecycle, nginx
|
||||
domain and forwards diffs — the same latency, plus connection lifecycle, proxy
|
||||
buffering, and auth on a long-lived connection.
|
||||
- **WebSocket/SignalR** adds a dependency (CLAUDE.md §13) and makes the BFF stateful and
|
||||
sticky-session-bound. A genuine push path would *also* need the domain to publish task
|
||||
|
||||
@@ -0,0 +1,161 @@
|
||||
# ADR-0033: Kubernetes deployment is one values-driven Helm chart, not a chart per service
|
||||
|
||||
- **Status:** Accepted
|
||||
- **Date:** 2026-09-04
|
||||
- **Deciders:** Respellion engineering
|
||||
- **Slice:** _(none yet — raised directly as a deployment-target request; see
|
||||
"Process note" at the end)_
|
||||
|
||||
## Context
|
||||
|
||||
The stack is defined once, in `infra/docker-compose.yml`: 30-odd containers made of six
|
||||
upstream Common Ground modules (OpenZaak, Open Notificaties, Objecten, Objecttypen,
|
||||
Keycloak, Flowable), their databases and workers, five .NET services, four portals, six
|
||||
one-shot bootstrap containers, and an observability backplane (off by default here). Compose is the
|
||||
CI-canonical stack: `make verify` and every `verify-*` script drive it.
|
||||
|
||||
We now also want the stack on Kubernetes — first target a **single-node Talos VM on a
|
||||
laptop**. Four properties of this particular stack shape the answer:
|
||||
|
||||
- **The upstream images are used verbatim** and read their configuration from a mounted
|
||||
directory (`setup_configuration/data.yaml`, Keycloak realm exports, BPMN/DMN). Compose
|
||||
streams those files into external volumes (`infra/seed-config.sh`) because bind mounts
|
||||
don't reach sibling containers on the CI runner. Kubernetes needs the same files as
|
||||
ConfigMaps — from *somewhere*.
|
||||
- **Django's `URLValidator` rejects single-label hosts.** Compose works around it by
|
||||
handing the ACL and the seeds a container *IP* (ADR-0009, ADR-0020, ADR-0029, and the
|
||||
`objecten.local` network alias). In Kubernetes a Service FQDN is already multi-label, so
|
||||
the workaround has a natural replacement — but the hosts have to line up exactly, since
|
||||
Objecten reflects the request Host into the URLs it publishes to NRC.
|
||||
- **The OIDC issuer must be one string** for both the browser and the BFF (ADR-0010).
|
||||
`infra/host-browser.yml` already solved this for a host browser: pin `KC_HOSTNAME`, keep
|
||||
backchannel discovery in-cluster, and mount a `config.json` per portal.
|
||||
- **Nothing here is highly available.** One replica of everything, on one node.
|
||||
|
||||
## Decision
|
||||
|
||||
**One chart — `infra/helm/big-reference` — whose `values.yaml` is a near-literal
|
||||
transcription of the compose file, rendered by three generic templates (Deployment, Job,
|
||||
Service) over a `workloads` map.** Adding a service is a values edit.
|
||||
|
||||
Consequences of that shape, each chosen deliberately:
|
||||
|
||||
- **Config files are not copied into the chart.** `infra/helm/seed-configmaps.sh` creates
|
||||
the ConfigMaps from the files that already live in the repo — the Kubernetes sibling of
|
||||
`infra/seed-config.sh`. The chart therefore needs `make k8s-seed` before `helm install`,
|
||||
which is the same two-step dance compose already has.
|
||||
- **Bootstrap one-shots become Jobs, with no ordering mechanism.** Every one is idempotent
|
||||
(ADR-0020); each waits for the TCP ports it needs via a busybox init container and
|
||||
Kubernetes retries the rest. `make k8s-reseed` re-runs them.
|
||||
- **The four Django services apply their own `setup_configuration`** —
|
||||
`args: [sh, -c, "/setup_configuration.sh && exec /start.sh"]` — instead of getting a
|
||||
separate `*-init` Job like compose. Both of those image scripts run
|
||||
`manage.py migrate`, and compose serialises them with
|
||||
`depends_on: service_completed_successfully`; Kubernetes has no such edge, so a Job and
|
||||
its web pod migrate the same database concurrently and Django dies with
|
||||
*"relation zgw_consumers_service already exists"*. Running the two steps in order inside
|
||||
the one container leaves exactly one migrator per database, and deletes four workloads.
|
||||
- **`args`, never `command`.** Compose's `command:` replaces the image's CMD; Kubernetes'
|
||||
`command:` replaces its ENTRYPOINT. Transcribing one to the other silently broke every
|
||||
upstream image that relies on its entrypoint — postgres ran as root and refused to
|
||||
start, Keycloak tried to exec `start-dev` as a binary. The chart now `fail`s at render
|
||||
time if a workload sets `command`, because the symptom (a crashloop three layers down)
|
||||
is nothing like the cause.
|
||||
- **Published ports are NodePorts.** No ingress controller, no LoadBalancer, no TLS. The
|
||||
four portals are the exception in *use*, not in wiring: PKCE needs `crypto.subtle`, which
|
||||
browsers expose only in a secure context, so a portal has to be reached over `localhost`
|
||||
(`make k8s-portals` forwards them) or eventually over HTTPS. `.Values.host` is therefore
|
||||
"the address the browser uses", not "the node's address" — it pins Keycloak's issuer and
|
||||
each portal's `config.json`, and both must agree with the URL bar (ADR-0010).
|
||||
- **Databases are `emptyDir` by default**, so the stack comes up on a cluster with no CSI
|
||||
driver; setting `persistence.storageClass` switches every database to a PVC.
|
||||
- **Only two hosts become FQDNs** — OpenZaak (for the ACL and the zaaktype seed) and
|
||||
Objecten (for the ACL's register writes), the two that Django validates as URLs.
|
||||
Everything else keeps the short compose service name, because the upstream
|
||||
`setup_configuration` files name those and Objecten matches an objecttype URL against the
|
||||
one it was configured with. The portals used to be a third case — nginx's `resolver` never
|
||||
appends search domains, so the bare `bff` upstream could not resolve on Kubernetes — which
|
||||
ADR-0034 removed by serving them with Caddy, whose resolver honours `/etc/resolv.conf`.
|
||||
- **Compose stays CI-canonical.** The chart is a second deployment target, not a
|
||||
replacement; the acceptance, verify and e2e lanes are unchanged.
|
||||
|
||||
### Alternatives considered
|
||||
|
||||
- **A chart per service, or an umbrella of 30 subcharts.** The conventional layout, and
|
||||
roughly 1,500 lines of near-identical YAML for a stack where 28 of 30 workloads are
|
||||
"one pod, one image, some env". It buys independent versioning we don't want (the stack
|
||||
is demoed as a whole) and costs the eye-diffability against the compose file that keeps
|
||||
the two stacks honest.
|
||||
- **`kompose convert`.** One-shot generation, no ongoing artefact to maintain — but it
|
||||
drops exactly the parts that carry the design (init ordering, the config volumes, the
|
||||
issuer pinning) and produces output nobody owns.
|
||||
- **Bitnami PostgreSQL/Redis subcharts.** Six more dependencies (CLAUDE.md §13) and a
|
||||
second way of expressing the same three-line database.
|
||||
- **ingress-nginx with hostname routing.** Needs a controller, `/etc/hosts` entries and a
|
||||
matching issuer host; NodePorts need none of it and reuse the mechanism
|
||||
`infra/host-browser.yml` already proves.
|
||||
- **A registry on the laptop** (the obvious home for images built there). Talos cannot
|
||||
side-load an image, so a registry is required either way — but reaching one on the host
|
||||
means opening an inbound port on firewalld's `libvirt` zone, which needs root, and
|
||||
pushing to it over plain HTTP means an `insecure-registries` entry in the Docker daemon,
|
||||
which needs root again. `infra/helm/registry.yaml` runs the registry *in* the cluster on
|
||||
a NodePort instead: pushing laptop → node is outbound and unfiltered, the node pulls from
|
||||
its own NodePort, and `docker save | crane push --insecure` needs no daemon
|
||||
configuration. Cost: one more (throwaway, `emptyDir`) workload, and a re-push if its pod
|
||||
is replaced.
|
||||
- **Helm hooks (`pre-install`/`post-install`) for bootstrap ordering.** Hooks run after
|
||||
`--wait`, which would deadlock: OpenZaak's readiness needs the migrations that the hook
|
||||
is supposed to run. Idempotent Jobs plus retries need no such sequencing.
|
||||
|
||||
- ponytail ceiling: single-node assumptions are baked in — one replica per workload,
|
||||
`Recreate` rollouts, ReadWriteOnce volumes, no PodDisruptionBudgets, no resource
|
||||
requests or limits (a laptop VM schedules everything or nothing), plain HTTP.
|
||||
Upgrade path for a real cluster: add requests/limits per workload (the field is already
|
||||
passed through), swap NodePorts for an Ingress with TLS, and give the databases a real
|
||||
StorageClass — none of which changes the workload graph.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Positive**
|
||||
|
||||
- One file to read to see what the cluster runs, and it lines up with the compose file
|
||||
line for line.
|
||||
- The compose IP workarounds disappear: cluster DNS supplies multi-label hosts.
|
||||
- `make k8s-lint` renders and schema-checks the whole stack without a cluster.
|
||||
- The config inputs have exactly one home (the repo) for both stacks — no fork to drift.
|
||||
|
||||
**Negative / costs**
|
||||
|
||||
- A second deployment description to keep in step with compose. Nothing enforces that
|
||||
today; a drift check belongs in CI (follow-up).
|
||||
- `helm install` alone is not enough — the ConfigMaps must be seeded first, and a missing
|
||||
one surfaces as `ContainerCreating`, not as a clear error.
|
||||
- Generic templates mean a values typo can render valid-but-wrong YAML; `k8s-lint` catches
|
||||
schema errors, not intent.
|
||||
- The verify/e2e lanes do not run against the chart, so the Kubernetes path is verified by
|
||||
hand (docs/runbooks/kubernetes-talos.md §5) rather than by CI.
|
||||
- The chart deviates from compose in four places now (args, self-configuring Django pods,
|
||||
FQDN hosts, NodePorts). Each is forced by the platform and commented where it appears,
|
||||
but it is four more things that can drift.
|
||||
|
||||
## Coupling rules touched (CLAUDE.md §8)
|
||||
|
||||
None. The chart deploys the same graph: portals reach only the BFF (§8.3), only the ACL
|
||||
holds ZGW credentials (§8.1), only the Workflow Client talks to Flowable (§8.2), each
|
||||
service keeps its own database (§8.5). No workload gained a peer it didn't have in compose.
|
||||
|
||||
## Verified
|
||||
|
||||
Brought up from scratch on a single-node Talos v1.14.0 VM (6 vCPU / 10 GB, virtio disk)
|
||||
under virt-manager: 29 pods ready and four bootstrap Jobs complete in under three minutes,
|
||||
with zero restarts, using ~4.4 GB of the VM's 10 GB. The smoke test in the runbook's §5
|
||||
walks the whole path — portal proxy → BFF → domain → Flowable → ACL → OpenZaak + Objecten →
|
||||
NRC → event-subscriber → projection → public register — plus a werkbak read with an
|
||||
MFA'd medewerker token. The browser flow itself was driven with Playwright against
|
||||
`http://localhost:30140`: secure context, PKCE, Keycloak form, login, no console errors.
|
||||
|
||||
## Process note
|
||||
|
||||
CLAUDE.md §14 wants the ADR proposal issue opened before the code, and §7 wants a slice
|
||||
issue behind the work. This landed the other way round — chart first, on request. The
|
||||
issue and the CI drift check are the outstanding follow-ups.
|
||||
@@ -0,0 +1,103 @@
|
||||
# ADR-0034: The portals are served by Caddy, not nginx
|
||||
|
||||
- **Status:** Accepted
|
||||
- **Date:** 2026-09-04
|
||||
- **Deciders:** Respellion engineering
|
||||
- **Slice:** _(none yet — raised directly alongside the Kubernetes deployment, ADR-0033)_
|
||||
|
||||
## Context
|
||||
|
||||
Each portal ships as one image that does two jobs: serve the built Angular app, and
|
||||
reverse-proxy *its own* BFF endpoint group so the browser calls a single origin (no CORS,
|
||||
and the DigiD/medewerker token rides along — ADR-0010, ADR-0013). Until now that was nginx
|
||||
with a hand-written `nginx.conf` per app.
|
||||
|
||||
Two workarounds had accumulated around nginx's resolver, both for the same root cause:
|
||||
**nginx resolves a variable `proxy_pass` upstream itself**, using only the `resolver`
|
||||
directive, and never the search domains in `/etc/resolv.conf`.
|
||||
|
||||
1. `resolver 127.0.0.11` (Docker's embedded DNS) is wrong on rootless podman, which uses a
|
||||
network-specific aardvark address — so `apps/portal-nginx-resolver.sh` rewrote the
|
||||
directive at container start by reading the pod's actual nameserver.
|
||||
2. On Kubernetes the bare `bff` name cannot resolve at all without the `svc.cluster.local`
|
||||
search domain, so the same script gained a `BFF_HOST` override that the Helm chart set
|
||||
per portal (ADR-0033).
|
||||
|
||||
Both existed only to tell the proxy how to resolve one hostname.
|
||||
|
||||
## Decision
|
||||
|
||||
**Serve the portals with `caddy:2-alpine` and a small `Caddyfile` per app, replacing the
|
||||
nginx runtime stage, the four `nginx.conf` files, and the resolver workaround.**
|
||||
|
||||
Caddy dials its upstream per request through Go's resolver, which reads
|
||||
`/etc/resolv.conf` — nameserver *and* search domains. So `reverse_proxy bff:8080` resolves
|
||||
correctly under Docker, rootless podman and Kubernetes with no per-engine configuration,
|
||||
and it still starts before the BFF exists and picks up its restarts (the property the
|
||||
variable `proxy_pass` was there to buy). `apps/portal-nginx-resolver.sh`, its unit test and
|
||||
the chart's `BFF_HOST` env are deleted.
|
||||
|
||||
The Caddyfile uses `handle` blocks rather than a bare `try_files`:
|
||||
|
||||
```
|
||||
handle /behandel/* { reverse_proxy bff:8080 }
|
||||
handle { root * /usr/share/caddy; try_files {path} /index.html; file_server }
|
||||
```
|
||||
|
||||
`handle` blocks are mutually exclusive and matched most-specific-first. This matters:
|
||||
Caddy's default directive order puts rewrites (`try_files`) *before* `reverse_proxy`, so a
|
||||
top-level `try_files {path} /index.html` would rewrite every API path to `/index.html`
|
||||
before the proxy ever saw it — the SPA fallback would silently eat the API. The `handle`
|
||||
form makes the routing explicit instead of relying on directive-order trivia.
|
||||
|
||||
`infra/test_portal_caddyfiles.py` (in `make unit`) asserts each portal proxies exactly its
|
||||
own endpoint groups and keeps the SPA fallback. The four files are near-identical, so a
|
||||
copy-paste slip is cheap to make and expensive to find: proxying another portal's group
|
||||
hands a browser an endpoint its token isn't for, and the failure surfaces as a 401 three
|
||||
services away.
|
||||
|
||||
### Alternatives considered
|
||||
|
||||
- **Keep nginx.** Zero migration, and it works — but the resolver workaround stays, and it
|
||||
had already grown a second head for Kubernetes. Both heads are nginx-specific.
|
||||
- **Keep nginx, hard-code the FQDN.** Would need a different config per deployment target
|
||||
(compose vs Kubernetes), which is exactly the fork the chart was written to avoid.
|
||||
- **Drop the proxy and use CORS.** Turns the same-origin design (ADR-0010) inside out:
|
||||
CORS preflights, an explicit origin allowlist in the BFF, and a token attached
|
||||
cross-origin. Not a serving decision — an architectural regression.
|
||||
- **Kubernetes Ingress in front of the portals.** Solves nothing about compose, adds a
|
||||
controller, and the portals would still need something to serve static files.
|
||||
|
||||
- ponytail ceiling: plain HTTP on `:80`, no compression, no cache headers beyond Caddy's
|
||||
defaults, and Caddy's automatic HTTPS deliberately unused (there is no hostname to get a
|
||||
certificate for). Upgrade path: `encode zstd gzip` and a cache policy for immutable
|
||||
Angular bundles; a real hostname makes TLS a one-line `Caddyfile` change, which is the
|
||||
main reason this is worth having in place.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Positive**
|
||||
|
||||
- One resolver behaviour across compose, podman and Kubernetes; a script, a unit test and a
|
||||
chart env var are deleted rather than maintained.
|
||||
- The images gain `curl` for free (the alpine nginx image had only busybox `wget`), which
|
||||
the compose healthchecks can use.
|
||||
- Routing intent is readable: one `handle` block per endpoint group, one for the app.
|
||||
- TLS later is a one-line change instead of a new component.
|
||||
|
||||
**Negative / costs**
|
||||
|
||||
- A new runtime dependency in four images (CLAUDE.md §13): Caddy replaces nginx rather than
|
||||
joining it, so the count is unchanged, but it is a less familiar config language for
|
||||
anyone who has only read nginx configs.
|
||||
- The images grew: 90.6 MB against nginx's 75.7 MB, because `caddy:2-alpine` carries a
|
||||
bigger static binary than nginx's. Measured, not estimated.
|
||||
- Caddy's directive-order rule is a genuine footgun (see above); the `handle` form and the
|
||||
Caddyfile comments exist to keep the next person out of it.
|
||||
- Any operational note that says "the portal's nginx" is now wrong; the ones in `docs/` were
|
||||
updated with this ADR.
|
||||
|
||||
## Coupling rules touched (CLAUDE.md §8)
|
||||
|
||||
None. §8.3 is unchanged and unchanged in kind: the portals still talk only to the BFF, and
|
||||
the proxy is still the thing that makes that same-origin.
|
||||
Reference in New Issue
Block a user