S-24/#25 · Helm chart + Kubernetes deployment, and Caddy for the portals (#166) (#167)

## What & why

Two changes, made and verified together on a real cluster.

**S-24 / #25 — a Helm chart for the platform.** One chart, `infra/helm/big-reference`,
whose `values.yaml` is a near-literal transcription of `infra/docker-compose.yml`, rendered
by three generic templates (Deployment, Job, Service) over a `workloads` map. Adding a
service is a values edit. `make k8s-lint` renders and schema-checks the whole stack without
a cluster. The issue asked for a *sketch*; this is deployed and verified end to end (see
below), which is more than it asked for — the part it asked for that is **not** here is the
production-posture write-up (HA, secrets, backup), see Known gaps.

**#166 — Caddy replaces nginx in the portals.** nginx resolves a variable `proxy_pass`
upstream itself, using only the `resolver` directive and never `/etc/resolv.conf`'s search
domains. That had cost two workarounds in one script: rewriting the resolver address for
rootless podman, and injecting a full FQDN so the bare `bff` name could resolve on
Kubernetes. Caddy dials per request through the system resolver, so `reverse_proxy
bff:8080` works on every engine unchanged; `apps/portal-nginx-resolver.sh` and the chart's
`BFF_HOST` env are deleted.

Closes #25
Closes #166

## Definition of Done

- [x] Linked Gitea issue (above).
- [x] Failing test committed before the implementation — twice: the Caddyfile contract test
      before the Caddyfiles, `make k8s-lint` before the chart.
- [x] Implementation makes the test pass.
- [x] Conventional Commits referencing the issues (`refs #25` / `refs #166`).
- [ ] CI green — awaiting the run on this PR (`make k8s-lint`, `dotnet format` and the new
      unit self-check pass locally; the compose e2e and mutation lanes are CI's).
- [ ] `docker compose up` from a fresh clone reaches green health checks within 3 minutes —
      the portal images were rebuilt and verified standalone, but a full `make up` run has
      not been done on this branch. Please confirm in review or let CI's smoke test speak.
- [x] Docs updated — `docs/runbooks/kubernetes-talos.md` (new), `frontend-decisions.md`,
      `demo-script.md`, and the docs that named nginx.
- [x] ADR added — ADR-0033 (chart) and ADR-0034 (Caddy).
- [ ] Demo note in `docs/demo-script.md` — not added: the deployment target is not a
      user-visible slice, and the Caddy swap is invisible to the demo script beyond the
      wording fix included here.

## How it was verified

Brought up from scratch on a single-node Talos v1.14.0 VM (6 vCPU / 10 GB, virtio disk)
under virt-manager: **29 pods ready and four bootstrap Jobs complete in under three
minutes, zero restarts**, using ~4.4 GB of the VM's 10 GB.

- Full Common Ground path: portal Caddy → BFF → domain → Flowable → ACL → OpenZaak +
  Objecten → NRC → event-subscriber → projection → public register (`INGEDIEND`, reference
  matching the submitted registration).
- Werkbak read with an MFA'd medewerker token → 200.
- The browser flow driven with Playwright against `http://localhost:30140`: secure context,
  `crypto.subtle` present, Keycloak form reached, login completed, **no console errors**.
- Routing checked against a stub BFF: SPA fallback serves deep links, each portal proxies
  its own groups, and a portal does *not* proxy a neighbour's group.

## Notes for reviewers

Three bugs this shook out, each fixed at the cause rather than the symptom:

1. **`command` vs `args`.** Compose's `command:` replaces the image CMD; Kubernetes'
   replaces the ENTRYPOINT. Transcribing one to the other broke every upstream image that
   relies on its entrypoint — postgres refused to run as root, Keycloak tried to exec
   `start-dev`. The chart now `fail`s at render time on `command`.
2. **Concurrent migrations.** Both `/setup_configuration.sh` and `/start.sh` run
   `manage.py migrate`; compose serialises them with `depends_on`, Kubernetes has no such
   edge, so the init Job and its web pod raced (`relation "zgw_consumers_service" already
   exists`). The four Django services now do both steps in order in the web pod — which
   also deletes four workloads.
3. **`emptyDir` databases are wiped by any pod-template change.** `make k8s-reseed` now
   also restarts `event-subscriber` and `projection-api`, which create the projection
   schema on start and otherwise keep writing to a schema-less database.

Known gaps / follow-ups:

- **Secrets.** `values.yaml` carries the dev credentials in plain text (`admin/admin`, the
  ZGW client secret, the two Objecten tokens) and the chart has no `Secret` objects. Fine
  for a laptop demo, and exactly what #25's "production posture" ADR should address — I
  suggest a follow-up issue rather than stretching this PR.
- **No CI gate for the chart yet.** `make k8s-lint` exists but is not wired into
  `.gitea/workflows/ci.yaml`, and nothing enforces that the chart and the compose file stay
  in step. Worth a small follow-up.
- **This is two slices in one PR.** They were built and verified together and the diff is
  entangled (the chart was written against Caddy from the start), so splitting now would
  mean re-creating an nginx-shaped chart to throw away. Happy to split if you'd rather.
- **Rebased onto #161** (merged as #165) rather than merged, to keep the history linear.
  One conflict, in the `unit:` target where both branches add a self-check line — resolved
  by keeping both. #161's `infra/host-browser.yml` arrived with
  `/usr/share/nginx/html/config.json` and is fixed to `/usr/share/caddy/` inside the
  `feat(portals)` commit, so no commit on this branch leaves that overlay pointing at a
  path the images no longer have.Reviewed-on: #167

apps/beheer/Caddyfile

@@ -0,0 +1,21 @@
+:80 {
+	# Same-origin API: beheerders use the same medewerker realm as behandel (S-15a).
+	# `handle` blocks are mutually exclusive and matched most-specific-first, so the
+	# SPA fallback below can never swallow an API call — unlike a bare `try_files`,
+	# which Caddy sorts *before* reverse_proxy and would rewrite it to /index.html.
+	#
+	# No `resolver` stanza is needed: Caddy dials the upstream per
+	# request through the system resolver, so it starts before the BFF is up, picks up
+	# its restarts, and honours the DNS search domains in /etc/resolv.conf — which is
+	# what lets the bare `bff` name resolve on Kubernetes as well as under compose.
+	handle /beheer/* {
+		reverse_proxy bff:8080
+	}
+
+	# The Angular app. Client-side routing: an unknown path serves index.html.
+	handle {
+		root * /usr/share/caddy
+		try_files {path} /index.html
+		file_server
+	}
+}

apps/beheer/Dockerfile

@@ -1,4 +1,4 @@
-# Multi-stage build for the beheer portal (Angular → nginx).
+# Multi-stage build for the beheer portal (Angular → Caddy).
 # Build context is the repo root (the app needs the pnpm workspace + libs). See infra/docker-compose.yml.
 FROM node:24-slim AS build
 WORKDIR /src
@@ -13,15 +13,12 @@ COPY apps/beheer apps/beheer
 COPY libs libs
 RUN pnpm nx build beheer
 
-FROM nginx:1.27-alpine AS runtime
-COPY apps/beheer/nginx.conf /etc/nginx/conf.d/default.conf
-COPY --from=build /src/dist/apps/beheer/browser /usr/share/nginx/html
+FROM caddy:2-alpine AS runtime
+COPY apps/beheer/Caddyfile /etc/caddy/Caddyfile
+COPY --from=build /src/dist/apps/beheer/browser /usr/share/caddy
 # Compose-time OIDC config: the browser (Playwright, on the compose network) reaches Keycloak by
 # service name, so the token issuer matches the BFF's medewerker authority (host-consistent, ADR-0013).
-RUN printf '{ "authority": "http://keycloak:8080/realms/medewerker" }\n' > /usr/share/nginx/html/config.json
-# Make the reverse-proxy resolver engine-portable (Docker 127.0.0.11 vs podman aardvark); runs from
-# the nginx image's /docker-entrypoint.d before nginx starts.
-COPY apps/portal-nginx-resolver.sh /docker-entrypoint.d/40-resolver.sh
-RUN chmod +x /docker-entrypoint.d/40-resolver.sh
+# Kubernetes mounts a ConfigMap over this file with the node address instead (ADR-0033).
+RUN printf '{ "authority": "http://keycloak:8080/realms/medewerker" }\n' > /usr/share/caddy/config.json
 
 EXPOSE 80

apps/beheer/nginx.conf

@@ -1,24 +0,0 @@
-server {
-  listen 80;
-  server_name _;
-  root /usr/share/nginx/html;
-  index index.html;
-
-  # Resolve the BFF via Docker's embedded DNS at request time (variable proxy_pass), so nginx starts
-  # even before the BFF is up and picks up restarts — instead of failing to load the config.
-  resolver 127.0.0.11 ipv6=off valid=30s;
-
-  # Same-origin API: proxy the beheer endpoint group to the bff service. The api-client uses
-  # relative URLs, so the browser calls this origin and nginx forwards to the BFF — no CORS, and the
-  # medewerker token (same-origin) is attached by the app's interceptor (ADR-0013).
-  location /beheer/ {
-    set $bff http://bff:8080;
-    proxy_pass $bff;
-    proxy_set_header Host $host;
-  }
-
-  # SPA fallback — Angular client-side routing.
-  location / {
-    try_files $uri $uri/ /index.html;
-  }
-}

apps/beheer/src/app/app.config.ts

@@ -12,7 +12,7 @@ export interface RuntimeConfig {
 
 /**
  * Route prefixes whose requests carry the medewerker token. These MUST match the **relative** URLs
- * the api-client actually calls (same-origin via the nginx proxy) — the interceptor matches on
+ * the api-client actually calls (same-origin via the Caddy proxy) — the interceptor matches on
  * `req.url`, which stays relative, so an absolute origin would never match and the token would go
  * unattached. Only `/beheer/` is secured; the app calls no other endpoint group.
  */