Compare commits

..
Author SHA1 Message Date
notandClaude Opus 5 399d110663 docs(arch): ADR-0035 and the runbook section for publishing the stack (refs #177)
CI / k8s (pull_request) Successful in 6s
CI / build (pull_request) Successful in 1m28s
CI / lint (pull_request) Successful in 1m51s
CI / unit (pull_request) Successful in 1m12s
CI / frontend (pull_request) Successful in 2m11s
CI / mutation (pull_request) Successful in 3m45s
CI / verify-stack (pull_request) Successful in 8m15s
The ADR records why the edge is in the cluster rather than on the Fedora host —
routing and certificates should be state a `helm upgrade` can see — and the
three costs that buys: the host forward nobody in the cluster can repair, the
Let's Encrypt rate limit that makes `persistence.storageClass` non-optional, and
publishing behandel and beheer to the internet behind synthetic accounts.

Runbook §10 is the operational half: the five DNS records, the two firewalld
rules (including the masquerade that makes the return path work), and the
symptoms each missing piece produces.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 16:30:38 +02:00
notandClaude Opus 5 56cba9c340 feat(k8s): terminate TLS in the cluster for a public domain (refs #177)
`public.domain` is the whole switch. Empty — the default, and what compose, CI
and a laptop cluster use — renders nothing new and leaves every manifest as it
was. Set it and templates/edge.yaml adds a Caddy deployment that gets its own
certificates from Let's Encrypt and proxies the five browser-facing hostnames to
the ClusterIP services, so a public deployment doesn't use their NodePorts at
all.

Caddy rather than an ingress controller because the four portals already run
caddy:2-alpine (ADR-0034, whose ceiling note called exactly this out): no new
dependency, no cert-manager, no CRDs, no Ingress objects for five hostnames that
never change. The Fedora host keeps only a layer-4 forward of 80/443, because
the public IP is there and nothing in the cluster can claim it.

KC_HOSTNAME and the portals' config.json now both come from `big.keycloakUrl`,
so the issuer a token carries and the authority the BFF discovers are one string
by construction (ADR-0010) rather than by two templates agreeing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 16:30:38 +02:00
notandClaude Opus 5 88fda30008 test(k8s): pin the issuer, the portal authority and the public edge (refs #177)
Keycloak pins one issuer and each portal is configured with one authority; when
they drift the symptom lands three services away — a login that bounces back
logged out, or a 401 from the BFF (ADR-0010) — so assert they are the same
string. The same check states what publishing the stack has to mean: with
`public.domain` set the five hostnames are served and both halves become
`https://auth.<domain>`, and with it empty nothing of the edge renders, which is
what compose, CI and a laptop cluster depend on.

Red: the chart has no `public.domain`, so setting it changes nothing and no
hostname is published.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 16:30:38 +02:00
not 9d7e8e5b65 ci(k8s): gate the Helm chart in CI + a compose↔chart drift check (closes #168) (#171)
CI / k8s (push) Successful in 5s
CI / lint (push) Successful in 1m27s
CI / build (push) Successful in 1m22s
CI / unit (push) Successful in 1m12s
CI / frontend (push) Successful in 2m7s
CI / mutation (push) Successful in 3m9s
CI / verify-stack (push) Successful in 6m20s
## What & why

The Helm chart landed in #167 with two gaps written into ADR-0033: `make k8s-lint` existed
but no CI job ran it, and *"a second deployment description to keep in step with compose —
nothing enforces that today; a drift check belongs in CI (follow-up)"*. Both are closed here.

**`make k8s-drift`** (`infra/helm/check-drift.py`, stdlib only) compares what each stack
actually deploys rather than diffing two files that differ by design: workload names and
resolved container images, taken from `docker compose config --format json` and a rendered
chart. The six differences that exist today are declared in `DEVIATIONS` with the reason
each was forced — the four `*-init` Django services folded into their web pods, and the two
bootstrap Jobs compose runs from the host — so only a *new* difference fails.

**A `k8s` CI job** runs `k8s-lint` then `k8s-drift` on every push and PR. No cluster, no
marketplace action: helm is fetched as the pinned static binary the Talos runbook already
gives developers.

Closes #168

## Definition of Done

- [x] Linked Gitea issue (above).
- [x] Failing test committed before the implementation — the red commit reports all six
      real differences; the green commit declares them.
- [x] Implementation makes the test pass.
- [x] Conventional Commits referencing the issue (`refs #168`).
- [x] CI green — awaiting the run on this PR (`make k8s-lint` and `make k8s-drift` pass locally).
- [x] `docker compose up` unaffected — no service, image or compose file is touched.
- [x] Docs updated — `docs/runbooks/ci.md` (job table + the one place local and CI now
      differ), `docs/runbooks/kubernetes-talos.md` §7/§"not ported", and ADR-0033's cost note.
- [x] No ADR needed: no new dependency (python stdlib, and helm/docker were already
      prerequisites of the `k8s-*` targets), no boundary moved, no §8 rule bent.
- [x] Not user-visible, so no demo note.

## Notes for reviewers

Verified by hand that both drift classes fail the check, not just that it passes today:

- bumping `OPENZAAK_TAG` in compose alone → reports `openzaak` and `oz-celery` with both
  image strings;
- adding a workload to `values.yaml` alone → reports it by name.

Deliberate limits (there is a `ponytail:` note in the script):

- **Names and images only**, as sets — no per-workload env, ports or volumes. Those differ
  by design in four documented places, so comparing them would mean re-encoding every
  deviation field by field for very little more signal.
- **The three observability workloads are rendered with `enabled=true`** by the check, even
  though both stacks default them off, so their images can't drift unwatched.
- **`k8s-lint`/`k8s-drift` are not in `make ci`**, to avoid making `helm` a hard
  prerequisite for everyone. That is now the only local/CI difference; it's called out in
  `docs/runbooks/ci.md`.

Follow-ups filed while reviewing the chart, not addressed here: #169 (the published docs
omit every ADR after 0010 and all runbooks but `ci.md`) and #170 (the production-posture
ADR #25 asked for — secrets are still plain text in `values.yaml`).Reviewed-on: #171
2026-09-18 13:25:24 +00:00
not 17f1f2f809 docs(nav): publish every ADR and runbook, gated by a nav check (closes #169) (#172)
CI / lint (push) Successful in 1m46s
CI / build (push) Successful in 1m29s
CI / unit (push) Successful in 1m5s
CI / frontend (push) Successful in 1m41s
CI / mutation (push) Successful in 3m30s
CI / verify-stack (push) Successful in 5m50s
## What & why

`docs/` is the source of truth (CLAUDE.md §12), but only pages listed in `mkdocs.yml`'s nav
are published — and mkdocs' own `validation.nav.omitted_files: warn` keeps the build green
while dropping the rest. So the site had quietly stopped at **ADR-0010** and
**`runbooks/ci.md`**: 31 pages, including every ADR from 0011 to 0034, six of the seven
runbooks, and `synthetic-data.md`, existed in the repo and nowhere else.

- `infra/check-docs-nav.py` fails when a page under `docs/` is not in the nav. It runs in
  `make lint`, so the existing CI job gates it — python3 only, no new tooling, and no
  mkdocs install needed to check it.
- The nav now lists all 34 ADRs, all 7 runbooks and `synthetic-data.md`.
- ADR-0033's `Slice:` header said "none yet"; #25 closed it.
- The landing page gained a pointer to the Talos runbook.

Closes #169

## Definition of Done

- [x] Linked Gitea issue (above).
- [x] Failing test committed before the fix — the red commit lists all 31 missing pages.
- [x] Implementation makes the test pass.
- [x] Conventional Commits referencing the issue (`refs #169`).
- [ ] CI green — awaiting the run on this PR (`python3 infra/check-docs-nav.py` passes locally;
      `make lint` also needs the .NET SDK, which CI has).
- [x] `docker compose up` unaffected — docs and `mkdocs.yml` only, plus one `make lint` line.
- [x] Docs updated — that is the change.
- [x] No ADR needed: no dependency, no boundary, no §8 rule touched.
- [x] Not user-visible, so no demo note.

## Notes for reviewers

- The check is a **substring test**, not a YAML parse (there's a `ponytail:` note in the
  script): a page's path either appears in `mkdocs.yml` or it doesn't. That keeps it
  dependency-free — `mkdocs.yml` can't be read by `yaml.safe_load` anyway, it carries a
  `!!python/name:` tag for the mermaid fence. It does not check that an entry *points at a
  file that exists*; mkdocs' `not_found: warn` covers that direction.
- ADR labels in the nav are shortened by hand (`"ADR-0013: Behandel-portal wiring"`), since
  several H1s are a full sentence.

**Known gap, not fixed here:** CLAUDE.md §12 says the site is "published via a Gitea Actions
workflow to Gitea Pages", and no such workflow exists — `mkdocs build` is never run, by CI or
by any make target. Gitea has no built-in Pages, so publishing needs a decision (a
`gitea-pages` server, an artifact, or a static host) rather than a patch. Worth its own issue
if the published site is actually wanted; until then this PR makes the nav correct for whoever
runs `mkdocs serve`.Reviewed-on: #172
2026-09-18 12:55:20 +00:00
not 1dd8bd4e1b S-24/#25 · Helm chart + Kubernetes deployment, and Caddy for the portals (#166) (#167)
CI / lint (push) Successful in 1m17s
CI / build (push) Successful in 1m12s
CI / unit (push) Successful in 1m26s
CI / frontend (push) Successful in 2m58s
CI / mutation (push) Successful in 9m1s
CI / verify-stack (push) Successful in 8m53s
## What & why

Two changes, made and verified together on a real cluster.

**S-24 / #25 — a Helm chart for the platform.** One chart, `infra/helm/big-reference`,
whose `values.yaml` is a near-literal transcription of `infra/docker-compose.yml`, rendered
by three generic templates (Deployment, Job, Service) over a `workloads` map. Adding a
service is a values edit. `make k8s-lint` renders and schema-checks the whole stack without
a cluster. The issue asked for a *sketch*; this is deployed and verified end to end (see
below), which is more than it asked for — the part it asked for that is **not** here is the
production-posture write-up (HA, secrets, backup), see Known gaps.

**#166 — Caddy replaces nginx in the portals.** nginx resolves a variable `proxy_pass`
upstream itself, using only the `resolver` directive and never `/etc/resolv.conf`'s search
domains. That had cost two workarounds in one script: rewriting the resolver address for
rootless podman, and injecting a full FQDN so the bare `bff` name could resolve on
Kubernetes. Caddy dials per request through the system resolver, so `reverse_proxy
bff:8080` works on every engine unchanged; `apps/portal-nginx-resolver.sh` and the chart's
`BFF_HOST` env are deleted.

Closes #25
Closes #166

## Definition of Done

- [x] Linked Gitea issue (above).
- [x] Failing test committed before the implementation — twice: the Caddyfile contract test
      before the Caddyfiles, `make k8s-lint` before the chart.
- [x] Implementation makes the test pass.
- [x] Conventional Commits referencing the issues (`refs #25` / `refs #166`).
- [ ] CI green — awaiting the run on this PR (`make k8s-lint`, `dotnet format` and the new
      unit self-check pass locally; the compose e2e and mutation lanes are CI's).
- [ ] `docker compose up` from a fresh clone reaches green health checks within 3 minutes —
      the portal images were rebuilt and verified standalone, but a full `make up` run has
      not been done on this branch. Please confirm in review or let CI's smoke test speak.
- [x] Docs updated — `docs/runbooks/kubernetes-talos.md` (new), `frontend-decisions.md`,
      `demo-script.md`, and the docs that named nginx.
- [x] ADR added — ADR-0033 (chart) and ADR-0034 (Caddy).
- [ ] Demo note in `docs/demo-script.md` — not added: the deployment target is not a
      user-visible slice, and the Caddy swap is invisible to the demo script beyond the
      wording fix included here.

## How it was verified

Brought up from scratch on a single-node Talos v1.14.0 VM (6 vCPU / 10 GB, virtio disk)
under virt-manager: **29 pods ready and four bootstrap Jobs complete in under three
minutes, zero restarts**, using ~4.4 GB of the VM's 10 GB.

- Full Common Ground path: portal Caddy → BFF → domain → Flowable → ACL → OpenZaak +
  Objecten → NRC → event-subscriber → projection → public register (`INGEDIEND`, reference
  matching the submitted registration).
- Werkbak read with an MFA'd medewerker token → 200.
- The browser flow driven with Playwright against `http://localhost:30140`: secure context,
  `crypto.subtle` present, Keycloak form reached, login completed, **no console errors**.
- Routing checked against a stub BFF: SPA fallback serves deep links, each portal proxies
  its own groups, and a portal does *not* proxy a neighbour's group.

## Notes for reviewers

Three bugs this shook out, each fixed at the cause rather than the symptom:

1. **`command` vs `args`.** Compose's `command:` replaces the image CMD; Kubernetes'
   replaces the ENTRYPOINT. Transcribing one to the other broke every upstream image that
   relies on its entrypoint — postgres refused to run as root, Keycloak tried to exec
   `start-dev`. The chart now `fail`s at render time on `command`.
2. **Concurrent migrations.** Both `/setup_configuration.sh` and `/start.sh` run
   `manage.py migrate`; compose serialises them with `depends_on`, Kubernetes has no such
   edge, so the init Job and its web pod raced (`relation "zgw_consumers_service" already
   exists`). The four Django services now do both steps in order in the web pod — which
   also deletes four workloads.
3. **`emptyDir` databases are wiped by any pod-template change.** `make k8s-reseed` now
   also restarts `event-subscriber` and `projection-api`, which create the projection
   schema on start and otherwise keep writing to a schema-less database.

Known gaps / follow-ups:

- **Secrets.** `values.yaml` carries the dev credentials in plain text (`admin/admin`, the
  ZGW client secret, the two Objecten tokens) and the chart has no `Secret` objects. Fine
  for a laptop demo, and exactly what #25's "production posture" ADR should address — I
  suggest a follow-up issue rather than stretching this PR.
- **No CI gate for the chart yet.** `make k8s-lint` exists but is not wired into
  `.gitea/workflows/ci.yaml`, and nothing enforces that the chart and the compose file stay
  in step. Worth a small follow-up.
- **This is two slices in one PR.** They were built and verified together and the diff is
  entangled (the chart was written against Caddy from the start), so splitting now would
  mean re-creating an nginx-shaped chart to throw away. Happy to split if you'd rather.
- **Rebased onto #161** (merged as #165) rather than merged, to keep the history linear.
  One conflict, in the `unit:` target where both branches add a self-check line — resolved
  by keeping both. #161's `infra/host-browser.yml` arrived with
  `/usr/share/nginx/html/config.json` and is fixed to `/usr/share/caddy/` inside the
  `feat(portals)` commit, so no commit on this branch leaves that overlay pointing at a
  path the images no longer have.Reviewed-on: #167
2026-09-10 08:53:58 +00:00
not d6b3f9764f fix(e2e): bound the Playwright run and make a failed login say why (closes #161) (#165)
CI / build (push) Successful in 1m9s
CI / lint (push) Successful in 1m27s
CI / unit (push) Successful in 1m32s
CI / frontend (push) Successful in 3m19s
CI / mutation (push) Successful in 6m24s
CI / verify-stack (push) Successful in 9m33s
## What & why

#161 is really two defects, and the second one is why the first was undiagnosable.

**A wedged suite consumed the job, and took the post-mortem with it.** Nothing bounded the
Playwright run, so CI stopped the job mid-suite — and `if: always()` does not survive that. Run
739's job metadata shows every step after the e2e as a **0-second failure** stamped at the kill:

```
14 failure  09:48:17 -> 10:14:54  Self-service e2e (Playwright …)
15 failure  10:14:54 -> 10:14:54  verify-stack check summary       ← if: always()
16 failure  10:14:54 -> 10:14:54  e2e spec summary                 ← if: always()
17 failure  10:14:54 -> 10:14:54  Dump container logs on failure   ← if: failure()
18 failure  10:14:54 -> 10:14:54  Tear down                        ← if: always()
```

So the per-spec summary, the container-log dump and the teardown never ran, and the log lost
whatever the killed process had buffered — leaving the single `✘` line the issue was filed from.
`globalTimeout` now makes Playwright stop and *report*: the JSON report is written and those steps
still get their turn. (A `timeout-minutes` on the job would have reproduced the same failure, so
there isn't one.) The "~24-minute gap" is that kill, not necessarily a hang — note run 739 shows
`run_attempt: 2`, and `concurrency.cancel-in-progress` kills an in-flight run on any re-run or push.

**A login that never got its form ate the 90-second test timeout.** Playwright actions auto-wait
until the *test* timeout, not `expect.timeout` — so a portal that serves its page but never
bootstraps (its `config.json` fetch or the OIDC discovery behind `authorize()` failed; `main.ts`
only `console.error`s) spent 90s to report `locator.fill: Test timeout of 90000ms exceeded`: the
symptom, not the cause. That is catalogus.spec's 1.8 minutes. Both Keycloak forms are now asserted
visible first, with a 20s budget and a message naming the step that never happened.

Verified against a real blank-bootstrap portal — the beheer image served with a `config.json` that
is not JSON — which fails in **20.2s** with *"the Keycloak login form never appeared — the portal
did not reach Keycloak (check its config.json fetch and the OIDC discovery …)"*.

**And the summary now says why.** The per-spec table (#136) rendered a verdict icon and nothing
else, so even a surviving summary cost a log dive. Failing specs now carry their first error,
flattened for a table cell (ANSI stripped, newlines collapsed, `|` escaped, clipped) — shape
verified against a real @playwright/test 1.61 failing report, with a stdlib assert self-check on
`make unit`.

Closes #161

## Definition of Done

- [x] Linked Gitea issue (above).
- [x] Failing test committed before the implementation.
- [x] Implementation makes the test pass; refactor commit follows (login helper dedup).
- [x] Conventional Commits referencing the issue (`refs #161`).
- [ ] CI green — all Gitea Actions jobs.
- [x] `docker compose up` from a fresh clone reaches green health checks within 3 minutes (untouched).
- [x] Docs updated — `docs/runbooks/gitea-actions-gotchas.md` §9.
- [x] ADR — not needed: no boundary, dependency or coupling rule touched (test/CI infra only).
- [x] Demo note — not applicable: nothing user-visible.

## Notes for reviewers

**What this does not do: identify why the beheerder login failed that once.** The evidence to do
that was destroyed by defect 2, which is what this PR fixes. The suite ran green here five times
today (catalogus.spec 1.1–5.3s each) — but a local box is not the loaded CI runner, so that is weak
evidence and I am not claiming the flake is gone. What changes is that the next occurrence is
bounded and self-describing: it fails in 20s naming the failing step, the JSON report survives, and
the summary prints the error. Please keep #161 in mind rather than treating this as proof.

**Two follow-ups I did not pull into this PR:**
- *All four portals show a permanently blank page if their startup fetch fails* — `main.ts` does
  `fetch('config.json').then(bootstrap).catch(console.error)`, one shot, no UI and no recovery. That
  is a real product gap (the deliberately-broken portal above is exactly what a user would see) and
  wants its own slice, not a test-infra PR.
- `retries: 1` is untouched. CLAUDE.md §15 says flaky tests are fixed rather than retried, but
  removing retries while a real flake is unexplained would trade a rare red for a frequent one.
  Worth revisiting once #161 recurs (or doesn't) with the new diagnostics.

The login-helper rename (`medewerker-login.ts` → `keycloak-login.ts`, citizen logins routed through
`loginBurger`) is its own no-behaviour-change commit: the three citizen specs each duplicated the
same three-line login, so guarding the login path once meant routing them through it first.Reviewed-on: #165
2026-09-04 10:53:35 +00:00
29 changed files with 980 additions and 86 deletions
+21
View File
@@ -41,6 +41,27 @@ jobs:
nuget-${{ runner.os }}-
- run: make lint
# The Helm chart's only automated gate: it renders and schema-checks the whole
# stack, and checks it still describes the same stack as the compose file
# (ADR-0033). No cluster involved — see docs/runbooks/kubernetes-talos.md.
k8s:
runs-on: ubuntu-latest
steps:
- uses: https://github.com/actions/checkout@v4
# helm as its pinned static binary rather than a marketplace action: one URL,
# the same one the Talos runbook §0 gives a developer, and no third-party
# action to vet (CLAUDE.md §13). The drift check also needs `docker compose`,
# which the runner already has (see docs/runbooks/ci.md).
- name: Install helm
run: |
mkdir -p "$HOME/.local/bin"
curl -sSL https://get.helm.sh/helm-v3.16.4-linux-amd64.tar.gz \
| tar xz -O linux-amd64/helm > "$HOME/.local/bin/helm"
chmod +x "$HOME/.local/bin/helm"
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
- run: make k8s-lint
- run: make k8s-drift
build:
runs-on: ubuntu-latest
steps:
+15 -1
View File
@@ -43,7 +43,7 @@ export DOCKER_HOST := unix://$(PODMAN_SOCK)
endif
endif
.PHONY: ci lint build unit mutation frontend integration verify verify-up verify-acl verify-nrc verify-projection verify-bff verify-domain verify-observability verify-tracing verify-metrics verify-objecttypen verify-objecten verify-registerrecord verify-objecten-notifications verify-notifications smoke up down local verify-local local-down changelog openzaak-up openzaak-smoke openzaak-seed openzaak-down stack-up stack-smoke stack-down keycloak-up keycloak-smoke keycloak-down flowable-up flowable-smoke flowable-down k8s-lint k8s-registry k8s-images k8s-seed k8s-up k8s-reseed k8s-portals k8s-down k8s-purge help
.PHONY: ci lint build unit mutation frontend integration verify verify-up verify-acl verify-nrc verify-projection verify-bff verify-domain verify-observability verify-tracing verify-metrics verify-objecttypen verify-objecten verify-registerrecord verify-objecten-notifications verify-notifications smoke up down local verify-local local-down changelog openzaak-up openzaak-smoke openzaak-seed openzaak-down stack-up stack-smoke stack-down keycloak-up keycloak-smoke keycloak-down flowable-up flowable-smoke flowable-down k8s-lint k8s-drift k8s-registry k8s-images k8s-seed k8s-up k8s-reseed k8s-portals k8s-down k8s-purge help
## ci: run the full pipeline — lint, build, unit, mutation, frontend, verify (mirrors Gitea Actions)
## `verify` is the live-stack stage (full stack up once → ACL + notification checks).
@@ -64,6 +64,9 @@ frontend:
## lint: verify formatting (no changes)
lint:
dotnet format $(SLN) --verify-no-changes
# Only pages in mkdocs.yml's nav are published, and mkdocs keeps a build green
# when one is missing — so the nav is checked here rather than not at all.
python3 infra/check-docs-nav.py
## build: release build
build:
@@ -71,8 +74,11 @@ build:
## unit: run unit tests (excludes the container-backed Integration lane)
# TRX per test project (→ TestResults/) feeds the CI per-service summary (#136); harmless locally.
# The CI reporting scripts are stdlib Python with their own assert-based self-checks (#161) — they
# ride this lane so a broken job summary is caught by CI rather than by the next red pipeline.
unit:
dotnet test $(SLN) -c Release --filter "Category!=Integration" --logger trx --results-directory TestResults
python3 infra/test_playwright_summary.py
python3 infra/test_portal_caddyfiles.py
## mutation: run the Stryker.NET ratchet on each service with branching logic (fails below baseline)
@@ -346,6 +352,14 @@ K8S_IMAGES := acl domain bff event-subscriber projection-api self-service open
k8s-lint:
helm lint $(K8S_CHART)
helm template big $(K8S_CHART) -n $(K8S_NS) --set images.registry=registry.invalid:5000 >/dev/null
python3 infra/helm/check-issuer.py
## k8s-drift: fail if compose and the Helm chart describe different stacks
# Compose is CI-canonical (ADR-0033) and the chart is a transcription of it; this
# compares what each one deploys — workload names and resolved images. Needs
# `docker compose` and `helm`, no cluster.
k8s-drift:
python3 infra/helm/check-drift.py
## k8s-registry: deploy the in-cluster image registry (NodePort 30500)
k8s-registry:
@@ -3,8 +3,8 @@
- **Status:** Accepted
- **Date:** 2026-09-04
- **Deciders:** Respellion engineering
- **Slice:** _(none yet — raised directly as a deployment-target request; see
"Process note" at the end)_
- **Slice:** #25 (S-24) — raised directly as a deployment-target request and matched to
that issue afterwards; see the "Process note" at the end
## Context
@@ -126,8 +126,9 @@ Consequences of that shape, each chosen deliberately:
**Negative / costs**
- A second deployment description to keep in step with compose. Nothing enforces that
today; a drift check belongs in CI (follow-up).
- A second deployment description to keep in step with compose. `make k8s-drift` (#168)
now enforces the part that bites — the workload set and the resolved images, with the
four deviations below declared — but not per-workload env, ports or volumes.
- `helm install` alone is not enough — the ConfigMaps must be seeded first, and a missing
one surfaces as `ContainerCreating`, not as a clear error.
- Generic templates mean a values typo can render valid-but-wrong YAML; `k8s-lint` catches
@@ -0,0 +1,102 @@
# ADR-0035: The public TLS edge is a Caddy deployment in the cluster
- **Status:** Accepted
- **Date:** 2026-09-18
- **Deciders:** Respellion engineering
- **Slice:** [#177](https://git.labs.respellion.tech/eho/register-referentie/issues/177)
## Context
The stack deploys to a Talos VM on the lab server (ADR-0033, issue #175). Until now it was
only usable through five SSH port-forwards: the portals' OIDC flow uses PKCE, PKCE needs
`crypto.subtle`, and browsers expose that only in a **secure context** — HTTPS or an origin
on `localhost`. A NodePort on the VM's address is neither, so the deployment was pinned to
`host: localhost` and every viewer had to forward all five browser-facing ports (a portal
without Keycloak on the same `localhost:30180` fails on the discovery document).
That is not a demo anyone can be sent a link to. We want public hostnames with real
certificates — and we want the routing and the certificates to be cluster state, not
host-side configuration that no `helm upgrade` can see.
The public IP is on the Fedora host (`46.224.220.37`); the cluster is a libvirt guest
behind it.
## Decision
**Terminate TLS in the cluster, with a Caddy deployment rendered by the chart
(`templates/edge.yaml`), and give the Fedora host nothing but a layer-4 forward.**
- `public.domain` is the single switch. Empty — the default, and what compose and CI use —
renders nothing: the stack is reached on its NodePorts and `host` pins the OIDC origin
exactly as before. Set it, and the edge appears.
- `public.routes` maps a subdomain to an in-cluster `service:port`. Caddy proxies to the
**ClusterIP** services, so a public deployment does not use the browser-facing NodePorts
at all.
- Caddy obtains and renews certificates itself (ACME HTTP-01). There is no cert-manager.
- The host forwards `:80`/`:443` to two NodePorts with two `firewall-cmd
--add-forward-port` rules. No TLS, no routing, no per-service knowledge there — adding a
portal is a chart change, not a host change.
- `KC_HOSTNAME` and the portals' `config.json` stop being `host` + NodePort. Both now come
from one helper, `big.keycloakUrl`, so the issuer Keycloak pins and the authority the
portals are configured with cannot drift apart (ADR-0010).
### Alternatives considered
- **Caddy on the Fedora host.** Fewest moving parts — but the routing table and the
certificates would live outside the cluster, in a file no deployment touches, and adding
a portal would mean editing a host we deploy to over SSH. Rejected on exactly the ground
this ADR exists to record.
- **Traefik or ingress-nginx, plus cert-manager.** The conventional answer, and the right
one for a cluster with many teams and changing hostnames. Here it buys a controller, a
set of CRDs and Ingress objects to describe five hostnames that never change — and
cert-manager to do what Caddy already does unprompted.
- **A `LoadBalancer` service (MetalLB).** Solves address allocation, which is not the
problem; the node has exactly one address and it still is not the public one.
- **Keep the SSH forwards.** Free, and genuinely fine for one developer. It is not a demo
you can send to someone.
## Consequences
**Positive**
- No new dependency: the four portals already run `caddy:2-alpine` (ADR-0034), whose
ceiling note called this out — *"a real hostname makes TLS a one-line `Caddyfile`
change"*. This is that change.
- Routing is cluster state: `kubectl -n big get cm caddy-edge-config -o yaml` is the whole
truth about what is published, and `helm upgrade` is how it changes.
- The secure context is real, so `TALOS_HOST=localhost` and the five forwards disappear —
and with them the class of failure where a mismatched issuer logs the user out silently.
- Nothing changes for compose, CI or a laptop cluster: with `public.domain` empty the
rendered manifests are byte-identical to before.
**Negative / costs**
- The host forward is irreducible. Two firewalld rules, applied by hand once, with `sudo`
on a machine our pipeline reaches only over SSH. If someone rebuilds that host, the stack
is unreachable until they are re-applied, and nothing in the cluster can tell them so.
- **Certificates need a volume.** On the default `emptyDir` every pod restart asks Let's
Encrypt again, and its duplicate-certificate limit is five per week — a handful of
restarts and the edge serves an untrusted certificate for a week. `persistence.storageClass`
stops being optional for anything public (runbook §6).
- **All five hostnames are published, including `behandel` and `beheer`**, which approve
registrations and administer the register. They are protected by synthetic accounts with
well-known passwords, and by MFA on the medewerker realm (ADR-0031). That is a deliberate
choice for a demonstration environment holding synthetic data only, and it is the reason
this bullet is in the ADR rather than in a comment: if this stack ever holds anything
real, this decision is the first one to revisit.
- One more workload in the chart with no counterpart in compose — compose has no edge
because it has no hostname. The drift check (`make k8s-drift`) renders the defaults, so
it does not see it.
- `auth` is load-bearing: `big.keycloakUrl` builds the issuer from that subdomain, so
renaming the key in `public.routes` without the helper breaks every login. Both carry a
comment saying so.
- ponytail ceiling: one replica, no HSTS, no security headers beyond Caddy's defaults, no
rate limiting, and HTTP-01 rather than DNS-01 (so a wildcard certificate is not
available). Upgrade path in that order; DNS-01 first if the subdomain list ever grows.
## Coupling rules touched (CLAUDE.md §8)
None. §8.3 holds — the browser reaches a portal, the portal reverse-proxies its own BFF
group, and the edge is in front of all of it. The edge terminates TLS and routes by
hostname; it does not know what any service does.
+2
View File
@@ -14,6 +14,8 @@ should teach.
In Dutch; the strategic framing lives in `Respellion/innovation-lab`.
- **[Working in Gitea](gitea-workflow.md)** — issues, milestones, branches, PRs.
- **[CI runbook](runbooks/ci.md)** — the pipeline and the `make ci` local gate.
- **[Kubernetes on Talos](runbooks/kubernetes-talos.md)** — the second deployment target:
one Helm chart, a single-node cluster, and the parts that bite (ADR-0033).
## Quickstart
+6 -2
View File
@@ -2,8 +2,10 @@
> **Status: active.** The workflow `.gitea/workflows/ci.yaml` runs on Gitea's
> hosted `ubuntu-latest` runner — no self-hosted runner required.
> **`make ci` is still the local gate** — it runs the exact same checks
> (the workflow calls the same `make` targets).
> **`make ci` is still the local gate** — it runs the same checks via the same
> `make` targets, with one exception: the `k8s` job's targets are not in `make ci`,
> because `helm` is optional for everyone not deploying to Kubernetes. Run
> `make k8s-lint k8s-drift` by hand after touching the chart or the compose file.
## The pipeline
@@ -16,6 +18,8 @@ and CI cannot drift:
| `lint` | `make lint``dotnet format … --verify-no-changes` | .NET 10 SDK |
| `build` | `make build``dotnet build … -c Release` | .NET 10 SDK |
| `unit` | `make unit``dotnet test … -c Release --filter "Category!=Integration"` | .NET 10 SDK |
| `frontend` | `make frontend` → Nx lint/test/build for the four portals | pnpm + Node |
| `k8s` | `make k8s-lint` (render + schema-check the Helm chart) → `make k8s-drift` (chart still describes the same stack as `infra/docker-compose.yml`) | pinned `helm` binary + `docker compose` |
| `mutation` | `make mutation``dotnet tool restore``dotnet stryker` (ACL); uploads the HTML report as an artifact | .NET 10 SDK |
| `verify-stack` | the single live-stack stage — steps: `make verify-up` (full stack up + health, the DoD smoke) → `make verify-acl` (ACL ↔ OpenZaak) → `make verify-nrc` (OpenZaak → NRC delivery) → `make down` | container engine + egress (base images, nuget, `selectielijst.openzaak.nl`) |
+44
View File
@@ -245,3 +245,47 @@ the verify-stack check table, and per-spec e2e results (`infra/playwright-summar
- Getting a report out of the e2e container: Playwright writes `playwright-report.json`
inside the container; `infra/run-e2e-check.sh` `docker cp`s it back to the host
(capturing the test exit code first) so the summary step can read it.
---
## 9. `if: always()` does not survive the job being killed — bound the work itself
`if: always()` makes a step run when an *earlier step failed*. It does **not** help when
the job as a whole is stopped: the run's remaining steps are simply never dispatched.
That is how #161 lost its diagnosis. `verify-stack` entered `make verify-e2e` at 09:48:17
and the job ended at 10:14:54 — 26½ minutes later, mid-suite. Every step after the e2e
shows a **0-second `failure`** stamped at that same instant:
```
14 failure 09:48:17 -> 10:14:54 Self-service e2e (Playwright, login → submit → success)
15 failure 10:14:54 -> 10:14:54 verify-stack check summary ← if: always()
16 failure 10:14:54 -> 10:14:54 e2e spec summary ← if: always()
17 failure 10:14:54 -> 10:14:54 Dump container logs on failure ← if: failure()
18 failure 10:14:54 -> 10:14:54 Tear down ← if: always()
```
So the per-spec summary, the container-log dump and the teardown never ran, and the job
log — which also loses whatever the killed process had buffered — ended at a single `✘`
line. A job that dies takes its own post-mortem with it.
**Read the step timings, not just the log.** `GET /api/v1/repos/{owner}/{repo}/actions/jobs/{id}`
returns every step with `started_at`/`completed_at`; a row of identical zero-length
steps at the end means *killed*, not *silent*. (Job ids come from
`…/actions/runs/{run}/jobs`, and that route returns only the **latest attempt** — a
re-run hides the failed one, so keep the failing job id from the original report. Logs:
`…/actions/jobs/{id}/logs`, see also `gitea-ci-logs`.)
**Conventions that follow:**
- **Bound long-running work inside the tool**, where it can still report. Playwright's
`globalTimeout` (`tests/e2e/playwright.config.ts`) ends the run, writes the JSON
report and exits, so the summary and log-dump steps still get their turn. A
`timeout-minutes` on the job would reproduce the very failure above.
- **Never let an auto-waiting action be the timeout.** Playwright actions (`fill`,
`click`) inherit the *test* timeout, not `expect.timeout`, so a missing element costs
the full 90 s and reports `locator.fill: Test timeout …` — the symptom. Assert the
element visible first with its own budget and a message (`tests/e2e/keycloak-login.ts`).
- Remember `concurrency.cancel-in-progress: true` in `ci.yaml`: a new push to the same
ref, or a re-run, kills the in-flight run the same way. Check `run_attempt` before
concluding a job hung.
+69 -1
View File
@@ -222,6 +222,9 @@ string, so the port the browser uses has to match the one baked into `config.jso
This is the same mechanism `infra/host-browser.yml` uses for the compose stack (which pins
`localhost:8180`); only the addresses differ.
All of this is what §10 removes: with a public domain the portals have real certificates,
so the browser gets its secure context and no forwarding is involved.
### The admin UIs work straight off the NodePorts
These are server-rendered and need no secure context, so they are reachable at the VM's
@@ -322,6 +325,7 @@ The PVCs carry `helm.sh/resource-policy: keep`, so `make k8s-down` leaves the da
```bash
make k8s-lint # render + schema-check the chart, no cluster needed
make k8s-drift # fail if compose and the chart describe different stacks
make k8s-portals # forward the portals + Keycloak to localhost (browser access)
make k8s-images K8S_REGISTRY=... # after changing a service or a portal
make k8s-up TALOS_HOST=... K8S_REGISTRY=...
@@ -359,6 +363,68 @@ immutable, so `helm upgrade` is rejected with `cannot patch "…" with kind Job`
| Pods `Evicted` / `OOMKilled` | the VM is too small (§0) |
| A Job shows `BackoffLimitExceeded` | read it: `kubectl -n big logs job/<name>` |
## 10. Publishing it on a public domain
By default the stack has no hostname: it is reached on NodePorts, and §5's secure-context
problem forces `TALOS_HOST=localhost` plus five SSH forwards. Setting `public.domain` puts a
Caddy deployment in front of it that terminates TLS for real hostnames (ADR-0035), and the
forwards go away.
### Once, outside the cluster
**DNS** — five A records to the *host's* public address (the cluster is behind it):
```
register.<domain> mijn.<domain> behandel.<domain> beheer.<domain> auth.<domain> → 46.224.220.37
```
**The host's forward** — the public IP is on the Fedora host, so it has to hand 80/443 to
the node. This is the only host-side configuration, and it is dumb layer 4:
```bash
sudo firewall-cmd --permanent --zone=public --add-forward-port=port=80:proto=tcp:toaddr=<TALOS_VM_IP>:toport=32080
sudo firewall-cmd --permanent --zone=public --add-forward-port=port=443:proto=tcp:toaddr=<TALOS_VM_IP>:toport=32443
sudo firewall-cmd --permanent --zone=public --add-masquerade
sudo firewall-cmd --reload
```
`--add-masquerade` is what makes the return path work: without it the node answers the
client's address directly and the reply never goes back through the host.
**A StorageClass.** Caddy's certificates live in `/data`, which is an `emptyDir` unless
`persistence.storageClass` is set (§6). Let's Encrypt allows five duplicate certificates per
week, so on an `emptyDir` a handful of pod restarts leaves the edge serving an untrusted
certificate until the limit resets. Install local-path first (§6).
### Deploy
```bash
make k8s-up TALOS_HOST=<domain-facing name> K8S_REGISTRY=<TALOS_VM_IP>:30500 \
K8S_SET='--set public.domain=<domain> --set public.email=<ops address> --set persistence.storageClass=local-path'
```
`public.domain` is the only switch: with it empty nothing in `templates/edge.yaml` renders
and the stack behaves exactly as §4 describes. With it set, `KC_HOSTNAME` and the portals'
`config.json` both become `https://auth.<domain>` — one helper builds both, so the issuer
and the authority cannot drift (ADR-0010).
Watch the first certificate being issued:
```bash
kubectl -n big logs deploy/caddy-edge -f # "certificate obtained successfully"
curl -sSI https://register.<domain>/openbaar/register | head -1
```
### When it doesn't work
| Symptom | Cause |
|---|---|
| ACME fails with `connection refused` or a timeout on the HTTP-01 challenge | the host's 80 → 32080 forward is missing, or `--add-masquerade` is |
| ACME fails with `NXDOMAIN` / `no such host` | the A record isn't there yet. Caddy retries with backoff; fix DNS and it recovers |
| An untrusted certificate after several restarts | the Let's Encrypt duplicate limit, from certificates on an `emptyDir` — see above |
| The portal loads but login bounces back logged out | `public.domain` changed without the portals rolling. The chart hashes the issuer into their pod template, so `helm upgrade` should do it — check `kubectl -n big describe deploy/self-service` |
| `404` from the edge on a name that should work | the name isn't in `public.routes`; Caddy answers 404 for a Host it has no site block for |
## What is not ported
- **Observability** (Tempo, Prometheus, Grafana) is defined but disabled — those are built
@@ -366,5 +432,7 @@ immutable, so `helm upgrade` is rejected with `cannot patch "…" with kind Job`
`K8S_SET='--set workloads.tempo.enabled=true --set workloads.prometheus.enabled=true --set workloads.grafana.enabled=true'`.
The .NET services still export OTLP; the exporter fails harmlessly when Tempo is absent.
- **The verify/e2e lanes.** `make verify*` and the Playwright e2e drive compose, not the
chart. The Kubernetes path is verified with §5's smoke test.
chart. The Kubernetes path is verified with §5's smoke test. CI's `k8s` job runs the two
clusterless checks (`k8s-lint`, `k8s-drift`) on every PR — a values typo or a compose
image bump that skipped the chart fails there, but nothing deploys the chart in CI.
- **Ingress, TLS, and resource requests.** See the ponytail ceiling in ADR-0033.
+30
View File
@@ -0,0 +1,30 @@
#!/usr/bin/env python3
"""Fail when a page under docs/ is missing from mkdocs.yml's nav.
docs/ is the source of truth (CLAUDE.md §12), but only the pages listed in the nav
are published — and mkdocs' own `omitted_files: warn` keeps a build green while
silently dropping them, which is how every ADR after 0010 and every runbook but
ci.md fell off the site.
ponytail: a substring test, not a YAML parse — a page's path either appears in
mkdocs.yml or it doesn't, and that needs no dependency.
"""
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
nav = (ROOT / "mkdocs.yml").read_text()
missing = sorted(
str(page.relative_to(ROOT / "docs"))
for page in (ROOT / "docs").rglob("*.md")
if str(page.relative_to(ROOT / "docs")) not in nav
)
if missing:
print(f"{len(missing)} page(s) under docs/ are not in mkdocs.yml's nav:")
print("\n".join(f" {m}" for m in missing))
sys.exit(1)
print("docs nav complete: every page under docs/ is published")
@@ -135,6 +135,20 @@ cluster-internal hosts ({{ .Release.Namespace }}) and the node address
{{- end }}
{{- end -}}
{{/*
The origin a browser reaches Keycloak on, and so the issuer its tokens carry and
the authority the portals are configured with (ADR-0010). With a public edge that
is the `auth` hostname on `public.domain` — which must stay in step with the `auth`
key in `public.routes`; without one it is the node address plus Keycloak's NodePort.
*/}}
{{- define "big.keycloakUrl" -}}
{{- if .Values.public.domain -}}
https://auth.{{ .Values.public.domain }}
{{- else -}}
http://{{ .Values.host }}:{{ index .Values.nodePorts "keycloak" }}
{{- end -}}
{{- end -}}
{{- define "big.labels" -}}
app.kubernetes.io/name: {{ .name }}
app.kubernetes.io/instance: {{ .root.Release.Name }}
@@ -40,5 +40,5 @@ metadata:
{{- include "big.labels" (dict "root" $ "name" (printf "portal-config-%s" $realm)) | nindent 4 }}
data:
config.json: |
{ "authority": "{{ printf "http://%s:%v" $.Values.host (index $.Values.nodePorts "keycloak") }}/realms/{{ $realm }}" }
{ "authority": "{{ include "big.keycloakUrl" $ }}/realms/{{ $realm }}" }
{{- end }}
@@ -28,7 +28,7 @@ spec:
{{- range $w.files }}
{{- if hasPrefix "portal-config-" .configMap }}
annotations:
checksum/portal-config: {{ printf "%s|%v" $.Values.host (index $.Values.nodePorts "keycloak") | sha256sum }}
checksum/portal-config: {{ include "big.keycloakUrl" $ | sha256sum }}
{{- end }}
{{- end }}
labels:
@@ -0,0 +1,136 @@
{{- /*
The public TLS edge (ADR-0035). Rendered only when `public.domain` is set; with it
empty the stack is reached on the NodePorts below and nothing here exists.
Caddy rather than an ingress controller: the four portals already run caddy:2-alpine,
so this adds no dependency, and it does ACME itself — no cert-manager, no CRDs, no
Ingress objects for five hostnames that never change. It proxies to the ClusterIP
services, so the browser-facing NodePorts are not involved in a public deployment.
The public IP lives on the Fedora host, which forwards 80/443 to the two NodePorts
below. That forward is dumb L4 — no TLS, no routing — see the runbook.
*/}}
{{- if .Values.public.domain }}
{{- $pub := .Values.public }}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: caddy-edge-config
labels:
{{- include "big.labels" (dict "root" $ "name" "caddy-edge") | nindent 4 }}
data:
Caddyfile: |
{
{{- with $pub.email }}
email {{ . }}
{{- end }}
}
{{- range $sub, $target := $pub.routes }}
{{ $sub }}.{{ $pub.domain }} {
reverse_proxy {{ $target }}
}
{{- end }}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: caddy-edge
labels:
{{- include "big.labels" (dict "root" $ "name" "caddy-edge") | nindent 4 }}
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app.kubernetes.io/name: caddy-edge
app.kubernetes.io/instance: {{ .Release.Name }}
template:
metadata:
annotations:
# A ConfigMap mounted with subPath never updates in place, so a changed
# Caddyfile has to roll the pod.
checksum/caddyfile: {{ printf "%s|%v|%v" $pub.domain $pub.email $pub.routes | sha256sum }}
labels:
{{- include "big.labels" (dict "root" $ "name" "caddy-edge") | nindent 8 }}
spec:
containers:
- name: caddy-edge
image: {{ $pub.image }}
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
# TCP, not HTTP: a GET with no matching Host gets a 404 from Caddy, which
# would fail an httpGet probe for a perfectly healthy edge.
readinessProbe:
tcpSocket: { port: 443 }
volumeMounts:
- name: config
mountPath: /etc/caddy/Caddyfile
subPath: Caddyfile
readOnly: true
- name: data
mountPath: /data
- name: run
mountPath: /config
volumes:
- name: config
configMap:
name: caddy-edge-config
- name: run
emptyDir: {}
- name: data
{{- if .Values.persistence.storageClass }}
persistentVolumeClaim:
claimName: caddy-edge-data
{{- else }}
# Certificates live here. On an emptyDir every pod restart asks Let's
# Encrypt again, and its duplicate-certificate limit is five per week —
# set persistence.storageClass for anything that stays up.
emptyDir: {}
{{- end }}
---
apiVersion: v1
kind: Service
metadata:
name: caddy-edge
labels:
{{- include "big.labels" (dict "root" $ "name" "caddy-edge") | nindent 4 }}
spec:
type: NodePort
selector:
app.kubernetes.io/name: caddy-edge
app.kubernetes.io/instance: {{ .Release.Name }}
ports:
- name: http
port: 80
targetPort: 80
nodePort: {{ $pub.nodePorts.http }}
- name: https
port: 443
targetPort: 443
nodePort: {{ $pub.nodePorts.https }}
{{- if .Values.persistence.storageClass }}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: caddy-edge-data
labels:
{{- include "big.labels" (dict "root" $ "name" "caddy-edge") | nindent 4 }}
# Keep the certificates when the release is uninstalled — re-issuing them on
# every reinstall is what burns the rate limit.
annotations:
helm.sh/resource-policy: keep
spec:
accessModes: [ReadWriteOnce]
storageClassName: {{ .Values.persistence.storageClass }}
resources:
requests:
storage: 128Mi
{{- end }}
{{- end }}
+27 -1
View File
@@ -49,6 +49,32 @@ persistence:
# the data across pod restarts.
storageClass: ""
# The public TLS edge (ADR-0035). Empty `domain` = no edge at all: nothing in
# templates/edge.yaml is rendered and the stack is reached on the NodePorts below,
# with `host` above pinning the OIDC origin.
#
# Set it and an in-cluster Caddy terminates TLS for `<sub>.<domain>`, gets its own
# certificates from Let's Encrypt and proxies to the ClusterIP services. The node
# only has to be reachable on the two NodePorts here — the Fedora host forwards
# 80/443 to them (see docs/runbooks/kubernetes-talos.md).
public:
domain: ""
# ACME registration address; Let's Encrypt uses it for expiry warnings.
email: ""
image: docker.io/library/caddy:2-alpine
# <subdomain>: <in-cluster service:port>. `auth` is not free-form — big.keycloakUrl
# builds the pinned issuer from it.
routes:
register: openbaar:80
mijn: self-service:80
behandel: behandel:80
beheer: beheer:80
auth: keycloak:8080
# Where the host's 80/443 forward lands. Not 30080/30443: 30080 is the BFF.
nodePorts:
http: 32080
https: 32443
# The only place a port is published outside the cluster. A workload listed here
# gets a NodePort on its single port; everything else stays ClusterIP.
nodePorts:
@@ -268,7 +294,7 @@ workloads:
# Pin the issuer to the address the browser uses, and let backchannel calls
# keep using keycloak:8080 — the BFF discovers metadata in-cluster and gets
# this issuer back, which is what browser tokens carry (infra/host-browser.yml).
KC_HOSTNAME: "http://{{ .Values.host }}:{{ index .Values.nodePorts \"keycloak\" }}"
KC_HOSTNAME: '{{ include "big.keycloakUrl" . }}'
KC_HOSTNAME_BACKCHANNEL_DYNAMIC: "true"
ports: [{ name: http, port: 8080 }]
# TCP, not /health/ready on the management port: nothing here gates on realm
+116
View File
@@ -0,0 +1,116 @@
#!/usr/bin/env python3
"""Fail when the compose stack and the Helm chart stop describing the same stack.
`infra/docker-compose.yml` is CI-canonical; `infra/helm/big-reference` is a
transcription of it (ADR-0033), and until now nothing kept the two in step an
upstream image bump or a new service applied to only one of them landed
unnoticed. This compares what each side actually *deploys*, not the two files:
the rendered chart against `docker compose config`. Both tools are already
prerequisites of the `k8s-*` make targets.
Run it with `make k8s-drift`. No cluster needed.
ponytail: names and images only, as sets no per-workload env/ports/volumes.
Those differ by design in four documented places (ADR-0033), so comparing them
would mean re-encoding every deviation field by field; a tag bump and a missing
service are the drift that actually bites.
"""
import json
import re
import subprocess
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
COMPOSE = ROOT / "infra/docker-compose.yml"
CHART = ROOT / "infra/helm/big-reference"
# The busybox init container that every `waitFor` workload gets exists only in
# the chart (compose has `depends_on`). Rendering it under a sentinel makes it
# filterable without teaching the check what busybox is.
BUSYBOX = "drift-check-ignored-init-image"
# Differences that Kubernetes forces, not drift (ADR-0033). A name listed here is
# expected to be on exactly one side; anything else fails.
DEVIATIONS = {
# The four Django services apply their own setup_configuration in the web pod
# (`args: [sh, -c, "/setup_configuration.sh && exec /start.sh"]`) rather than in a
# separate init Job. Both that script and /start.sh run `manage.py migrate`, and
# Kubernetes has no `depends_on: service_completed_successfully` to serialise them,
# so the Job and its web pod migrated the same database concurrently.
"oz-init": "folded into the openzaak pod",
"nrc-init": "folded into the nrc-web pod",
"objecttypen-init": "folded into the objecttypen pod",
"objecten-init": "folded into the objecten pod",
# Compose seeds these from the host — the verify scripts `docker cp` the two
# scripts into a running container, and docker-compose.local.yml carries
# `local-seed` + `nrc-subscribe` for `make local`. A cluster has no host to seed
# from, so both became Jobs in the chart.
"seed-zaaktype": "compose seeds the catalogus from the host (infra/openzaak/seed_catalogus.py)",
"nrc-subscribe": "compose registers the abonnement from the host (infra/local/register-abonnement.py)",
}
# Workloads the observability backplane adds. Off by default in both stacks'
# defaults, so they are rendered on purpose here — otherwise their images drift
# unwatched.
OBSERVABILITY = ["tempo", "prometheus", "grafana"]
def compose_services() -> dict[str, str]:
"""Service name -> image, with ${TAG:-default} interpolation already applied."""
out = run(["docker", "compose", "-f", str(COMPOSE), "config", "--format", "json"])
return {name: svc.get("image", "") for name, svc in json.loads(out)["services"].items()}
def chart_workloads() -> dict[str, str]:
"""Workload name -> image, read back out of the rendered manifests."""
out = run(
["helm", "template", "big", str(CHART), "-n", "big", "--set", f"images.busybox={BUSYBOX}"]
+ [f"--set=workloads.{w}.enabled=true" for w in OBSERVABILITY]
)
workloads = {}
for doc in out.split("\n---"):
if not re.search(r"^kind: (Deployment|Job)$", doc, re.M):
continue
name = re.search(r"^ name: (\S+)$", doc, re.M)[1]
images = [i for i in re.findall(r"^\s+image: (\S+)$", doc, re.M) if i != BUSYBOX]
workloads[name] = images[0]
return workloads
def run(argv: list[str]) -> str:
proc = subprocess.run(argv, capture_output=True, text=True)
if proc.returncode != 0:
sys.exit(f"{argv[0]} failed:\n{proc.stderr}")
return proc.stdout
def main() -> int:
compose, chart = compose_services(), chart_workloads()
problems = []
for name in sorted(set(compose) - set(chart) - set(DEVIATIONS)):
problems.append(f" {name}: in docker-compose.yml, not in the chart")
for name in sorted(set(chart) - set(compose) - set(DEVIATIONS)):
problems.append(f" {name}: in the chart, not in docker-compose.yml")
for name in sorted(set(compose) & set(chart)):
if compose[name] != chart[name]:
problems.append(f" {name}: compose runs {compose[name]}, the chart runs {chart[name]}")
if problems:
print("compose and the Helm chart describe different stacks:\n" + "\n".join(problems))
print(
"\nPort the change to the other stack, or — if the difference is forced by\n"
"Kubernetes — declare it in DEVIATIONS in this file, with the reason."
)
return 1
print(f"no drift: {len(chart)} workloads, images identical on both stacks")
for name, why in sorted(DEVIATIONS.items()):
print(f" deviation (declared): {name}{why}")
return 0
if __name__ == "__main__":
sys.exit(main())
+80
View File
@@ -0,0 +1,80 @@
#!/usr/bin/env python3
"""Fail when the pinned issuer and the portals' OIDC authority stop agreeing.
Keycloak pins one issuer (`KC_HOSTNAME`) and each portal is configured with one
authority (`config.json`). A browser token carries the first; the BFF validates
against what it discovers from the second (ADR-0010). When the two drift the
symptom is three services away a login that bounces back logged out, or a 401
from the BFF so the chart builds both from one helper and this asserts it.
It also pins the two halves of the public edge (ADR-0035): that setting
`public.domain` actually publishes the hostnames, and that leaving it empty
renders no edge at all, which is what compose, CI and a laptop cluster rely on.
Run it with `make k8s-lint`. No cluster needed.
"""
import json
import re
import subprocess
import sys
from pathlib import Path
CHART = Path(__file__).resolve().parent / "big-reference"
DOMAIN = "example.test"
def render(*sets: str) -> str:
argv = ["helm", "template", "big", str(CHART), "-n", "big"]
for s in sets:
argv += ["--set", s]
proc = subprocess.run(argv, capture_output=True, text=True)
if proc.returncode != 0:
sys.exit(f"helm template failed:\n{proc.stderr}")
return proc.stdout
def issuer(out: str) -> str:
"""The value of KC_HOSTNAME in the rendered manifests."""
m = re.search(r"name: KC_HOSTNAME\n\s+value: \"(\S+)\"", out)
return m[1] if m else ""
def authorities(out: str) -> set[str]:
"""Every portal's OIDC authority, with the realm path stripped."""
found = set()
for line in re.findall(r'\{ "authority": .* \}', out):
url = json.loads(line)["authority"]
found.add(url.rsplit("/realms/", 1)[0])
return found
def main() -> int:
problems = []
public = render(f"public.domain={DOMAIN}")
if issuer(public) != f"https://auth.{DOMAIN}":
problems.append(f" with public.domain set, KC_HOSTNAME is {issuer(public)!r}, not https://auth.{DOMAIN}")
if authorities(public) != {f"https://auth.{DOMAIN}"}:
problems.append(f" with public.domain set, the portals point at {sorted(authorities(public))}")
for host in (f"register.{DOMAIN}", f"mijn.{DOMAIN}", f"behandel.{DOMAIN}", f"beheer.{DOMAIN}", f"auth.{DOMAIN}"):
if host not in public:
problems.append(f" {host} is not published by the edge")
private = render()
if authorities(private) != {issuer(private)}:
problems.append(f" by default the portals point at {sorted(authorities(private))}, the issuer is {issuer(private)!r}")
if "caddy-edge" in private:
problems.append(" the edge renders with no public.domain — compose, CI and a laptop cluster expect nothing")
if problems:
print("the chart's OIDC origin is inconsistent:\n" + "\n".join(problems))
print("\nBoth halves come from the `big.keycloakUrl` helper — change it, not one caller.")
return 1
print(f"issuer + portal authority agree, with and without a public domain")
return 0
if __name__ == "__main__":
raise SystemExit(main())
+20
View File
@@ -0,0 +1,20 @@
# Overlay: make the CI compose stack usable from a HOST browser.
# Same two mechanisms infra/docker-compose.local.yml already uses — pin Keycloak's issuer to the
# host-published address, and point each portal's runtime config.json at it. The BFF needs no
# change: it discovers metadata over keycloak:8080 and the discovered issuer is the pinned
# localhost:8180, which is what browser tokens carry.
services:
keycloak:
environment:
KC_HOSTNAME: http://localhost:8180
KC_HOSTNAME_BACKCHANNEL_DYNAMIC: "true"
self-service:
volumes:
- ./local-config/self-service.config.json:/usr/share/caddy/config.json:ro,z
behandel:
volumes:
- ./local-config/behandel.config.json:/usr/share/caddy/config.json:ro,z
# beheer is the same medewerker realm as behandel, so it reuses behandel's config verbatim.
beheer:
volumes:
- ./local-config/behandel.config.json:/usr/share/caddy/config.json:ro,z
+37 -5
View File
@@ -7,10 +7,34 @@ redirects it into $GITHUB_STEP_SUMMARY. Stdlib only.
"""
import json
import os
import re
import sys
STATUS_ICON = {"expected": "", "unexpected": "", "skipped": "⏭️", "flaky": "⚠️"}
# A verdict alone still costs a log dive, and a killed or truncated job leaves no log to dive into
# (#161) — so a failing spec carries its first error into the table. Playwright errors are multi-line
# with a "Call log:", which a markdown table cell cannot hold, so they are flattened and clipped.
ERROR_CLIP = 300
def first_error(spec):
"""The first error message across a spec's test results, flattened for one table cell."""
for test in spec.get("tests", []):
for result in test.get("results", []):
for error in result.get("errors", []):
message = (error.get("message") or "").strip()
if not message:
continue
# Strip ANSI colour, collapse to one line, and keep it inside the cell.
message = re.sub(r"\x1b\[[0-9;]*m", "", message)
message = " ".join(message.split())
if len(message) > ERROR_CLIP:
message = message[:ERROR_CLIP - 1].rstrip() + ""
# `|` would end the cell early.
return message.replace("|", "\\|")
return ""
def walk(suite, out):
for spec in suite.get("specs", []):
@@ -22,7 +46,8 @@ def walk(suite, out):
else "expected" if spec.get("ok", False)
else "unexpected")
out.append({"file": spec.get("file") or suite.get("file") or suite.get("title", ""),
"title": spec.get("title", ""), "status": status})
"title": spec.get("title", ""), "status": status,
"error": first_error(spec) if status in ("unexpected", "flaky") else ""})
for child in suite.get("suites", []):
walk(child, out)
@@ -46,10 +71,17 @@ def main(path):
if not specs:
print("_No specs ran._")
return 0
print("| Spec | Result |")
print("| ---- | :----: |")
for s in specs:
print(f"| {s['file']} {s['title']} | {STATUS_ICON.get(s['status'], '')} |")
# The failure column only earns its width when something failed.
if any(s["error"] for s in specs):
print("| Spec | Result | Why |")
print("| ---- | :----: | --- |")
for s in specs:
print(f"| {s['file']} {s['title']} | {STATUS_ICON.get(s['status'], '')} | {s['error']} |")
else:
print("| Spec | Result |")
print("| ---- | :----: |")
for s in specs:
print(f"| {s['file']} {s['title']} | {STATUS_ICON.get(s['status'], '')} |")
return 0
+108
View File
@@ -0,0 +1,108 @@
#!/usr/bin/env python3
"""Self-check for infra/playwright-summary.py — stdlib asserts, no framework.
Run: python3 infra/test_playwright_summary.py (also runs in `make unit`).
A red e2e is only useful if the job summary says WHY it failed: #161 lost a 36-minute
verify-stack job whose only surviving output was one line with no assertion detail.
"""
import importlib.util
import io
import json
import os
import tempfile
from contextlib import redirect_stdout
# The script's filename is not a valid module name, so load it by path.
spec = importlib.util.spec_from_file_location(
"playwright_summary",
os.path.join(os.path.dirname(os.path.abspath(__file__)), "playwright-summary.py"),
)
summary = importlib.util.module_from_spec(spec)
spec.loader.exec_module(summary)
def render(report):
"""Run the renderer over a report dict and return its markdown."""
with tempfile.NamedTemporaryFile("w", suffix=".json", delete=False) as fh:
json.dump(report, fh)
path = fh.name
try:
out = io.StringIO()
with redirect_stdout(out):
summary.main(path)
return out.getvalue()
finally:
os.unlink(path)
def spec_entry(title, status, errors=()):
return {
"title": title,
"file": "catalogus.spec.ts",
"ok": status == "expected",
"tests": [{"status": status, "results": [{"errors": [{"message": m} for m in errors]}]}],
}
def test_failing_spec_reports_why():
md = render({
"stats": {"expected": 4, "unexpected": 1, "flaky": 0, "skipped": 0, "duration": 108_000},
"suites": [{"file": "catalogus.spec.ts", "specs": [
spec_entry("a beheerder sees the published zaaktypen in the catalogus", "unexpected",
["locator.fill: Test timeout of 90000ms exceeded.\n"
"Call log:\n - waiting for locator('#username')\n"]),
]}],
})
assert "" in md, md
# The point of the slice: the summary names the cause, not just the verdict.
assert "Test timeout of 90000ms exceeded" in md, md
assert "waiting for locator('#username')" in md, md
# A multi-line Playwright error must not break out of its table row.
assert not any(line.startswith("Call log:") for line in md.splitlines()), md
def test_real_playwright_error_is_flattened():
# A real report's message is multi-line and ANSI-coloured, and embeds the source snippet with
# `|` gutters — all three would break the table cell. Shape verified against an actual
# @playwright/test 1.61 JSON report.
md = render({
"stats": {"expected": 0, "unexpected": 1, "flaky": 0, "skipped": 0, "duration": 1_000},
"suites": [{"file": "catalogus.spec.ts", "specs": [
spec_entry("a beheerder sees the catalogus", "unexpected",
["Error: expect(locator).toBeVisible() failed\n\n"
"\x1b[2mLocator: \x1b[22mgetByRole('heading')\n"
" 12 | await login(page);\n> 13 | await expect(heading).toBeVisible();\n"]),
]}],
})
row = [line for line in md.splitlines() if line.startswith("| catalogus.spec.ts")][0]
assert "\x1b" not in row, row
assert "Locator: getByRole('heading')" in row, row
# Every literal `|` from the snippet gutters is escaped, so the row keeps exactly 3 cells.
assert row.count("|") - row.count("\\|") == 4, row
def test_passing_run_stays_quiet():
md = render({
"stats": {"expected": 1, "unexpected": 0, "flaky": 0, "skipped": 0, "duration": 5_000},
"suites": [{"file": "catalogus.spec.ts",
"specs": [spec_entry("a beheerder sees the catalogus", "expected")]}],
})
assert "" in md, md
assert "timeout" not in md.lower(), md
def test_missing_report_is_not_a_crash():
out = io.StringIO()
with redirect_stdout(out):
rc = summary.main("/nonexistent/playwright-report.json")
assert rc == 0
assert "did not reach the e2e step" in out.getvalue()
if __name__ == "__main__":
for name, fn in sorted(globals().items()):
if name.startswith("test_") and callable(fn):
fn()
print(f" ok {name}")
print("playwright-summary self-check passed")
+32
View File
@@ -32,6 +32,31 @@ nav:
- "ADR-0008: Read projection store": architecture/adr-0008-read-projection-store.md
- "ADR-0009: External-task job worker": architecture/adr-0009-external-task-job-worker.md
- "ADR-0010: BFF OIDC validation": architecture/adr-0010-bff-oidc.md
- "ADR-0011: Approval status flow": architecture/adr-0011-approval-status-flow.md
- "ADR-0012: Citizen reference correlation": architecture/adr-0012-citizen-reference-correlation.md
- "ADR-0013: Behandel-portal wiring": architecture/adr-0013-behandel-portal-wiring.md
- "ADR-0014: Withdrawal cancels the process": architecture/adr-0014-withdrawal-cancels-the-process.md
- "ADR-0015: Beoordeling escalation": architecture/adr-0015-beoordeling-escalation.md
- "ADR-0016: Diploma eligibility DMN": architecture/adr-0016-diploma-eligibility-dmn.md
- "ADR-0017: Document-wait timeout": architecture/adr-0017-document-wait-timeout-cancellation.md
- "ADR-0018: Diploma upload via the ACL": architecture/adr-0018-diploma-upload-via-acl-documenten.md
- "ADR-0019: Zaak cancellation on timeout": architecture/adr-0019-zaak-cancellation-on-timeout.md
- "ADR-0020: Local stack self-seeds": architecture/adr-0020-local-stack-self-seeds.md
- "ADR-0021: Zaaktype by identificatie": architecture/adr-0021-acl-resolves-zaaktype-by-identificatie.md
- "ADR-0022: Quartz scheduler": architecture/adr-0022-quartz-scheduler.md
- "ADR-0023: Observability stack": architecture/adr-0023-observability-stack.md
- "ADR-0024: Prometheus AspNetCore exporter": architecture/adr-0024-prometheus-aspnetcore-exporter.md
- "ADR-0025: BFF reads catalogus via the ACL": architecture/adr-0025-bff-reads-catalogus-via-acl.md
- "ADR-0026: Mutable default-fill store": architecture/adr-0026-mutable-default-fill-store.md
- "ADR-0027: RegisterRecord objecttype": architecture/adr-0027-registerrecord-objecttype-schema.md
- "ADR-0028: Objecten holds the register": architecture/adr-0028-objecten-holds-the-register.md
- "ADR-0029: Objecten publishes to NRC": architecture/adr-0029-objecten-publishes-to-nrc.md
- "ADR-0030: Projection sourced from the register": architecture/adr-0030-projection-sourced-from-the-register.md
- "ADR-0031: MFA on the medewerker realm": architecture/adr-0031-mfa-on-the-medewerker-realm.md
- "ADR-0032: Werkbak live refresh": architecture/adr-0032-werkbak-live-refresh.md
- "ADR-0033: Kubernetes via one Helm chart": architecture/adr-0033-kubernetes-via-one-helm-chart.md
- "ADR-0034: Caddy serves the portals": architecture/adr-0034-caddy-serves-the-portals.md
- "ADR-0035: Public TLS edge in the cluster": architecture/adr-0035-public-tls-edge-in-cluster.md
- FDS-architectuur:
- Overzicht: architecture/fds/README.md
- Componentview (L3): architecture/fds/c4-component-view.md
@@ -46,8 +71,15 @@ nav:
- Working in Gitea: gitea-workflow.md
- Frontend decisions: frontend-decisions.md
- Demo script: demo-script.md
- Synthetic data: synthetic-data.md
- Runbooks:
- CI: runbooks/ci.md
- OpenZaak: runbooks/openzaak.md
- Open Notificaties (NRC): runbooks/opennotificaties.md
- Keycloak: runbooks/keycloak.md
- Flowable: runbooks/flowable.md
- Kubernetes on Talos: runbooks/kubernetes-talos.md
- Gitea Actions gotchas: runbooks/gitea-actions-gotchas.md
markdown_extensions:
- admonition
+1 -1
View File
@@ -1,5 +1,5 @@
import { expect, test } from '@playwright/test';
import { loginMedewerker } from './medewerker-login';
import { loginMedewerker } from './keycloak-login';
// S-15a walking skeleton: a beheerder logs in to the beheer portal (medewerker realm) and sees the
// read-only ZTC catalogus. The verify stack seeds and publishes the BIG-REGISTRATIE zaaktype (the
+1 -1
View File
@@ -1,5 +1,5 @@
import { expect, test } from '@playwright/test';
import { loginMedewerker } from './medewerker-login';
import { loginMedewerker } from './keycloak-login';
// S-15b: a beheerder edits the ACL default-fill in the beheer portal and gets a saved confirmation.
// Runs against the shared verify stack; it edits + saves (the ACL store is in-memory, ADR-0026) and
@@ -1,5 +1,5 @@
import { expect, test } from '@playwright/test';
import { OTP_PERIOD_MS, nextUnusedCounter } from './medewerker-login';
import { OTP_PERIOD_MS, nextUnusedCounter } from './keycloak-login';
// Pure check of the TOTP counter guard in loginMedewerker — no browser, no stack. Keycloak refuses
// a code it has already accepted (its otpPolicyCodeReusable defaults to false), so two logins as
+99
View File
@@ -0,0 +1,99 @@
import { createHmac } from 'node:crypto';
import { readFileSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { expect, type Page } from '@playwright/test';
// Every portal login in the suite goes through this module — citizen realms (mock DigiD) and the
// medewerker realm alike — so the shared Keycloak form handling lives in exactly one place.
// The medewerker realm enforces MFA (S-15c), so a staff login is two steps: password, then a TOTP
// code. The realm export seeds every medewerker with this fixture secret — Keycloak HMACs the raw
// secret bytes — so the e2e can compute a valid code instead of enrolling an authenticator.
const OTP_SECRET = 'BIGMEDEWERKEROTPSEED';
export const OTP_PERIOD_MS = 30_000;
/**
* How long a Keycloak form gets to appear. Generous enough for a cold first browser launch and a
* loaded stack, far short of the 90-second test timeout an auto-waiting action would otherwise eat.
*/
const FORM_TIMEOUT_MS = 20_000;
const FORM_NEVER_APPEARED =
'the Keycloak login form never appeared — the portal did not reach Keycloak (check its ' +
'config.json fetch and the OIDC discovery on the authority it was built with)';
const OTP_NEVER_APPEARED =
'the Keycloak OTP form never appeared — the password step did not complete (check the ' +
'medewerker realm seeded this user with both a password and a TOTP credential)';
// RFC 6238 TOTP: HMAC-SHA1 over the 30-second counter, dynamically truncated to 6 digits.
export function totp(secret = OTP_SECRET, at = Date.now()): string {
const counter = Buffer.alloc(8);
counter.writeBigUInt64BE(BigInt(Math.floor(at / OTP_PERIOD_MS)));
const mac = createHmac('sha1', secret).update(counter).digest();
const offset = mac[mac.length - 1] & 0x0f;
return String((mac.readUInt32BE(offset) & 0x7fffffff) % 1_000_000).padStart(6, '0');
}
// Keycloak refuses a TOTP code it has already accepted (its otpPolicyCodeReusable defaults to
// false), so two logins as the same medewerker inside one 30-second window would both submit the
// same code and the second is rejected. Spend the first counter this medewerker has left.
export function nextUnusedCounter(now: number, spent: number): number {
return Math.max(Math.floor(now / OTP_PERIOD_MS), spent + 1);
}
// The spent counter lives on disk rather than in module state: Playwright starts a fresh worker
// process for a retry, which would otherwise forget it and resubmit the rejected code.
function spendCounter(username: string): number {
const file = join(tmpdir(), `otp-counter-${username}`);
let spent = -1;
try {
spent = Number(readFileSync(file, 'utf8')) || -1;
} catch {
// first login as this medewerker in this run
}
const counter = nextUnusedCounter(Date.now(), spent);
writeFileSync(file, String(counter));
return counter;
}
/**
* Fill Keycloak's login form. Every portal is guarded, so the first navigation redirects here; the
* form ids are stable across themes.
*
* The form is asserted visible *before* it is filled. A portal that never reaches Keycloak its
* runtime `config.json` fetch or the OIDC discovery behind `authorize()` failed, so it never
* bootstrapped and shows a blank page (main.ts only logs to the console) would otherwise leave
* `fill()` auto-waiting until the whole test times out: 90 seconds spent to report
* `locator.fill: Test timeout of 90000ms exceeded`, naming the symptom and not the cause. That is
* how #161's catalogus.spec burned 1.8 minutes. This fails in a quarter of the time and says which
* step never happened.
*/
async function submitPassword(page: Page, username: string): Promise<void> {
await expect(page.locator('#username'), FORM_NEVER_APPEARED).toBeVisible({ timeout: FORM_TIMEOUT_MS });
await page.locator('#username').fill(username);
await page.locator('#password').fill('test123');
await page.locator('#kc-login').click();
}
/** A citizen login on a mock-DigiD realm — no second factor (ADR-0031). */
export async function loginBurger(page: Page, username: string): Promise<void> {
await submitPassword(page, username);
}
/** A staff login on the medewerker realm: password, then the enforced TOTP second factor. */
export async function loginMedewerker(page: Page, username: string): Promise<void> {
await submitPassword(page, username);
// Keycloak's conditional-OTP step. Same reasoning as the password form above: assert it arrived
// rather than letting `fill()` swallow the test timeout.
await expect(page.locator('#otp'), OTP_NEVER_APPEARED).toBeVisible({ timeout: FORM_TIMEOUT_MS });
// Wait out the rest of the window if the counter we may spend is still in the future; Keycloak's
// lookAheadWindow would accept the code a moment early, but only by one counter — waiting keeps a
// third login in the same window valid too.
const counter = spendCounter(username);
await page.waitForTimeout(Math.max(0, counter * OTP_PERIOD_MS - Date.now()));
await page.locator('#otp').fill(totp(OTP_SECRET, counter * OTP_PERIOD_MS));
await page.locator('#kc-login').click();
}
-57
View File
@@ -1,57 +0,0 @@
import { createHmac } from 'node:crypto';
import { readFileSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import type { Page } from '@playwright/test';
// The medewerker realm enforces MFA (S-15c), so a staff login is two steps: password, then a TOTP
// code. The realm export seeds every medewerker with this fixture secret — Keycloak HMACs the raw
// secret bytes — so the e2e can compute a valid code instead of enrolling an authenticator.
const OTP_SECRET = 'BIGMEDEWERKEROTPSEED';
export const OTP_PERIOD_MS = 30_000;
// RFC 6238 TOTP: HMAC-SHA1 over the 30-second counter, dynamically truncated to 6 digits.
export function totp(secret = OTP_SECRET, at = Date.now()): string {
const counter = Buffer.alloc(8);
counter.writeBigUInt64BE(BigInt(Math.floor(at / OTP_PERIOD_MS)));
const mac = createHmac('sha1', secret).update(counter).digest();
const offset = mac[mac.length - 1] & 0x0f;
return String((mac.readUInt32BE(offset) & 0x7fffffff) % 1_000_000).padStart(6, '0');
}
// Keycloak refuses a TOTP code it has already accepted (its otpPolicyCodeReusable defaults to
// false), so two logins as the same medewerker inside one 30-second window would both submit the
// same code and the second is rejected. Spend the first counter this medewerker has left.
export function nextUnusedCounter(now: number, spent: number): number {
return Math.max(Math.floor(now / OTP_PERIOD_MS), spent + 1);
}
// The spent counter lives on disk rather than in module state: Playwright starts a fresh worker
// process for a retry, which would otherwise forget it and resubmit the rejected code.
function spendCounter(username: string): number {
const file = join(tmpdir(), `otp-counter-${username}`);
let spent = -1;
try {
spent = Number(readFileSync(file, 'utf8')) || -1;
} catch {
// first login as this medewerker in this run
}
const counter = nextUnusedCounter(Date.now(), spent);
writeFileSync(file, String(counter));
return counter;
}
export async function loginMedewerker(page: Page, username: string): Promise<void> {
await page.locator('#username').fill(username);
await page.locator('#password').fill('test123');
await page.locator('#kc-login').click();
// Keycloak's conditional-OTP step. Wait out the rest of the window if the counter we may spend is
// still in the future; its lookAheadWindow would accept the code a moment early, but only by one
// counter — waiting keeps a third login in the same window valid too.
const counter = spendCounter(username);
await page.waitForTimeout(Math.max(0, counter * OTP_PERIOD_MS - Date.now()));
await page.locator('#otp').fill(totp(OTP_SECRET, counter * OTP_PERIOD_MS));
await page.locator('#kc-login').click();
}
+6
View File
@@ -15,6 +15,12 @@ export default defineConfig({
timeout: 90_000,
expect: { timeout: 15_000 },
retries: 1,
// Bound the whole run, not just each test (#161). A wedged suite used to run until CI killed the
// job — which also killed the `if: always()` steps that would have said why: the per-spec summary
// and the container-log dump never ran, leaving a 36-minute job whose entire surviving output was
// one ✘ line. On `globalTimeout` Playwright stops and *reports*, so the JSON report is written and
// those steps still run. Generous over the ~1-minute suite: this is a backstop, not a budget.
globalTimeout: 12 * 60_000,
// Run the specs serially. Each spec drives a full `channel: 'chromium'` browser, and the e2e
// shares an 8 GB runner with the entire compose stack (OpenZaak, NRC, Keycloak, Flowable, 4×
// Postgres, every service + 3 portals). Two parallel browsers exhaust memory and the renderer is
+2 -4
View File
@@ -1,5 +1,5 @@
import { expect, request, test } from '@playwright/test';
import { loginMedewerker } from './medewerker-login';
import { loginBurger, loginMedewerker } from './keycloak-login';
// Walking-skeleton happy path (S-08d + S-09 + S-09b + S-12 + S-10a + S-19b-2): a zorgprofessional
// logs in via mock DigiD and submits through the self-service portal → BFF → domain; the entry
@@ -23,9 +23,7 @@ test('DigiD submit → public INGEDIEND → documenten → behandelaar goedkeurt
// checks submit as jan-burger (bsn 123456782) before the e2e runs on the shared stack, and
// resume-on-load (S-26) would otherwise restore one of those on login — so each self-service spec
// uses a dedicated citizen no other actor touches.
await page.locator('#username').fill('emma-burger');
await page.locator('#password').fill('test123');
await page.locator('#kc-login').click();
await loginBurger(page, 'emma-burger');
// Back on the portal, authenticated.
await expect(page.getByRole('heading', { name: /Zelfservice/i })).toBeVisible();
+2 -3
View File
@@ -1,4 +1,5 @@
import { expect, test } from '@playwright/test';
import { loginBurger } from './keycloak-login';
// S-26: a zorgprofessional submits, then reloads the self-service portal. On load the portal asks the
// BFF for the caller's current open registration (owner-scoped by the DigiD token's bsn) and restores
@@ -9,9 +10,7 @@ test('DigiD submit → reload → self-service restores the existing registratio
// Its own DigiD user (like every self-service spec): on the shared verify stack, resume-on-load
// (S-26) restores any open registration for the bsn, so each spec uses a dedicated citizen that no
// other spec or verify-* check touches. This one in particular leaves an open registration.
await page.locator('#username').fill('sanne-burger');
await page.locator('#password').fill('test123');
await page.locator('#kc-login').click();
await loginBurger(page, 'sanne-burger');
await expect(page.getByRole('heading', { name: /Zelfservice/i })).toBeVisible();
await page.getByRole('button', { name: /indienen/i }).click();
+2 -3
View File
@@ -1,4 +1,5 @@
import { expect, test } from '@playwright/test';
import { loginBurger } from './keycloak-login';
// S-11 (Flow 3): a zorgprofessional logs in via mock DigiD, submits a registration, then withdraws
// it ("trek aanvraag in") from the self-service portal. The withdrawal goes portal → BFF (owner-
@@ -10,9 +11,7 @@ test('DigiD submit → trek aanvraag in → self-service confirms ingetrokken',
// Its own DigiD user — isolated from the verify-* checks (jan-burger/123456782) so resume-on-load
// (S-26) can't restore someone else's registration on the shared stack.
await page.locator('#username').fill('lars-burger');
await page.locator('#password').fill('test123');
await page.locator('#kc-login').click();
await loginBurger(page, 'lars-burger');
await expect(page.getByRole('heading', { name: /Zelfservice/i })).toBeVisible();