not
94742a261f
## What & why S-19b-2, closing out ADR-0028's stated direction: **the read projection is now derived from the `RegisterRecord` in Objecten, not from ZGW zaak events.** Until now the subscriber listened on `zaken` and *inferred* register state from case events — a `zaak/create` meant INGEDIEND, and any `status/create` was assumed to be the approval (it may not read OpenZaak, so it could not tell statustypen apart). The reference wasn't in the notification at all, so every projection made a second hop to the ACL. The register — a fact about a person — was being reconstructed by guessing at the lifecycle of the case that produced it. - The subscriber's abonnement moves to the `objecten` kanaal (S-19b-1 made it publish). - An Objecten notification carries **no record data**, only the object URL, so the record is read back through the ACL (`POST /register-records/read`) — §8.1 applies to Objecten exactly as ADR-0028 established. - The record carries `id`, `status` and `reference`, so the row *is* the record: `IsZaakCreated`, `IsZaakStatusSet`, `ZaakUrl`, `ZaakId` and `ToEntry`'s `Resource == "status"` inference are all gone, and so is the ACL enrichment hop. - **The ACL now writes an INGEDIEND record on submit.** Without it, re-sourcing would silently drop every submitted registration from the public register, since only approval wrote a record. - `processed_notifications` holds the projected row (`register_id`, `status`, `reference`) instead of the ZGW event, so a rebuild is a replay with no mapping rules and no upstream reads at all. **ADR-0030** records it. ADR-0028's open caveat — record written but not yet read, "the two must agree" — is closed: there is one source now. Closes #153 ## Definition of Done - [x] Linked Gitea issue (above). - [x] Failing tests committed before the implementation — two red/green pairs, ACL side (06c0444→566ef7d) and subscriber side (142ed45→8af09b2). - [x] Refactor commit follows (b496ac9). - [x] Conventional Commits referencing the issue (`refs #153`). - [x] CI green — all six jobs onb30fa66, `verify-stack` end to end including the e2e. - [x] `docker compose up` from a fresh clone reaches green health checks within 3 minutes (`verify-stack`'s bring-up step — see the wait-healthy fix below). - [x] Docs updated — ADR-0030 added, ADR-0028's consequence + caveat annotated, BACKLOG.md, e2e header comment. - [x] ADR added in `docs/architecture/`. - [x] Demo note in `docs/demo-script.md` — n/a: no user-visible change. The openbaar register shows the same two statuses for the same registrations; only where they come from changed. ## Notes for reviewers **The decision I'd most like a second opinion on** is the one the issue didn't settle: what happens to INGEDIEND. Objecten held only INGESCHREVEN records, so re-sourcing forced a choice between (a) the ACL also writing on submit, (b) a public register that lists only actual registrations, or (c) a hybrid keeping both kanalen. I took (a): visible behaviour is unchanged and the register holds the whole lifecycle. (b) is arguably the better *semantics* for a public register but narrows what the portal shows and reads against PRD §68 ("~50 register entries with diverse statuses"); (c) leaves the projection half-derived from ZGW, which is the coupling ADR-0028 set out to remove. All three are laid out in ADR-0030. **The dedup key is the projected row**, `objecten:object:{url}:{status}:{reference}` — not the object URL (the ACL upserts *one object per registration*, so submit and approval notify about the same URL and the approval would be swallowed as a duplicate) and not URL+actie (a retried approval is a second `update`). Redeliveries collapse, genuine state changes don't. §8.6. **The migration drops columns rather than renaming them.** EF scaffolded renames — `resource` → `register_id`, `zaak_id` → `status` — which would have carried ZGW values into columns meaning something else, and a rebuild would then have projected that garbage. It also empties both tables: a pre-slice row describes a zaak event the new projector can't reproject, and those registrations have no RegisterRecord in Objecten either, so they're not re-derivable from the new source. Stated as a ceiling in the ADR — fine while stacks are ephemeral, backfill from Objecten if a long-lived environment ever needs it. **`run-projection-check.sh` now opens its zaak through the ACL** instead of straight against OpenZaak, because the ACL is what writes the record. A zaak created behind the ACL's back produces no projection row — that's the re-source working, not a gap. ## Three fixes CI found, none of them in the projection logic 1. **`wait-healthy.sh` matched the wrong container** (744f91a). Bring-up timed out with `TIMEOUT: 'objecten' not healthy (status=none)` while the `docker ps` it dumps showed objecten `Up 9 minutes (healthy)`. `--filter name=` is a substring match, so `objecten` also matches `objecten-db`/`objecten-redis`/`objecten-celery`, and `head -1` took whichever docker listed first — the celery worker has no healthcheck, hence `status=none`. Latent since those services landed and decided purely by listing order; `objecttypen` matches `objecttypen-db` the same way. Anchored on the compose replica suffix, which the verify scripts already do. 2. **The ACL had to be repointed at OpenZaak's IP** (7e0897a). Opening the zaak through the ACL put this check in the same bind run-domain-check.sh already handles: `400 {"name":"zaaktype","code":"bad-url","reason":"Voer een geldige URL in."}`. OpenZaak reflects the request Host into the zaaktype URL and then rejects it on zaak-create when single-label — the mechanism compose already documents on `ACL_OPENZAAK_BASEURL`. 3. **Approval arrives as `partial_update`, not `update`** (0dd26a7→b30fa66) — the one real bug in the slice. The ACL upserts with PATCH; DRF routes it through the notifying `update()` but names the action `partial_update`, so the projector dropped every approval. Only the e2e could catch it: `verify-projection` drives a submit, and per ADR-0028 the e2e is the only check that drives a *real* approval. `verify-tracing` also failed once (run 722) on a path this PR doesn't touch, and passed on a plain re-run of the same commit. Tempo logged `pusher failed to consume trace data` / `distributor_pool failing healthcheck` — it dropped spans under runner load rather than the trace chain being broken. Filed as **#156** rather than absorbed here. **Correction to the #152 PR notes:** I wrote there that celery concurrency was "the next knob" if verify-stack got tight. It isn't — `CELERY_WORKER_CONCURRENCY` already defaults to 1 in the Maykin image, so `objecten-celery` is already a single-process worker. Noted in #156. **Possible follow-up, deliberately not done here:** an `openzaak.local` network alias mirroring `objecten.local` would remove the ACL-repoint dance from both run-domain-check.sh and run-projection-check.sh. It changes the host in every zaak URL the system produces, which is too broad a ripple to land inside an unrelated slice — worth its own issue. **Known costs, all in the ADR:** submission is now two writes across two modules and eventually consistent (same posture ADR-0028 accepted for approval); projecting now depends on the ACL being reachable on the main path, not just for enrichment (NRC retries, so it converges); and OpenZaak still publishes to `zaken` with nothing in the product listening — kept because `verify-nrc` asserts that path.Reviewed-on: #155
101 lines
5.7 KiB
Bash
Executable File
101 lines
5.7 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
#
|
|
# Verify the end-to-end read-projection path (S-06, re-sourced by S-19b-2) against an ALREADY-RUNNING
|
|
# full stack: ACL → Objecten → NRC → Event Subscriber → projection → projection-api. Seeds a
|
|
# published BIG zaaktype (idempotent), registers an abonnement on the `objecten` kanaal pointing at
|
|
# the real Event Subscriber's /notifications callback (with the bearer it enforces), opens a zaak
|
|
# *through the ACL*, and asserts projection-api serves a row for it with status INGEDIEND.
|
|
#
|
|
# The zaak is opened through the ACL, not straight against OpenZaak: since ADR-0030 the projection is
|
|
# derived from the RegisterRecord in Objecten, and the ACL is what writes that record (INGEDIEND on
|
|
# submit). A zaak created behind the ACL's back produces no register write and so no projection row —
|
|
# which is the point of the re-source.
|
|
#
|
|
# All in-network, reaching services by container IP — single-label hosts aren't URL-valid and
|
|
# the runner can't reach published ports (gitea-actions-gotchas.md §5/§6). Does not own the stack
|
|
# lifecycle (the caller brings it up and tears it down), but does recreate the `acl` service to
|
|
# repoint it — see below, and run-domain-check.sh, which does the same. Plain docker primitives only.
|
|
# See ADR-0007/0008/0030.
|
|
set -euo pipefail
|
|
|
|
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
root="$(cd "$here/.." && pwd)"
|
|
compose="$root/infra/docker-compose.yml"
|
|
WEBHOOK_AUTH="${NOTIFICATION_WEBHOOK_TOKEN:-Bearer big-reference-notifications}"
|
|
|
|
cleanup() { docker rm -f rr-pverify rr-pquery >/dev/null 2>&1 || true; }
|
|
trap cleanup EXIT
|
|
|
|
ip() { docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$1"; }
|
|
|
|
oz="$(docker ps -q --filter 'name=[-_]openzaak[-_]' | head -1)"
|
|
nrc="$(docker ps -q --filter 'name=nrc-web' | head -1)"
|
|
es="$(docker ps -q --filter 'name=event-subscriber' | head -1)"
|
|
proj="$(docker ps -q --filter 'name=projection-api' | head -1)"
|
|
acl="$(docker ps -q --filter 'name=[-_]acl[-_]' | head -1)"
|
|
[ -n "$oz" ] && [ -n "$nrc" ] || { echo "ERROR: OpenZaak and/or NRC not running — bring the stack up first" >&2; exit 1; }
|
|
[ -n "$es" ] && [ -n "$proj" ] || { echo "ERROR: event-subscriber and/or projection-api not running — bring the stack up first" >&2; exit 1; }
|
|
[ -n "$acl" ] || { echo "ERROR: acl not running — bring the stack up first" >&2; exit 1; }
|
|
net="$(docker inspect -f '{{range $k,$_ := .NetworkSettings.Networks}}{{$k}}{{"\n"}}{{end}}' "$oz" | head -1)"
|
|
oz_ip="$(ip "$oz")"; nrc_ip="$(ip "$nrc")"; es_ip="$(ip "$es")"; proj_ip="$(ip "$proj")"; acl_ip="$(ip "$acl")"
|
|
echo ">> network=$net openzaak=$oz_ip nrc=$nrc_ip event-subscriber=$es_ip projection-api=$proj_ip acl=$acl_ip"
|
|
|
|
echo ">> seeding a published BIG zaaktype (idempotent)"
|
|
sid="$(docker create --network "$net" -e "OZ_BASE=http://$oz_ip:8000" -e OZ_PUBLISH=1 \
|
|
python:3-slim python /seed.py)"
|
|
docker cp "$here/openzaak/seed_catalogus.py" "$sid:/seed.py" >/dev/null
|
|
docker start -a "$sid"
|
|
docker rm -f "$sid" >/dev/null
|
|
|
|
echo ">> registering the event-subscriber abonnement on the objecten kanaal"
|
|
docker rm -f rr-pverify >/dev/null 2>&1 || true
|
|
# The same script the local stack uses (ADR-0020), so both paths register the identical abonnement.
|
|
drv="$(docker create --network "$net" --name rr-pverify \
|
|
-e "NRC_BASE=http://$nrc_ip:8000" \
|
|
-e "SINK_HOST=$es_ip" -e "SINK_PORT=8080" -e "SINK_AUTH=$WEBHOOK_AUTH" \
|
|
python:3-slim python /subscribe.py)"
|
|
docker cp "$here/local/register-abonnement.py" "$drv:/subscribe.py" >/dev/null
|
|
docker start -a "$drv"
|
|
docker rm -f rr-pverify >/dev/null
|
|
|
|
# OpenZaak reflects the request Host into the zaaktype `url` it returns, and then rejects that same
|
|
# URL on zaak-create when the host is single-label ("Voer een geldige URL in."). The stack's ACL is
|
|
# configured with `http://openzaak:8000/`, so it must be repointed at OpenZaak's container IP before
|
|
# it can open a zaak — exactly what run-domain-check.sh does, and the same class of constraint as the
|
|
# `objecten.local` alias (ADR-0029). The ACL resolves the zaaktype itself (S-27, ADR-0021), so the
|
|
# base URL is the only thing to inject.
|
|
echo ">> recreating the acl service pointed at OpenZaak's IP"
|
|
ACL_OPENZAAK_BASEURL="http://$oz_ip:8000/" docker compose -f "$compose" up -d acl
|
|
WAIT_TIMEOUT="${WAIT_TIMEOUT:-120}" bash "$here/wait-healthy.sh" acl
|
|
# The container is replaced, so its IP may have changed.
|
|
acl="$(docker ps -q --filter 'name=[-_]acl[-_]' | head -1)"
|
|
acl_ip="$(ip "$acl")"
|
|
|
|
echo ">> opening a zaak through the ACL (which writes the INGEDIEND register record)"
|
|
reference="PROJ-$(date +%s)"
|
|
zaak_url="$(docker run --rm --network "$net" curlimages/curl:latest \
|
|
-fsS -X POST "http://$acl_ip:8080/zaken" -H 'Content-Type: application/json' \
|
|
-d "{\"bsn\":\"123456782\",\"reference\":\"$reference\"}" \
|
|
| sed -n 's/.*"zaakUrl":"\([^"]*\)".*/\1/p')"
|
|
[ -n "$zaak_url" ] || { echo "ERROR: the ACL did not open a zaak" >&2; exit 1; }
|
|
zaak_uuid="${zaak_url##*/}"
|
|
echo ">> zaak created: $zaak_url (reference $reference)"
|
|
|
|
echo ">> polling projection-api for the projected row (status INGEDIEND)"
|
|
for _ in $(seq 1 30); do
|
|
body="$(docker run --rm --network "$net" curlimages/curl:latest \
|
|
-fsS "http://$proj_ip:8080/register/$zaak_uuid" 2>/dev/null || true)"
|
|
if echo "$body" | grep -q '"INGEDIEND"'; then
|
|
echo "OK — projection-api serves zaak $zaak_uuid with status INGEDIEND"
|
|
echo "$body" | cut -c1-300
|
|
exit 0
|
|
fi
|
|
sleep 2
|
|
done
|
|
echo "FAIL — projection-api never served an INGEDIEND row for zaak $zaak_uuid" >&2
|
|
echo " The chain is ACL → Objecten → NRC → event-subscriber → projection (ADR-0030)." >&2
|
|
echo "--- event-subscriber log ---" >&2; docker logs "$es" 2>&1 | tail -10 >&2
|
|
echo "--- projection-api log ---" >&2; docker logs "$proj" 2>&1 | tail -10 >&2
|
|
echo "--- acl log ---" >&2; docker logs "$acl" 2>&1 | tail -10 >&2
|
|
exit 1
|