feat(openzaak): idempotent catalogus/zaaktype/zaak provisioning (WP-56)

bootstrap-catalogus.sh now looks up every resource by its natural key before creating it (catalogus by domein+rsin, zaaktype by catalogus+identificatie, statustype by zaaktype+volgnummer, roltype by zaaktype+omschrijvingGeneriek, zaaktype-publish by checking `concept` first, zaak by identificatie, status/rol by existence-under-the-zaak), so rerunning against an already-seeded instance reuses what's there instead of erroring. The WP's original plan (move this into OpenZaak's `setup_configuration` mechanism) turned out not to be achievable: reading the actual django_setup_configuration steps installed inside the open-zaak image shows no step exists for Catalogi/Zaken content anywhere in this OpenZaak version — only sites/credentials/applicaties/selectielijst. Documented as a deviation; the WP's own Risks section already anticipated this and sanctioned falling back to an idempotent script. Verified live: fresh instance -> full run (all created) -> integration test green -> reran the script twice more against the same instance (all reused, identical URLs, no duplicates) -> integration test still green. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-30 12:57:37 +02:00
parent 4d5ed7ca36
commit 89ad3490b0
4 changed files with 169 additions and 52 deletions
@@ -22,13 +22,24 @@ published), and one zaak (`BIG-2026-000123`) with an initiator rol for the seede
 (`111222333` — the same fixture BSN `OpenZaakZaakSourceTests.cs` uses). It writes what it
 seeded to `seeded.env` (gitignored) and prints a summary.

-**Not idempotent** — re-running against the same (already-seeded) instance fails on OpenZaak's
-`domein`+`rsin` uniqueness constraint for the catalogus. Reset with:
+**Idempotent (WP-56)** — every resource is looked up by its natural key (the same field(s)
+OpenZaak enforces identity on: catalogus by `domein`+`rsin`, zaaktype by `catalogus`+
+`identificatie`, statustype by `zaaktype`+`volgnummer`, roltype by `zaaktype`+
+`omschrijvingGeneriek`, zaak by `identificatie`) before creating it, so re-running against an
+already-seeded instance reuses what's there instead of erroring. Safe to run repeatedly
+against a long-lived instance, not just once per fresh volume — a full reset is still
+available if you want a truly clean slate:

 ```bash
 docker compose -f docker-compose.openzaak.yml down -v && docker compose -f docker-compose.openzaak.yml up -d
 ```

+This content has no `setup_configuration` (declarative-YAML) equivalent: reading the
+`django_setup_configuration` steps installed inside the `openzaak/open-zaak:1.29.1` image
+itself confirms the only app-registered steps are sites/credentials/applicaties (already used
+by `setup_configuration/data.yaml`) and Selectielijst API config — nothing for Catalogi/Zaken
+content. Hence this stays a script, made safe to rerun instead.
+
 ## Run the integration test against it

 ```bash
@@ -103,7 +114,7 @@ app change.
  live run of this exact script, not guessed (two OpenZaak quirks it works around: a zaaktype
  needs ≥1 resultaattype and 2 statustypen before it can be published, and its
  `selectielijstklasse` and the zaaktype's `selectielijstProcestype` must reference the same
-  `procesType` on the public VNG selectielijst API).
+  `procesType` on the public VNG selectielijst API). Idempotent (WP-56) — see "Bring it up" above.
 - **Not here**: Documenten (DRC) / Notificaties (NRC) content — add if a later WP needs to prove
  those round-trips against a live instance too (WP-51/52 are fixture-tested today).
 - `docker-compose.openzaak.prod.yml` (WP-55) — production overrides layered on top of
@@ -1,15 +1,21 @@
 #!/usr/bin/env bash
-# WP-54 — seeds business content (catalogus/zaaktype/statustype/roltype/zaak/status/rol) into
-# the OpenZaak harness started by docker-compose.openzaak.yml. `setup_configuration/data.yaml`
-# only covers infra config (the JWTSecret + Applicatie); Catalogi/Zaken content has no
-# declarative-YAML equivalent upstream, so this script does it the same way the BFF itself
-# does at runtime — plain REST calls with a hand-rolled HS256 JWT (see ZgwTokenProvider.cs,
-# mirrored here in bash+openssl so this script has no extra dependency beyond curl/openssl).
+# WP-54 (seeding) / WP-56 (idempotency) — seeds business content
+# (catalogus/zaaktype/statustype/roltype/zaak/status/rol) into the OpenZaak harness started
+# by docker-compose.openzaak.yml. `setup_configuration/data.yaml` only covers infra config
+# (JWTSecret + Applicatie) — confirmed by reading the installed `django_setup_configuration`
+# steps inside the `openzaak/open-zaak` image itself: the only app-registered step besides the
+# generic sites/credentials/applicaties ones is Selectielijst API config. There is NO
+# declarative-YAML equivalent upstream for Catalogi/Zaken content, so this script does it the
+# same way the BFF itself does at runtime — plain REST calls with a hand-rolled HS256 JWT (see
+# ZgwTokenProvider.cs, mirrored here in bash+openssl so this script has no extra dependency
+# beyond curl/openssl/python3, python3 already required by the JSON bodies below).
 #
-# Idempotent-ish: re-running creates duplicate catalogus/zaaktype rows (OpenZaak doesn't
-# dedupe by name) — meant to be run once per fresh `docker compose up`, not repeatedly against
-# a long-lived instance. Prints the seeded zaak's `identificatie` + `url` on success; also
-# writes them to seeded.env (repo-ignored) for OpenZaakIntegrationTests.cs to assert against.
+# Idempotent: every resource is looked up by its natural key (GET with the same filter OpenZaak
+# enforces uniqueness/identity on) before creating it, so re-running against an
+# already-seeded instance reuses what's there instead of erroring or duplicating. Safe to run
+# repeatedly against a long-lived instance, not just once per fresh volume. Prints the seeded
+# zaak's `identificatie` + `url` on success; also writes them to seeded.env (repo-ignored) for
+# OpenZaakIntegrationTests.cs to assert against.
 set -euo pipefail
 cd "$(dirname "${BASH_SOURCE[0]}")"

@@ -55,18 +61,47 @@ oz() {
  echo "$json"
 }

+# $1 = list path+query (server-side-filtered to the natural key). Prints the first result's
+# `url`, or nothing if the list is empty — the GET-before-POST idempotency check.
+existing_url() {
+  oz GET "$1" | python3 -c 'import json,sys; r=json.load(sys.stdin)["results"]; print(r[0]["url"] if r else "")'
+}
+
+# $1 = zaaktype URL, $2 = volgnummer. Statustype has no server-side volgnummer filter, so this
+# lists by zaaktype (server-filtered) and matches volgnummer client-side.
+existing_statustype_url() {
+  oz GET "/catalogi/api/v1/statustypen?zaaktype=$1" | python3 -c '
+import json, sys
+data = json.load(sys.stdin)
+vol = int(sys.argv[1])
+for r in data["results"]:
+    if r["volgnummer"] == vol:
+        print(r["url"])
+        break
+' "$2"
+}
+
 echo "Waiting for OpenZaak..."
 until curl -sS -o /dev/null -w '%{http_code}' "$BASE/catalogi/api/v1/catalogussen" | grep -q '^2\|^401\|^403'; do
  sleep 2
 done

-echo "Creating catalogus..."
-catalogus=$(oz POST /catalogi/api/v1/catalogussen "$(printf '{"domein":"BIGR","rsin":"%s","contactpersoonBeheerNaam":"BIG Register"}' "$RSIN")")
-catalogus_url=$(echo "$catalogus" | python3 -c 'import json,sys; print(json.load(sys.stdin)["url"])')
-echo "  $catalogus_url"
+echo "Catalogus..."
+catalogus_url=$(existing_url "/catalogi/api/v1/catalogussen?domein=BIGR&rsin=$RSIN")
+if [ -n "$catalogus_url" ]; then
+  echo "  exists: $catalogus_url"
+else
+  catalogus=$(oz POST /catalogi/api/v1/catalogussen "$(printf '{"domein":"BIGR","rsin":"%s","contactpersoonBeheerNaam":"BIG Register"}' "$RSIN")")
+  catalogus_url=$(echo "$catalogus" | python3 -c 'import json,sys; print(json.load(sys.stdin)["url"])')
+  echo "  created: $catalogus_url"
+fi

-echo "Creating zaaktype (concept)..."
-zaaktype=$(oz POST /catalogi/api/v1/zaaktypen "$(python3 -c '
+echo "Zaaktype (concept)..."
+zaaktype_url=$(existing_url "/catalogi/api/v1/zaaktypen?catalogus=$catalogus_url&identificatie=ZT-HERREG")
+if [ -n "$zaaktype_url" ]; then
+  echo "  exists: $zaaktype_url"
+else
+  zaaktype=$(oz POST /catalogi/api/v1/zaaktypen "$(python3 -c '
 import json, sys
 print(json.dumps({
  "identificatie": "ZT-HERREG",
@@ -95,32 +130,62 @@ print(json.dumps({
  "selectielijstProcestype": "https://selectielijst.openzaak.nl/api/v1/procestypen/e1b73b12-b2f6-4c4e-8929-94f84dd2a57d",
 }))
 ' "$catalogus_url")")
-zaaktype_url=$(echo "$zaaktype" | python3 -c 'import json,sys; print(json.load(sys.stdin)["url"])')
-echo "  $zaaktype_url"
+  zaaktype_url=$(echo "$zaaktype" | python3 -c 'import json,sys; print(json.load(sys.stdin)["url"])')
+  echo "  created: $zaaktype_url"
+fi

-echo "Creating statustypen (publish needs a begin AND an end status)..."
-statustype=$(oz POST /catalogi/api/v1/statustypen "$(printf '{"zaaktype":"%s","omschrijving":"Ontvangen","volgnummer":1}' "$zaaktype_url")")
-echo "  $(echo "$statustype" | python3 -c 'import json,sys; print(json.load(sys.stdin)["url"])')"
-statustype_eind=$(oz POST /catalogi/api/v1/statustypen "$(printf '{"zaaktype":"%s","omschrijving":"Afgehandeld","volgnummer":2}' "$zaaktype_url")")
-echo "  $(echo "$statustype_eind" | python3 -c 'import json,sys; print(json.load(sys.stdin)["url"])')"
+echo "Statustypen (publish needs a begin AND an end status)..."
+statustype_url=$(existing_statustype_url "$zaaktype_url" 1)
+if [ -n "$statustype_url" ]; then
+  echo "  exists (Ontvangen): $statustype_url"
+else
+  statustype_url=$(oz POST /catalogi/api/v1/statustypen "$(printf '{"zaaktype":"%s","omschrijving":"Ontvangen","volgnummer":1}' "$zaaktype_url")" | python3 -c 'import json,sys; print(json.load(sys.stdin)["url"])')
+  echo "  created (Ontvangen): $statustype_url"
+fi
+statustype_eind_url=$(existing_statustype_url "$zaaktype_url" 2)
+if [ -n "$statustype_eind_url" ]; then
+  echo "  exists (Afgehandeld): $statustype_eind_url"
+else
+  statustype_eind_url=$(oz POST /catalogi/api/v1/statustypen "$(printf '{"zaaktype":"%s","omschrijving":"Afgehandeld","volgnummer":2}' "$zaaktype_url")" | python3 -c 'import json,sys; print(json.load(sys.stdin)["url"])')
+  echo "  created (Afgehandeld): $statustype_eind_url"
+fi

-echo "Creating resultaattype (publish needs at least one)..."
-# The two URLs below are real reference-list entries on the public VNG selectielijst API
-# (selectielijst.openzaak.nl) — OpenZaak validates both by fetching them, same as it does
-# for a zaaktype URL, so a made-up URL 404s here.
-resultaattype=$(oz POST /catalogi/api/v1/resultaattypen "$(printf '{"zaaktype":"%s","omschrijving":"Afgehandeld","resultaattypeomschrijving":"https://selectielijst.openzaak.nl/api/v1/resultaattypeomschrijvingen/7cb315fb-4f7b-4a43-aca1-e4522e4c73b3","selectielijstklasse":"https://selectielijst.openzaak.nl/api/v1/resultaten/cc5ae4e3-a9e6-4386-bcee-46be4986a829","archiefnominatie":"blijvend_bewaren"}' "$zaaktype_url")")
-echo "  $(echo "$resultaattype" | python3 -c 'import json,sys; print(json.load(sys.stdin)["url"])')"
+echo "Resultaattype (publish needs at least one; existence-only requirement)..."
+if [ "$(oz GET "/catalogi/api/v1/resultaattypen?zaaktype=$zaaktype_url" | python3 -c 'import json,sys; print(len(json.load(sys.stdin)["results"]))')" != "0" ]; then
+  echo "  exists"
+else
+  # The two URLs below are real reference-list entries on the public VNG selectielijst API
+  # (selectielijst.openzaak.nl) — OpenZaak validates both by fetching them, same as it does
+  # for a zaaktype URL, so a made-up URL 404s here.
+  oz POST /catalogi/api/v1/resultaattypen "$(printf '{"zaaktype":"%s","omschrijving":"Afgehandeld","resultaattypeomschrijving":"https://selectielijst.openzaak.nl/api/v1/resultaattypeomschrijvingen/7cb315fb-4f7b-4a43-aca1-e4522e4c73b3","selectielijstklasse":"https://selectielijst.openzaak.nl/api/v1/resultaten/cc5ae4e3-a9e6-4386-bcee-46be4986a829","archiefnominatie":"blijvend_bewaren"}' "$zaaktype_url")" >/dev/null
+  echo "  created"
+fi

-echo "Creating roltype (initiator)..."
-roltype=$(oz POST /catalogi/api/v1/roltypen "$(printf '{"zaaktype":"%s","omschrijving":"Initiator","omschrijvingGeneriek":"initiator"}' "$zaaktype_url")")
-echo "  $(echo "$roltype" | python3 -c 'import json,sys; print(json.load(sys.stdin)["url"])')"
+echo "Roltype (initiator)..."
+roltype_url=$(existing_url "/catalogi/api/v1/roltypen?zaaktype=$zaaktype_url&omschrijvingGeneriek=initiator")
+if [ -n "$roltype_url" ]; then
+  echo "  exists: $roltype_url"
+else
+  roltype_url=$(oz POST /catalogi/api/v1/roltypen "$(printf '{"zaaktype":"%s","omschrijving":"Initiator","omschrijvingGeneriek":"initiator"}' "$zaaktype_url")" | python3 -c 'import json,sys; print(json.load(sys.stdin)["url"])')
+  echo "  created: $roltype_url"
+fi

 echo "Publishing zaaktype..."
 zaaktype_uuid=$(echo "$zaaktype_url" | sed 's#.*/##')
-oz POST "/catalogi/api/v1/zaaktypen/$zaaktype_uuid/publish" >/dev/null
+zaaktype_concept=$(oz GET "/catalogi/api/v1/zaaktypen/$zaaktype_uuid" | python3 -c 'import json,sys; print(json.load(sys.stdin)["concept"])')
+if [ "$zaaktype_concept" = "False" ]; then
+  echo "  already published"
+else
+  oz POST "/catalogi/api/v1/zaaktypen/$zaaktype_uuid/publish" >/dev/null
+  echo "  published"
+fi

-echo "Creating zaak..."
-zaak=$(oz POST /zaken/api/v1/zaken "$(python3 -c '
+echo "Zaak..."
+zaak_url=$(existing_url "/zaken/api/v1/zaken?identificatie=$ZAAK_REF")
+if [ -n "$zaak_url" ]; then
+  echo "  exists: $zaak_url"
+else
+  zaak=$(oz POST /zaken/api/v1/zaken "$(python3 -c '
 import json, sys
 print(json.dumps({
  "zaaktype": sys.argv[1],
@@ -130,19 +195,26 @@ print(json.dumps({
  "identificatie": sys.argv[3],
 }))
 ' "$zaaktype_url" "$RSIN" "$ZAAK_REF")")
-zaak_url=$(echo "$zaak" | python3 -c 'import json,sys; print(json.load(sys.stdin)["url"])')
-echo "  $zaak_url"
+  zaak_url=$(echo "$zaak" | python3 -c 'import json,sys; print(json.load(sys.stdin)["url"])')
+  echo "  created: $zaak_url"
+fi

-echo "Creating status..."
-statustype_url=$(echo "$statustype" | python3 -c 'import json,sys; print(json.load(sys.stdin)["url"])')
-oz POST /zaken/api/v1/statussen "$(python3 -c '
+echo "Status..."
+if [ "$(oz GET "/zaken/api/v1/statussen?zaak=$zaak_url" | python3 -c 'import json,sys; print(len(json.load(sys.stdin)["results"]))')" != "0" ]; then
+  echo "  exists"
+else
+  oz POST /zaken/api/v1/statussen "$(python3 -c '
 import json, sys
 print(json.dumps({"zaak": sys.argv[1], "statustype": sys.argv[2], "datumStatusGezet": "2026-07-28T12:00:00Z"}))
 ' "$zaak_url" "$statustype_url")" >/dev/null
+  echo "  created"
+fi

-echo "Creating rol (initiator, seeded BSN)..."
-roltype_url=$(echo "$roltype" | python3 -c 'import json,sys; print(json.load(sys.stdin)["url"])')
-oz POST /zaken/api/v1/rollen "$(python3 -c '
+echo "Rol (initiator, seeded BSN)..."
+if [ -n "$(existing_url "/zaken/api/v1/rollen?zaak=$zaak_url&omschrijvingGeneriek=initiator")" ]; then
+  echo "  exists"
+else
+  oz POST /zaken/api/v1/rollen "$(python3 -c '
 import json, sys
 print(json.dumps({
  "zaak": sys.argv[1],
@@ -152,6 +224,8 @@ print(json.dumps({
  "betrokkeneIdentificatie": {"inpBsn": sys.argv[3]},
 }))
 ' "$zaak_url" "$roltype_url" "$BSN")" >/dev/null
+  echo "  created"
+fi

 cat > seeded.env <<EOF
 ZAAK_REFERENTIE=$ZAAK_REF
@@ -106,7 +106,7 @@ for its existing violations, so every WP ends green.
 | [WP-53](WP-53-inbound-identity-and-citizen-scoping.md) | Inbound identity seam + citizen-scoping (per-request BSN, ZGW audit claims)        | 9 · OpenZaak/ZGW            | done   |
 | [WP-54](WP-54-openzaak-integration-harness.md)         | Docker OpenZaak integration-test harness (opt-in, live round-trip)                 | 9 · OpenZaak/ZGW            | done   |
 | [WP-55](WP-55-openzaak-secrets-tls.md)                 | Real secrets + TLS for the OpenZaak harness                                        | 10 · OpenZaak hardening     | done   |
-| [WP-56](WP-56-openzaak-catalogus-provisioning.md)      | Idempotent catalogus provisioning                                                  | 10 · OpenZaak hardening     | todo   |
+| [WP-56](WP-56-openzaak-catalogus-provisioning.md)      | Idempotent catalogus provisioning                                                  | 10 · OpenZaak hardening     | done   |
 | [WP-57](WP-57-openzaak-least-privilege-scopes.md)      | Least-privilege client scopes                                                      | 10 · OpenZaak hardening     | todo   |
 | [WP-58](WP-58-openzaak-notifications.md)               | Real notifications (celery + scripted abonnement)                                  | 10 · OpenZaak hardening     | todo   |
 | [WP-59](WP-59-document-confidentialiteit-config.md)    | Per-document-type confidentialiteit config                                         | 10 · OpenZaak hardening     | todo   |
@@ -1,6 +1,6 @@
 # WP-56 — Idempotent catalogus provisioning

-Status: todo
+Status: done
 Phase: 10 — OpenZaak production hardening

 ## Why
@@ -49,12 +49,44 @@ content too.

 ## Acceptance criteria

- [ ] Catalogus/zaaktype/statustype/roltype provisioning is declarative
-      (`setup_configuration`), not imperative curl.
- [ ] Running the compose stack up twice in a row doesn't error.
- [ ] WP-54's `OpenZaakIntegrationTests` still pass unchanged (same content, different
+- [x] ~~Catalogus/zaaktype/statustype/roltype provisioning is declarative
+      (`setup_configuration`), not imperative curl.~~ Not achievable — see Deviation below;
+      replaced by: provisioning is safe to rerun against an already-seeded instance.
+- [x] Running the compose stack up twice in a row doesn't error (verified: `bootstrap-catalogus.sh`
+      run 3× in a row against the same instance, all reused/no duplicates/no errors).
+- [x] WP-54's `OpenZaakIntegrationTests` still pass unchanged (same content, different
      provisioning mechanism).

+## Deviation from the original plan
+
+Confirmed by reading the `django_setup_configuration` steps actually installed inside the
+`openzaak/open-zaak:1.29.1` image (`/app/src/openzaak/config/setup_configuration/steps/` +
+every third-party `contrib/setup_configuration/` package): the ONLY app-registered
+configuration steps are sites/credentials/applicaties (already used by `data.yaml`) and
+Selectielijst API config. There is no step for catalogus/zaaktype/statustype/roltype/zaak
+content anywhere in this OpenZaak version — the WP's core premise ("move provisioning into
+setup_configuration") is not achievable. This was explicitly anticipated by the WP's own Risks
+section ("if a piece genuinely can't be expressed declaratively, keep it in a clearly-labeled
+idempotent script rather than forcing a bad fit") and Decisions block ("either genuinely
+idempotent, or the compose is structured to only run it once... document which").
+
+Chose **genuinely idempotent**: `bootstrap-catalogus.sh` now looks up every resource by its
+natural key before creating it (catalogus by `domein`+`rsin`, zaaktype by `catalogus`+
+`identificatie`, statustype by `zaaktype`+`volgnummer` (no server-side volgnummer filter, so
+listed by zaaktype and matched client-side), resultaattype by existence-only (publish just
+needs ≥1), roltype by `zaaktype`+`omschrijvingGeneriek`, zaaktype-publish by checking `concept`
+on the zaaktype detail first, zaak by `identificatie`, status/rol by existence-under-the-zaak).
+Real gotcha hit and fixed: OpenZaak's query-parameter names are camelCase
+(`omschrijvingGeneriek`), not the Django model's snake_case (`omschrijving_generiek`) the
+filter is registered under internally — the snake_case form 400s with "Onbekende query
+parameters" (unknown query parameter). Verified for real: fresh instance → full run (all
+"created") → `dotnet test --filter Category=Integration` green → reran the script twice more
+against the same live instance (all "exists", identical URLs each time, no duplicates) →
+integration test still green.
+
+Not attempted: moving anything into `setup_configuration/data.yaml` (nothing there to move,
+per the finding above).
+
 ## Verification

 `docker compose -f backend/openzaak/docker-compose.openzaak.yml up` twice in a row (fresh