feat(openzaak): bounded retry + flagged write divergence (WP-60)

Local aanvraag/document writes and their paired ZGW writes aren't
transactional; a ZGW failure after the local write succeeds used to
diverge silently. ZgwHttpClient now retries transport-shaped failures
(not 500, which can follow a partial commit on the non-idempotent
statussen/rollen POSTs), and a ZGW failure that survives retry sets
Aanvraag.ZgwError plus a zgw:divergence audit row instead of failing
or diverging quietly. No outbox/reconcile job: three request-triggered
write paths don't justify a persisted queue that would also need to
carry citizen PII for the JWT audit claims.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

docs/project/backlog/README.md

@@ -110,7 +110,7 @@ for its existing violations, so every WP ends green.
 | [WP-57](WP-57-openzaak-least-privilege-scopes.md)      | Least-privilege client scopes                                                      | 10 · OpenZaak hardening     | done   |
 | [WP-58](WP-58-openzaak-notifications.md)               | Real notifications (celery + scripted abonnement)                                  | 10 · OpenZaak hardening     | done   |
 | [WP-59](WP-59-document-confidentialiteit-config.md)    | Per-document-type confidentialiteit config                                         | 10 · OpenZaak hardening     | done   |
-| [WP-60](WP-60-write-divergence-resilience.md)          | Write-divergence resilience (local + ZGW writes)                                   | 10 · OpenZaak hardening     | todo   |
+| [WP-60](WP-60-write-divergence-resilience.md)          | Write-divergence resilience (local + ZGW writes)                                   | 10 · OpenZaak hardening     | done   |
 | [WP-61](WP-61-behandelportal-bootstrap.md)             | Bootstrap the behandelportal app                                                   | 11 · Behandelportal         | todo   |
 | [WP-62](WP-62-medewerker-identity-authz.md)            | Backend: medewerker caller identity + authz seam                                   | 11 · Behandelportal         | todo   |
 | [WP-63](WP-63-aanvraag-status-lifecycle.md)            | Backend: aanvraag status lifecycle as a published DTO                              | 11 · Behandelportal         | todo   |
@@ -149,17 +149,16 @@ deployment of 49–52) and **54** (a docker OpenZaak harness + opt-in integratio
 CRUD arc and can land any time; 54 depends on 49 (something to read) and unlocks realistic
 testing for the rest. Both are self-contained (each WP file carries its own current-state
 handoff) and sized for a fresh Sonnet session.
-Phase 10 (OpenZaak production hardening, WP-55..60) and Phase 11 (Behandelportal,
-WP-61..66) are two independent tracks that can be worked concurrently — neither blocks
-the other. Within phase 10: 55/59/60 are fully independent; 57 and 58 both build on 56's
-provisioning mechanism, otherwise independent of each other. Within phase 11: 61
+Phase 10 (OpenZaak production hardening, WP-55..60 — now **done**) and Phase 11
+(Behandelportal, WP-61..66) are two independent tracks that can be worked concurrently —
+neither blocks the other. Within phase 10: 55/59/60 were fully independent; 57 and 58 both
+built on 56's provisioning mechanism, otherwise independent of each other. Within phase 11: 61
 (bootstrap), 62 (backend medewerker identity), and 63 (backend status DTO) are
 independent of each other and can land in any order; 64 needs all three (61 for the app
-to exist, 62 for identity, 63 for the status it reads); 65 needs 64; 66 needs 65 and
-benefits from — but doesn't strictly require — phase 10's WP-60 landing first (WP-66 is
-a second, currently-unprotected write pair otherwise). WP-60 is the one slice in phase 10
-sized for a `planner`-agent kickoff rather than direct implementation — its Decisions
-block is deliberately left open (outbox vs. retry+reconcile).
+to exist, 62 for identity, 63 for the status it reads); 65 needs 64; 66 needs 65 and — now
+that WP-60 has landed (bounded retry + flagged divergence in `ZgwHttpClient`/`Program.cs`) —
+inherits that retry for free, but must call `RecordZgwDivergence` on its own besluit write path
+to get the flagging half too.
 
 ## WP template
 

docs/project/backlog/WP-59-document-confidentialiteit-config.md

@@ -68,10 +68,10 @@ not "must every category be configured."
 
 ## Verification
 
-`cd backend && dotnet test` (161/161 green, incl. the 2 new `OpenZaakDocumentSourceTests`
-+ the new `StamdataValidationTests` reference entry); `dotnet format --verify-no-changes`
-clean. Manual: `/beheer/stamdata` shows and edits the new table; an upload for a mapped
-document type carries the mapped confidentiality level (test asserted).
+`cd backend && dotnet test` (161/161 green, incl. the 2 new `OpenZaakDocumentSourceTests` plus
+the new `StamdataValidationTests` reference entry); `dotnet format --verify-no-changes` clean.
+Manual: `/beheer/stamdata` shows and edits the new table; an upload for a mapped document type
+carries the mapped confidentiality level (test asserted).
 
 ## Out of scope
 

docs/project/backlog/WP-60-write-divergence-resilience.md

@@ -1,6 +1,6 @@
 # WP-60 — Write-divergence resilience (local + ZGW writes)
 
-Status: todo
+Status: done
 Phase: 10 — OpenZaak production hardening
 
 ## Why
@@ -23,37 +23,57 @@ production.
 
 ## Decisions
 
-Intentionally left open for kickoff — this is exactly the kind of ambiguous-root-cause,
-multi-file design call the `planner` agent should make, not something pre-decided here.
-Options to weigh at kickoff:
+Picked **(b), narrowed further: bounded synchronous retry + flag, no reconcile job.** The
+`planner` agent's kickoff review found the write side smaller than either option assumed:
 
-- (a) an outbox table — write local + an outbox row in one local transaction, a background
-  worker drains the outbox to ZGW with retry.
-- (b) a simpler synchronous retry-with-backoff at the call site, plus a reconciliation job
-  that periodically diffs local vs. ZGW state and flags/repairs divergence.
+- The only ZGW writes are `OpenZaakZaakSource.CreateZaak` (zaak/status/rol, one POST sequence
+  per submit) and `OpenZaakDocumentSource.Upload`/`LinkToZaak` (DRC + zaakinformatieobject).
+  There is no standalone status-transition write path yet (that's WP-66) — Step 2 below is
+  corrected accordingly.
+- Every path already does the local write first and never rolls it back on a ZGW failure — "the
+  ZGW half fails, local succeeded" is the only real scenario; the reverse can't happen.
+- An outbox was rejected: three request-triggered write paths don't justify a persisted queue,
+  and a ZGW call's `CallerIdentity` (needed for the JWT's audit claims, WP-53) would mean PII
+  sitting in a new table — the "generic outbox framework" this WP's own Risks section warns
+  against.
+- A reconcile job was judged unnecessary for the acceptance criteria: flagging (not silent
+  divergence) is sufficient, and repair is always possible on demand because a zaak's
+  `identificatie` equals the aanvraag's `Referentie` — no reconcile job ships in this WP.
 
-Pick the smaller one that closes the gap — don't build a generic outbox framework if a
-bounded retry+reconcile suffices for this POC's actual write volume.
+Shipped: bounded retry (3 attempts, doubling backoff from 200ms) in `ZgwHttpClient` for
+transport-shaped failures only (429/502/503/504/408 + connection errors/timeouts — deliberately
+**not** 500, which can follow a partial commit on the non-idempotent `/statussen`/`/rollen`
+POSTs); `Aanvraag.ZgwError` + a `zgw:divergence` audit row when a ZGW write still fails after
+retry (`Program.cs`'s submit endpoint, two separate try/catches so a create-zaak failure doesn't
+also skip the still-local document link); `OpenZaakDocumentSource.Upload` catches and logs
+without a separate flag column (`DrcUrl == null` already means "not registered in ZGW yet").
+Full reasoning + rejected sub-options: [openzaak-integration.md](../reference/openzaak-integration.md)'s
+"Write resilience" section.
 
 ## Files
 
-Likely `Data/ApplicationStore.cs`, a new reconciliation/outbox mechanism,
-`Zgw/OpenZaakZaakSource.cs`, `Program.cs` (background job registration if needed).
+`Zgw/ZgwHttpClient.cs` (retry), `Data/ApplicationStore.cs` (`ZgwError` column + migration),
+`Program.cs` (submit endpoint rewire + `RecordZgwDivergence` + HttpClient timeouts),
+`Zgw/OpenZaakDocumentSource.cs` (non-throwing upload). No new file for a mechanism — no
+outbox/background worker shipped (see Decisions).
 
 ## Steps
 
-1. Design review with the `planner` agent — pick outbox vs. retry+reconcile.
-2. Implement the chosen mechanism for the create-zaak and status-transition write paths.
+1. Design review with the `planner` agent — pick outbox vs. retry+reconcile. Done: retry+flag
+   (see Decisions).
+2. Implement the chosen mechanism for the create-zaak and document (upload + link) write
+   paths — not "status-transition" as originally scoped here; that path doesn't exist yet
+   (arrives with WP-66).
 3. Add a test that simulates a ZGW failure mid-write and asserts the system recovers
    (retries successfully, or is left in a detectably-inconsistent-but-flagged state)
    rather than silently diverging.
 
 ## Acceptance criteria
 
-- [ ] A simulated ZGW failure after a successful local write no longer leaves permanent
+- [x] A simulated ZGW failure after a successful local write no longer leaves permanent
       silent divergence — either it retries to consistency or the divergence is
       detectable/flagged.
-- [ ] No new synchronous latency added to the happy path beyond what the chosen mechanism
+- [x] No new synchronous latency added to the happy path beyond what the chosen mechanism
       requires.
 
 ## Verification

docs/project/backlog/WP-66-behandelportal-openzaak-write.md

@@ -62,8 +62,11 @@ Any further behandelportal screens beyond beoordeling.
 
 ## Risks
 
-If Phase 10's WP-60 (write-divergence resilience) hasn't landed yet, this introduces a
-second unprotected write pair — call this out explicitly if the two phases aren't
-sequenced together in practice.
+WP-60 (write-divergence resilience) has landed: bounded retry lives in `ZgwHttpClient`, so
+this write pair inherits it automatically. It does **not** get the flagging half for free —
+call `RecordZgwDivergence` (or the equivalent for whichever endpoint hosts the besluit write) on
+this path's catch too, the same way `Program.cs`'s submit endpoint does for create-zaak/document
+writes, or this becomes the "second, currently-unprotected write pair" WP-60's own scope note
+anticipated.
 
-Depends on: WP-65. Benefits from (but doesn't strictly require) WP-60.
+Depends on: WP-65.

docs/reference/architecture/0005-openzaak-behind-bff.md

@@ -67,5 +67,13 @@ up front — the migration stance ADR-0001 already prescribes.
 - **Also shipped (WP-50):** `IZaakSource.CreateZaak` — the first write. Submitting an aanvraag
   now also creates a Zaak + Status + Rol in OpenZaak when `Zgw:Enabled=true`, routed through the
   existing submit endpoint with zero DTO change (same seam, same anti-corruption boundary).
-- **Deferred:** real inbound OIDC/JWT auth (still header-stubbed), Documenten/DRC upload + link
-  (WP-51), Notificaties/NRC webhooks (WP-52), adding OpenZaak to docker-compose.
+- **Also shipped (WP-51):** `IDocumentSource` (`LocalDocumentSource`/`OpenZaakDocumentSource`,
+  same config-gated seam shape) — an upload registers a Documenten/DRC enkelvoudiginformatieobject
+  and, once a zaak exists, a submit links it in with a zaakinformatieobject.
+- **Also shipped (WP-60):** bounded retry in `ZgwHttpClient` for transport-shaped ZGW failures,
+  plus a flagged (not silent) divergence — `Aanvraag.ZgwError` + a `zgw:divergence` audit row —
+  when a ZGW write still fails after retry. No outbox/background worker (see WP-60 for the
+  ladder check that ruled it out for this POC's write volume).
+- **Deferred:** real inbound OIDC/JWT auth (still header-stubbed), Notificaties/NRC webhooks
+  (WP-52, shipped instead as a direct-to-BFF delivery in WP-58), adding OpenZaak to
+  docker-compose, an automated reconciliation/repair job for a flagged divergence (WP-60).

docs/reference/openzaak-integration.md

@@ -61,11 +61,51 @@ The created zaak's `identificatie` becomes the returned `Referentie`; its status
 same coarse `InBehandeling` shape `ZgwZaakMapper` already uses for a freshly-opened zaak
 (`ZgwZaakMapper.ToCreatedStatusDto`).
 
-ponytail shortcuts, marked at the call sites: (a) "first statustype/roltype Catalogi returns"
-rather than a fully-configured per-type map — fine while a zaaktype has exactly one initial
-status and initiator role; (b) no compensating transaction — if any ZGW call throws, the
-aanvraag is already `Submitted` locally with no matching zaak (acceptable for a demo backend;
-a production arc needs retry/reconciliation or an outbox before trusting this dual-write).
+ponytail shortcut still standing: "first statustype/roltype Catalogi returns" rather than a
+fully-configured per-type map — fine while a zaaktype has exactly one initial status and
+initiator role. The "no compensating transaction" gap this section used to flag here is closed
+by WP-60 — see "Write resilience" below.
+
+## Write resilience (WP-60)
+
+The local write (`ApplicationStore.Submit`, `DocumentStore.Add`/`Link`) and its paired ZGW
+write aren't transactional — this section covers what happens when the ZGW half fails after the
+local half already committed, closing the one gap the sections above used to flag as needing
+"retry/reconciliation or an outbox" before this integration could be called production-ready.
+Deliberately **not** an outbox: three write paths, each triggered by exactly one interactive
+request, don't justify a persisted queue (which would also need to carry the acting citizen's
+BSN for the JWT's audit claims — PII in a new table) — see WP-60 for the full reasoning.
+
+- **Bounded retry, in `ZgwHttpClient`.** Every ZGW call gets up to 3 attempts (200ms, doubling)
+  on transport-shaped failures — 429/502/503/504/408, connection errors, timeouts — with a
+  fresh request and JWT per attempt (a sent request/content can't be resent). **500 is
+  deliberately not retried**: it can follow a partial commit on the two non-idempotent POSTs
+  (`/statussen`, `/rollen`), so retrying risks a duplicate write. The create-zaak/document POSTs
+  are additionally safe to retry because OpenZaak enforces uniqueness on
+  (`bronorganisatie`, `identificatie`) — and WP-50/51 already set `identificatie` to the
+  locally-generated reference/document id, so a retry after a lost response 400s instead of
+  duplicating.
+- **The local write is never rolled back.** Un-submitting a local aanvraag after a partial ZGW
+  failure (e.g. the zaak POST succeeded but `/statussen` didn't) would let the citizen resubmit
+  under a _new_ reference, orphaning the first zaak — worse than leaving it flagged.
+- **A caught ZGW failure is flagged, not silent.** `Program.cs`'s submit endpoint wraps
+  `CreateZaak` and `LinkToZaak` in separate try/catches (separate so a create-zaak failure
+  doesn't also skip the still-local document link) and, on catch, logs the error, sets
+  `Aanvraag.ZgwError` (non-null = "the ZGW side of this submit didn't complete"), and records a
+  `zgw:divergence` audit row (same `AuthzAuditStore` trail every other decision uses, visible at
+  `/beheer/audit`) — see `RecordZgwDivergence`. The endpoint still returns 200 with the local
+  reference/status: that's truthful (the reference _is_ what would become the zaak's
+  `identificatie`) and never branches on `Zgw:Enabled` (an offline `LocalZaakSource` never
+  throws, so the catch is dead code there).
+- **The document upload path flags differently.** `OpenZaakDocumentSource.Upload` catches its
+  own ZGW failure (config gap or transport) and logs it, but doesn't set a separate flag column
+  — `DocumentStore.Get(id).DrcUrl == null` is already the meaningful "not registered in ZGW yet"
+  detector `LinkToZaak` skips on, so no second mechanism is needed for that half.
+- **Repair.** No automated reconcile job exists yet — a flagged zaak is repairable on demand
+  because its (would-be) `identificatie` always equals the aanvraag's `Referentie`, so a future
+  admin action can `GET /zaken?identificatie=...` and either adopt the existing zaak or retry
+  `CreateZaak`. Deferred until a second write pair (WP-66) or a real deployment makes it worth
+  building — at which point the outbox question above is also worth re-asking.
 
 ## Documenten / DRC upload + zaak link (WP-51)
 
@@ -88,8 +128,11 @@ happens first — it stays the record of truth for preview/download/audit regard
 `ZgwHttpClient` (shared GET/POST-with-bearer-JWT plumbing) was factored out of
 `OpenZaakZaakSource` once `OpenZaakDocumentSource` needed the identical boilerplate.
 
-ponytail shortcut: `vertrouwelijkheidaanduiding` is hardcoded to `"openbaar"` — a per-category
-confidentiality level would matter for production but isn't needed to prove the seam.
+`vertrouwelijkheidaanduiding` is driven by a per-document-type stamdata table (WP-59,
+`Stamdata/documentconfidentialiteit.json`, ADR-0004), falling back to `"openbaar"` for any
+category absent from it. Unlike the zaak side, an upload's ZGW failure (past
+`DocumentStore.Add`) is caught and logged rather than persisted as a separate flag column —
+see "Write resilience" below for why the two write paths differ.
 
 ## The ZGW client (`backend/src/BigRegister.Api/Zgw/`)