feat(openzaak): bounded retry + flagged write divergence (WP-60)

Local aanvraag/document writes and their paired ZGW writes aren't
transactional; a ZGW failure after the local write succeeds used to
diverge silently. ZgwHttpClient now retries transport-shaped failures
(not 500, which can follow a partial commit on the non-idempotent
statussen/rollen POSTs), and a ZGW failure that survives retry sets
Aanvraag.ZgwError plus a zgw:divergence audit row instead of failing
or diverging quietly. No outbox/reconcile job: three request-triggered
write paths don't justify a persisted queue that would also need to
carry citizen PII for the JWT audit claims.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

docs/project/backlog/WP-60-write-divergence-resilience.md

@@ -1,6 +1,6 @@
 # WP-60 — Write-divergence resilience (local + ZGW writes)
 
-Status: todo
+Status: done
 Phase: 10 — OpenZaak production hardening
 
 ## Why
@@ -23,37 +23,57 @@ production.
 
 ## Decisions
 
-Intentionally left open for kickoff — this is exactly the kind of ambiguous-root-cause,
-multi-file design call the `planner` agent should make, not something pre-decided here.
-Options to weigh at kickoff:
+Picked **(b), narrowed further: bounded synchronous retry + flag, no reconcile job.** The
+`planner` agent's kickoff review found the write side smaller than either option assumed:
 
-- (a) an outbox table — write local + an outbox row in one local transaction, a background
-  worker drains the outbox to ZGW with retry.
-- (b) a simpler synchronous retry-with-backoff at the call site, plus a reconciliation job
-  that periodically diffs local vs. ZGW state and flags/repairs divergence.
+- The only ZGW writes are `OpenZaakZaakSource.CreateZaak` (zaak/status/rol, one POST sequence
+  per submit) and `OpenZaakDocumentSource.Upload`/`LinkToZaak` (DRC + zaakinformatieobject).
+  There is no standalone status-transition write path yet (that's WP-66) — Step 2 below is
+  corrected accordingly.
+- Every path already does the local write first and never rolls it back on a ZGW failure — "the
+  ZGW half fails, local succeeded" is the only real scenario; the reverse can't happen.
+- An outbox was rejected: three request-triggered write paths don't justify a persisted queue,
+  and a ZGW call's `CallerIdentity` (needed for the JWT's audit claims, WP-53) would mean PII
+  sitting in a new table — the "generic outbox framework" this WP's own Risks section warns
+  against.
+- A reconcile job was judged unnecessary for the acceptance criteria: flagging (not silent
+  divergence) is sufficient, and repair is always possible on demand because a zaak's
+  `identificatie` equals the aanvraag's `Referentie` — no reconcile job ships in this WP.
 
-Pick the smaller one that closes the gap — don't build a generic outbox framework if a
-bounded retry+reconcile suffices for this POC's actual write volume.
+Shipped: bounded retry (3 attempts, doubling backoff from 200ms) in `ZgwHttpClient` for
+transport-shaped failures only (429/502/503/504/408 + connection errors/timeouts — deliberately
+**not** 500, which can follow a partial commit on the non-idempotent `/statussen`/`/rollen`
+POSTs); `Aanvraag.ZgwError` + a `zgw:divergence` audit row when a ZGW write still fails after
+retry (`Program.cs`'s submit endpoint, two separate try/catches so a create-zaak failure doesn't
+also skip the still-local document link); `OpenZaakDocumentSource.Upload` catches and logs
+without a separate flag column (`DrcUrl == null` already means "not registered in ZGW yet").
+Full reasoning + rejected sub-options: [openzaak-integration.md](../reference/openzaak-integration.md)'s
+"Write resilience" section.
 
 ## Files
 
-Likely `Data/ApplicationStore.cs`, a new reconciliation/outbox mechanism,
-`Zgw/OpenZaakZaakSource.cs`, `Program.cs` (background job registration if needed).
+`Zgw/ZgwHttpClient.cs` (retry), `Data/ApplicationStore.cs` (`ZgwError` column + migration),
+`Program.cs` (submit endpoint rewire + `RecordZgwDivergence` + HttpClient timeouts),
+`Zgw/OpenZaakDocumentSource.cs` (non-throwing upload). No new file for a mechanism — no
+outbox/background worker shipped (see Decisions).
 
 ## Steps
 
-1. Design review with the `planner` agent — pick outbox vs. retry+reconcile.
-2. Implement the chosen mechanism for the create-zaak and status-transition write paths.
+1. Design review with the `planner` agent — pick outbox vs. retry+reconcile. Done: retry+flag
+   (see Decisions).
+2. Implement the chosen mechanism for the create-zaak and document (upload + link) write
+   paths — not "status-transition" as originally scoped here; that path doesn't exist yet
+   (arrives with WP-66).
 3. Add a test that simulates a ZGW failure mid-write and asserts the system recovers
    (retries successfully, or is left in a detectably-inconsistent-but-flagged state)
    rather than silently diverging.
 
 ## Acceptance criteria
 
-- [ ] A simulated ZGW failure after a successful local write no longer leaves permanent
+- [x] A simulated ZGW failure after a successful local write no longer leaves permanent
       silent divergence — either it retries to consistency or the divergence is
       detectable/flagged.
-- [ ] No new synchronous latency added to the happy path beyond what the chosen mechanism
+- [x] No new synchronous latency added to the happy path beyond what the chosen mechanism
       requires.
 
 ## Verification