atomic-design-poc/docs/project/archive/backlog/WP-60-write-divergence-resilience.md

WP-60 — Write-divergence resilience (local + ZGW writes)

Status: done Phase: 10 — OpenZaak production hardening

Why

A citizen action today does a local Aanvraag/Document write and a paired ZGW write (create zaak/status/document); these aren't transactional. If the ZGW call fails after the local write succeeds (or vice versa), the two diverge silently — openzaak-integration.md flags this explicitly as "acceptable for a demo backend; a production arc needs retry/reconciliation or an outbox." This is the one genuine correctness gap standing between the current integration and something safe to call production.

Read first

• openzaak-integration.md (the section discussing this gap)
• backend/src/BigRegister.Api/Data/ApplicationStore.cs, Zgw/OpenZaakZaakSource.cs (the two write sides)
• ADR-0005 — OpenZaak behind the BFF

Decisions

Picked (b), narrowed further: bounded synchronous retry + flag, no reconcile job. The planner agent's kickoff review found the write side smaller than either option assumed:

• The only ZGW writes are OpenZaakZaakSource.CreateZaak (zaak/status/rol, one POST sequence per submit) and OpenZaakDocumentSource.Upload/LinkToZaak (DRC + zaakinformatieobject). There is no standalone status-transition write path yet (that's WP-66) — Step 2 below is corrected accordingly.
• Every path already does the local write first and never rolls it back on a ZGW failure — "the ZGW half fails, local succeeded" is the only real scenario; the reverse can't happen.
• An outbox was rejected: three request-triggered write paths don't justify a persisted queue, and a ZGW call's CallerIdentity (needed for the JWT's audit claims, WP-53) would mean PII sitting in a new table — the "generic outbox framework" this WP's own Risks section warns against.
• A reconcile job was judged unnecessary for the acceptance criteria: flagging (not silent divergence) is sufficient, and repair is always possible on demand because a zaak's identificatie equals the aanvraag's Referentie — no reconcile job ships in this WP.

Shipped: bounded retry (3 attempts, doubling backoff from 200ms) in ZgwHttpClient for transport-shaped failures only (429/502/503/504/408 + connection errors/timeouts — deliberately not 500, which can follow a partial commit on the non-idempotent /statussen//rollen POSTs); Aanvraag.ZgwError + a zgw:divergence audit row when a ZGW write still fails after retry (Program.cs's submit endpoint, two separate try/catches so a create-zaak failure doesn't also skip the still-local document link); OpenZaakDocumentSource.Upload catches and logs without a separate flag column (DrcUrl == null already means "not registered in ZGW yet"). Full reasoning + rejected sub-options: openzaak-integration.md's "Write resilience" section.

Files

Zgw/ZgwHttpClient.cs (retry), Data/ApplicationStore.cs (ZgwError column + migration), Program.cs (submit endpoint rewire + RecordZgwDivergence + HttpClient timeouts), Zgw/OpenZaakDocumentSource.cs (non-throwing upload). No new file for a mechanism — no outbox/background worker shipped (see Decisions).

Steps

1. Design review with the planner agent — pick outbox vs. retry+reconcile. Done: retry+flag (see Decisions).
2. Implement the chosen mechanism for the create-zaak and document (upload + link) write paths — not "status-transition" as originally scoped here; that path doesn't exist yet (arrives with WP-66).
3. Add a test that simulates a ZGW failure mid-write and asserts the system recovers (retries successfully, or is left in a detectably-inconsistent-but-flagged state) rather than silently diverging.

Acceptance criteria

• A simulated ZGW failure after a successful local write no longer leaves permanent silent divergence — either it retries to consistency or the divergence is detectable/flagged.
• No new synchronous latency added to the happy path beyond what the chosen mechanism requires.

Verification

A new integration test that fails a stubbed ZGW call mid-write and asserts recovery/flagging behavior; cd backend && dotnet test.

Out of scope

A general-purpose outbox framework reusable beyond this one write pair (YAGNI unless a second write pair appears — note WP-66 is exactly that second pair, so revisit scope if WP-66 lands first); UI surfacing of reconciliation state (backend-only fix for now).

Risks

Over-building this (a generic outbox/saga framework) for a POC's actual write volume — ladder check at kickoff: does a bounded retry + periodic reconcile job cover it before reaching for an outbox table?