## What & why S-19b-2, closing out ADR-0028's stated direction: **the read projection is now derived from the `RegisterRecord` in Objecten, not from ZGW zaak events.** Until now the subscriber listened on `zaken` and *inferred* register state from case events — a `zaak/create` meant INGEDIEND, and any `status/create` was assumed to be the approval (it may not read OpenZaak, so it could not tell statustypen apart). The reference wasn't in the notification at all, so every projection made a second hop to the ACL. The register — a fact about a person — was being reconstructed by guessing at the lifecycle of the case that produced it. - The subscriber's abonnement moves to the `objecten` kanaal (S-19b-1 made it publish). - An Objecten notification carries **no record data**, only the object URL, so the record is read back through the ACL (`POST /register-records/read`) — §8.1 applies to Objecten exactly as ADR-0028 established. - The record carries `id`, `status` and `reference`, so the row *is* the record: `IsZaakCreated`, `IsZaakStatusSet`, `ZaakUrl`, `ZaakId` and `ToEntry`'s `Resource == "status"` inference are all gone, and so is the ACL enrichment hop. - **The ACL now writes an INGEDIEND record on submit.** Without it, re-sourcing would silently drop every submitted registration from the public register, since only approval wrote a record. - `processed_notifications` holds the projected row (`register_id`, `status`, `reference`) instead of the ZGW event, so a rebuild is a replay with no mapping rules and no upstream reads at all. **ADR-0030** records it. ADR-0028's open caveat — record written but not yet read, "the two must agree" — is closed: there is one source now. Closes #153 ## Definition of Done - [x] Linked Gitea issue (above). - [x] Failing tests committed before the implementation — two red/green pairs, ACL side (06c0444→566ef7d) and subscriber side (142ed45→8af09b2). - [x] Refactor commit follows (b496ac9). - [x] Conventional Commits referencing the issue (`refs #153`). - [x] CI green — all six jobs onb30fa66, `verify-stack` end to end including the e2e. - [x] `docker compose up` from a fresh clone reaches green health checks within 3 minutes (`verify-stack`'s bring-up step — see the wait-healthy fix below). - [x] Docs updated — ADR-0030 added, ADR-0028's consequence + caveat annotated, BACKLOG.md, e2e header comment. - [x] ADR added in `docs/architecture/`. - [x] Demo note in `docs/demo-script.md` — n/a: no user-visible change. The openbaar register shows the same two statuses for the same registrations; only where they come from changed. ## Notes for reviewers **The decision I'd most like a second opinion on** is the one the issue didn't settle: what happens to INGEDIEND. Objecten held only INGESCHREVEN records, so re-sourcing forced a choice between (a) the ACL also writing on submit, (b) a public register that lists only actual registrations, or (c) a hybrid keeping both kanalen. I took (a): visible behaviour is unchanged and the register holds the whole lifecycle. (b) is arguably the better *semantics* for a public register but narrows what the portal shows and reads against PRD §68 ("~50 register entries with diverse statuses"); (c) leaves the projection half-derived from ZGW, which is the coupling ADR-0028 set out to remove. All three are laid out in ADR-0030. **The dedup key is the projected row**, `objecten:object:{url}:{status}:{reference}` — not the object URL (the ACL upserts *one object per registration*, so submit and approval notify about the same URL and the approval would be swallowed as a duplicate) and not URL+actie (a retried approval is a second `update`). Redeliveries collapse, genuine state changes don't. §8.6. **The migration drops columns rather than renaming them.** EF scaffolded renames — `resource` → `register_id`, `zaak_id` → `status` — which would have carried ZGW values into columns meaning something else, and a rebuild would then have projected that garbage. It also empties both tables: a pre-slice row describes a zaak event the new projector can't reproject, and those registrations have no RegisterRecord in Objecten either, so they're not re-derivable from the new source. Stated as a ceiling in the ADR — fine while stacks are ephemeral, backfill from Objecten if a long-lived environment ever needs it. **`run-projection-check.sh` now opens its zaak through the ACL** instead of straight against OpenZaak, because the ACL is what writes the record. A zaak created behind the ACL's back produces no projection row — that's the re-source working, not a gap. ## Three fixes CI found, none of them in the projection logic 1. **`wait-healthy.sh` matched the wrong container** (744f91a). Bring-up timed out with `TIMEOUT: 'objecten' not healthy (status=none)` while the `docker ps` it dumps showed objecten `Up 9 minutes (healthy)`. `--filter name=` is a substring match, so `objecten` also matches `objecten-db`/`objecten-redis`/`objecten-celery`, and `head -1` took whichever docker listed first — the celery worker has no healthcheck, hence `status=none`. Latent since those services landed and decided purely by listing order; `objecttypen` matches `objecttypen-db` the same way. Anchored on the compose replica suffix, which the verify scripts already do. 2. **The ACL had to be repointed at OpenZaak's IP** (7e0897a). Opening the zaak through the ACL put this check in the same bind run-domain-check.sh already handles: `400 {"name":"zaaktype","code":"bad-url","reason":"Voer een geldige URL in."}`. OpenZaak reflects the request Host into the zaaktype URL and then rejects it on zaak-create when single-label — the mechanism compose already documents on `ACL_OPENZAAK_BASEURL`. 3. **Approval arrives as `partial_update`, not `update`** (0dd26a7→b30fa66) — the one real bug in the slice. The ACL upserts with PATCH; DRF routes it through the notifying `update()` but names the action `partial_update`, so the projector dropped every approval. Only the e2e could catch it: `verify-projection` drives a submit, and per ADR-0028 the e2e is the only check that drives a *real* approval. `verify-tracing` also failed once (run 722) on a path this PR doesn't touch, and passed on a plain re-run of the same commit. Tempo logged `pusher failed to consume trace data` / `distributor_pool failing healthcheck` — it dropped spans under runner load rather than the trace chain being broken. Filed as **#156** rather than absorbed here. **Correction to the #152 PR notes:** I wrote there that celery concurrency was "the next knob" if verify-stack got tight. It isn't — `CELERY_WORKER_CONCURRENCY` already defaults to 1 in the Maykin image, so `objecten-celery` is already a single-process worker. Noted in #156. **Possible follow-up, deliberately not done here:** an `openzaak.local` network alias mirroring `objecten.local` would remove the ACL-repoint dance from both run-domain-check.sh and run-projection-check.sh. It changes the host in every zaak URL the system produces, which is too broad a ripple to land inside an unrelated slice — worth its own issue. **Known costs, all in the ADR:** submission is now two writes across two modules and eventually consistent (same posture ADR-0028 accepted for approval); projecting now depends on the ACL being reachable on the main path, not just for enrichment (NRC retries, so it converges); and OpenZaak still publishes to `zaken` with nothing in the product listening — kept because `verify-nrc` asserts that path.Reviewed-on: #155
192 lines
9.4 KiB
C#
192 lines
9.4 KiB
C#
using System.Net;
|
|
using System.Net.Http.Headers;
|
|
using System.Net.Http.Json;
|
|
using System.Text.Json.Serialization;
|
|
using Acl.Application;
|
|
|
|
namespace Acl.Infrastructure;
|
|
|
|
/// <summary>
|
|
/// The only code that talks to the Objecten API (ADR-0028). Writes the register record as an object
|
|
/// of the <c>RegisterRecord</c> objecttype registered in S-18c.
|
|
/// </summary>
|
|
public sealed class ObjectenGateway(HttpClient http, ObjectenOptions options, IClock clock) : IRegisterRecordGateway
|
|
{
|
|
// The objecttype URL + version are assigned by Objecttypen at seed time, so they are resolved by
|
|
// name on first use rather than pinned in config (same reasoning as ADR-0021).
|
|
// ponytail: memoised per instance only — the gateway is a transient typed client, so in practice
|
|
// that is one extra GET per approval against a neighbouring container. Lift it into a singleton
|
|
// cache (as CachedZaaktypeCatalog does for ZGW) if approvals ever get hot.
|
|
private Objecttype? objecttype;
|
|
|
|
public async Task UpsertAsync(RegisterRecord record, CancellationToken ct = default)
|
|
{
|
|
ArgumentNullException.ThrowIfNull(record);
|
|
|
|
var type = objecttype ??= await ResolveObjecttypeAsync(ct);
|
|
var existing = await FindExistingAsync(type.Url, record.Id, ct);
|
|
var data = new RecordDataDto(record.Id, record.Status, record.Reference);
|
|
|
|
// No existing object → create; otherwise PATCH, which appends a new record version to the same
|
|
// object. Either way the register ends up with exactly one object per registration (§8.6).
|
|
if (existing is null)
|
|
await SendAsync(HttpMethod.Post, new Uri(options.BaseUrl, "/api/v2/objects"),
|
|
new CreateObjectDto(type.Url.ToString(), NewRecord(type.Version, data)),
|
|
"Creating the register record", ct);
|
|
else
|
|
await SendAsync(HttpMethod.Patch, existing,
|
|
new PatchObjectDto(NewRecord(type.Version, data)),
|
|
"Updating the register record", ct);
|
|
}
|
|
|
|
public async Task<RegisterRecord?> GetAsync(Uri objectUrl, CancellationToken ct = default)
|
|
{
|
|
ArgumentNullException.ThrowIfNull(objectUrl);
|
|
|
|
// Fetched by the URL the notification carried, so no objecttype resolution and no search —
|
|
// unlike a write, which has to find the object for a registration id.
|
|
using var message = new HttpRequestMessage(HttpMethod.Get, objectUrl);
|
|
message.Headers.Authorization = new AuthenticationHeaderValue("Token", options.Token);
|
|
message.Headers.Add("Accept-Crs", "EPSG:4326");
|
|
|
|
using var response = await http.SendAsync(message, ct);
|
|
// The object may be gone by the time a (possibly redelivered) notification is handled —
|
|
// there is simply nothing to project, which is not a failure (§8.6).
|
|
if (response.StatusCode == HttpStatusCode.NotFound)
|
|
return null;
|
|
|
|
await EnsureSuccessAsync(response, "Reading the register record", ct);
|
|
|
|
var body = await response.Content.ReadFromJsonAsync<ReadObjectDto>(ct)
|
|
?? throw new InvalidOperationException("Objecten returned an empty object response");
|
|
var data = body.Record?.Data;
|
|
return data is null ? null : new RegisterRecord(data.Id, data.Status, data.Reference);
|
|
}
|
|
|
|
private RecordDto NewRecord(int typeVersion, RecordDataDto data) =>
|
|
new(typeVersion, data, clock.Today.ToString("yyyy-MM-dd"));
|
|
|
|
/// <summary>The URL + latest published version of the configured objecttype, read from Objecttypen.</summary>
|
|
private async Task<Objecttype> ResolveObjecttypeAsync(CancellationToken ct)
|
|
{
|
|
var page = await GetAsync<ObjecttypePage>(
|
|
new Uri(options.ObjecttypenBaseUrl, "/api/v2/objecttypes"),
|
|
options.ObjecttypenToken, crs: false, "objecttypen", ct);
|
|
|
|
var match = (page.Results ?? []).FirstOrDefault(o => o.Name == options.ObjecttypeName)
|
|
?? throw new InvalidOperationException(
|
|
$"No objecttype '{options.ObjecttypeName}' registered in Objecttypen — is the RegisterRecord seed applied?");
|
|
|
|
// Write against the highest *published* version: a draft version's schema is still being
|
|
// shaped, and objects written against it would be validated by a moving target. The objecttype
|
|
// carries its versions as URLs, so each is fetched for its status (the collection response
|
|
// gives no status) — once per gateway instance, alongside the lookup above.
|
|
var latest = 0;
|
|
foreach (var versionUrl in match.Versions ?? [])
|
|
{
|
|
var version = await GetAsync<ObjecttypeVersionDto>(
|
|
new Uri(versionUrl), options.ObjecttypenToken, crs: false, "objecttype version", ct);
|
|
if (version.Status == "published" && version.Version > latest)
|
|
latest = version.Version;
|
|
}
|
|
|
|
if (latest == 0)
|
|
throw new InvalidOperationException($"Objecttype '{options.ObjecttypeName}' has no published version");
|
|
|
|
return new Objecttype(new Uri(match.Url), latest);
|
|
}
|
|
|
|
/// <summary>The URL of the object already holding this registration's record, or null if there is none.</summary>
|
|
private async Task<Uri?> FindExistingAsync(Uri objecttypeUrl, string id, CancellationToken ct)
|
|
{
|
|
var query = new Uri(options.BaseUrl,
|
|
"/api/v2/objects?type=" + Uri.EscapeDataString(objecttypeUrl.ToString()) +
|
|
"&data_attrs=id__exact__" + Uri.EscapeDataString(id));
|
|
var page = await GetAsync<ObjectPage>(query, options.Token, crs: true, "objects", ct);
|
|
var match = (page.Results ?? []).FirstOrDefault();
|
|
return match is null ? null : new Uri(match.Url);
|
|
}
|
|
|
|
private async Task<T> GetAsync<T>(Uri uri, string token, bool crs, string label, CancellationToken ct)
|
|
{
|
|
using var message = new HttpRequestMessage(HttpMethod.Get, uri);
|
|
message.Headers.Authorization = new AuthenticationHeaderValue("Token", token);
|
|
if (crs)
|
|
message.Headers.Add("Accept-Crs", "EPSG:4326");
|
|
|
|
using var response = await http.SendAsync(message, ct);
|
|
await EnsureSuccessAsync(response, $"Querying {label}", ct);
|
|
|
|
return await response.Content.ReadFromJsonAsync<T>(ct)
|
|
?? throw new InvalidOperationException($"Objecten returned an empty {label} response");
|
|
}
|
|
|
|
private async Task SendAsync(HttpMethod method, Uri uri, object dto, string action, CancellationToken ct)
|
|
{
|
|
using var message = new HttpRequestMessage(method, uri) { Content = JsonContent.Create(dto) };
|
|
message.Headers.Authorization = new AuthenticationHeaderValue("Token", options.Token);
|
|
// The Objecten API is a geo API: it requires the CRS headers on reads and writes alike.
|
|
message.Headers.Add("Accept-Crs", "EPSG:4326");
|
|
message.Content.Headers.Add("Content-Crs", "EPSG:4326");
|
|
// As with OpenZaak, Objecten runs behind uwsgi, which rejects a chunked request body.
|
|
await message.Content.LoadIntoBufferAsync(ct);
|
|
|
|
using var response = await http.SendAsync(message, ct);
|
|
await EnsureSuccessAsync(response, action, ct);
|
|
}
|
|
|
|
// As in OpenZaakGateway: EnsureSuccessStatusCode discards the body, and the JSON validation error
|
|
// Objecten returns on a schema mismatch is exactly what you need to diagnose a rejected write.
|
|
private static async Task EnsureSuccessAsync(HttpResponseMessage response, string action, CancellationToken ct)
|
|
{
|
|
if (response.IsSuccessStatusCode)
|
|
return;
|
|
|
|
var body = await response.Content.ReadAsStringAsync(ct);
|
|
throw new HttpRequestException($"{action} failed: {(int)response.StatusCode} {response.ReasonPhrase}. {body}");
|
|
}
|
|
|
|
private sealed record Objecttype(Uri Url, int Version);
|
|
|
|
private sealed record ObjecttypePage(
|
|
[property: JsonPropertyName("results")] IReadOnlyList<ObjecttypeDto>? Results);
|
|
|
|
private sealed record ObjecttypeDto(
|
|
[property: JsonPropertyName("url")] string Url,
|
|
[property: JsonPropertyName("name")] string? Name,
|
|
[property: JsonPropertyName("versions")] IReadOnlyList<string>? Versions);
|
|
|
|
private sealed record ObjecttypeVersionDto(
|
|
[property: JsonPropertyName("version")] int Version,
|
|
[property: JsonPropertyName("status")] string? Status);
|
|
|
|
private sealed record ObjectPage(
|
|
[property: JsonPropertyName("results")] IReadOnlyList<ObjectDto>? Results);
|
|
|
|
private sealed record ObjectDto(
|
|
[property: JsonPropertyName("url")] string Url);
|
|
|
|
private sealed record ReadObjectDto(
|
|
[property: JsonPropertyName("record")] ReadRecordDto? Record);
|
|
|
|
private sealed record ReadRecordDto(
|
|
[property: JsonPropertyName("data")] RecordDataDto? Data);
|
|
|
|
private sealed record CreateObjectDto(
|
|
[property: JsonPropertyName("type")] string Type,
|
|
[property: JsonPropertyName("record")] RecordDto Record);
|
|
|
|
private sealed record PatchObjectDto(
|
|
[property: JsonPropertyName("record")] RecordDto Record);
|
|
|
|
private sealed record RecordDto(
|
|
[property: JsonPropertyName("typeVersion")] int TypeVersion,
|
|
[property: JsonPropertyName("data")] RecordDataDto Data,
|
|
[property: JsonPropertyName("startAt")] string StartAt);
|
|
|
|
private sealed record RecordDataDto(
|
|
[property: JsonPropertyName("id")] string Id,
|
|
[property: JsonPropertyName("status")] string Status,
|
|
[property: JsonPropertyName("reference")] string? Reference);
|
|
}
|