docs(architecture): record why single-binary Tempo has its ingester health check off (refs #156)

`verify-tracing` flaked on run 722: `FAIL — no single trace spanned ['bff',
'projection-api']`, green on a plain re-run of the same commit. Not a broken trace
chain — Tempo could not ingest:

    removing distributor_pool failing healthcheck addr=127.0.0.1:9095
      reason="rpc error: code = DeadlineExceeded"
    pusher failed to consume trace data  err="context canceled"   (x18)

Root cause is the mechanism of the data loss, not whatever caused the stall. Tempo
runs single-binary, so distributor and ingester are the *same process* and the
distributor's ingester pool holds exactly one, in-process, member. dskit still
health-checks it over loopback gRPC with a 1s deadline; on the shared runner a
transient stall blows that, the only ingester is dropped from the pool, and every
push fails until the next 15s check interval — spans silently lost. With one
in-process ingester the check can never route around a failure, so it can only ever
discard data.

Fix: `ingester_client.pool_config.healthcheckenabled: false`. This addresses both
candidate triggers (GC pressure near `mem_limit`, CPU contention) at the point where
they turn into lost data, so `mem_limit: 400m` stays untouched — raising it would
risk reintroducing the verify-e2e OOM of #144 on a memory-tight runner.

Also print `tempo_distributor_ingester_clients` on the check's failure path: a
recurrence then names Tempo-dropped-spans instead of costing another container-log
dive, since from the check's side that is indistinguishable from missing
instrumentation.

Verified against the built image: config parses (`-config.verify`), the effective
`/status/config` reports `healthcheckenabled: false`, and the diagnostic reads the
metric off a live Tempo.

docs/architecture/adr-0023-observability-stack.md

@@ -67,6 +67,14 @@ itself, so no in-image healthcheck tool is required.
 - Three more images built each CI run (kept small; not on the health-gate list).
 - Storage is ephemeral container fs — a demo backplane, not a retention target.
   Object storage for Tempo / remote-write for Prometheus is a later concern.
+- Tempo runs **single-binary**, so its distributor and ingester are one process and
+  some of its distributed-mode machinery is not just redundant but harmful. Its
+  ingester-pool health check is disabled (`ingester_client.pool_config`) because with
+  a single in-process ingester the check can never route around a failure — a 1s
+  loopback-gRPC deadline missed under CI load only evicted the one ingester and made
+  Tempo drop spans, which is how `verify-tracing` flaked (#156). Expect the same
+  shape from other distributed-mode knobs if we tune them; the fix is to switch to
+  real multi-ingester Tempo, not to re-enable them here.
 
 ## Coupling rules touched (CLAUDE.md §8)
 

docs/architecture/adr-0030-projection-sourced-from-the-register.md

@@ -34,6 +34,12 @@ notification points at. The projection is a cache of the register; ZGW is no lon
   as a kenmerk — so the record is read back through the ACL (`POST /register-records/read`).
   §8.1 applies to Objecten exactly as ADR-0028 established: the ACL is the only code that talks
   to it.
+- The accepted acties are `create`, `update` and `partial_update`. The last one is not
+  defensive breadth: the ACL upserts with PATCH, and DRF routes a PATCH through the notifying
+  `update()` while naming the action `partial_update` — which is what Objecten publishes. So
+  every approval arrives as `partial_update`, and accepting only `create`/`update` drops the
+  one state change this slice exists to project. `destroy` is deliberately not accepted:
+  removing a registration from the public register is its own decision.
 - The record already carries `id`, `status` and `reference`, so the row is the record. The
   zaak-shaped surface goes: `IsZaakCreated`, `IsZaakStatusSet`, `ZaakUrl`, `ZaakId`, and
   `ToEntry`'s `Resource == "status"` inference are replaced by `IsRegisterRecordWritten` +

infra/observability/tempo/tempo.yaml

@@ -25,3 +25,15 @@ storage:
       path: /var/tempo/blocks
     wal:
       path: /var/tempo/wal
+
+# #156: don't let the distributor evict its own ingester. Tempo runs single-binary here, so the
+# distributor and the ingester are the same process and the "pool" holds exactly one, in-process,
+# member. dskit still health-checks it over loopback gRPC with a 1s deadline (checkinterval 15s);
+# on the shared CI runner a transient stall blows that deadline, the only ingester is dropped from
+# the pool ("removing distributor_pool failing healthcheck"), and every push then fails ("pusher
+# failed to consume trace data", err="context canceled") until the next check — silently losing
+# spans, which is how verify-tracing flaked. With one in-process ingester the check can never route
+# around a failure, so it can only ever discard data. Turn it off.
+ingester_client:
+  pool_config:
+    healthcheckenabled: false

infra/run-projection-check.sh

@@ -12,11 +12,15 @@
 # which is the point of the re-source.
 #
 # All in-network, reaching services by container IP — single-label hosts aren't URL-valid and
-# the runner can't reach published ports (gitea-actions-gotchas.md §5/§6). Does NOT manage the stack
-# lifecycle (the caller owns bring-up + teardown). Plain docker primitives only. See ADR-0007/0008/0030.
+# the runner can't reach published ports (gitea-actions-gotchas.md §5/§6). Does not own the stack
+# lifecycle (the caller brings it up and tears it down), but does recreate the `acl` service to
+# repoint it — see below, and run-domain-check.sh, which does the same. Plain docker primitives only.
+# See ADR-0007/0008/0030.
 set -euo pipefail
 
 here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+root="$(cd "$here/.." && pwd)"
+compose="$root/infra/docker-compose.yml"
 WEBHOOK_AUTH="${NOTIFICATION_WEBHOOK_TOKEN:-Bearer big-reference-notifications}"
 
 cleanup() { docker rm -f rr-pverify rr-pquery >/dev/null 2>&1 || true; }
@@ -54,6 +58,19 @@ docker cp "$here/local/register-abonnement.py" "$drv:/subscribe.py" >/dev/null
 docker start -a "$drv"
 docker rm -f rr-pverify >/dev/null
 
+# OpenZaak reflects the request Host into the zaaktype `url` it returns, and then rejects that same
+# URL on zaak-create when the host is single-label ("Voer een geldige URL in."). The stack's ACL is
+# configured with `http://openzaak:8000/`, so it must be repointed at OpenZaak's container IP before
+# it can open a zaak — exactly what run-domain-check.sh does, and the same class of constraint as the
+# `objecten.local` alias (ADR-0029). The ACL resolves the zaaktype itself (S-27, ADR-0021), so the
+# base URL is the only thing to inject.
+echo ">> recreating the acl service pointed at OpenZaak's IP"
+ACL_OPENZAAK_BASEURL="http://$oz_ip:8000/" docker compose -f "$compose" up -d acl
+WAIT_TIMEOUT="${WAIT_TIMEOUT:-120}" bash "$here/wait-healthy.sh" acl
+# The container is replaced, so its IP may have changed.
+acl="$(docker ps -q --filter 'name=[-_]acl[-_]' | head -1)"
+acl_ip="$(ip "$acl")"
+
 echo ">> opening a zaak through the ACL (which writes the INGEDIEND register record)"
 reference="PROJ-$(date +%s)"
 zaak_url="$(docker run --rm --network "$net" curlimages/curl:latest \

infra/tracing-check.py

@@ -59,6 +59,20 @@ def services_in_trace(trace_id):
     return names
 
 
+def tempo_ingest_state():
+    """#156: distinguish a broken trace chain from Tempo dropping spans. `ingester_clients` is 0
+    when the distributor has evicted its (single, in-process) ingester over a failed loopback
+    health check — pushes fail and spans are lost, which looks identical to missing instrumentation
+    from here. Diagnostics only; never fails the check."""
+    try:
+        for line in _get(f"{TEMPO}/metrics").decode().splitlines():
+            if line.startswith("tempo_distributor_ingester_clients "):
+                return f"tempo {line.strip()} (0 = no ingester in the pool — evicted, so pushes\n       are failing and spans are being dropped; see #156)"
+    except Exception as e:
+        return f"tempo /metrics unreadable: {e}"
+    return "tempo_distributor_ingester_clients not reported"
+
+
 def main():
     deadline = time.time() + TIMEOUT
     generate_traffic()
@@ -74,6 +88,7 @@ def main():
         generate_traffic()
     print(f"FAIL — no single trace spanned {sorted(WANT)}; services seen: {sorted(seen)}",
           file=sys.stderr)
+    print(f"       {tempo_ingest_state()}", file=sys.stderr)
     return 1
 
 

infra/wait-healthy.sh

@@ -15,9 +15,13 @@ set -euo pipefail
 timeout="${WAIT_TIMEOUT:-420}"
 deadline=$(( $(date +%s) + timeout ))
 
-# compose service name -> container id. The name filter matches both docker
-# compose ("infra-openzaak-1") and podman-compose ("infra_openzaak_1") naming.
-cid_for() { docker ps -aq --filter "name=$1" | head -1; }
+# compose service name -> container id. `--filter name=` is a substring match, so it is anchored on
+# the compose replica suffix — otherwise 'objecten' also matches objecten-db / objecten-redis /
+# objecten-celery, and 'objecttypen' matches objecttypen-db. Whichever docker listed first won, so a
+# service with a sibling that has no healthcheck timed out with status=none while it was in fact
+# healthy. The pattern matches both docker compose ("infra-objecten-1") and podman-compose
+# ("infra_objecten_1") naming; the same anchoring the verify check scripts use.
+cid_for() { docker ps -aq --filter "name=$1[-_][0-9]+\$" | head -1; }
 
 for svc in "$@"; do
   echo "waiting for '$svc' to be healthy (timeout ${timeout}s)..."

services/event-subscriber/EventSubscriber.Application/Notification.cs

@@ -18,10 +18,20 @@ public sealed record Notification(
     string Actie,
     Uri ResourceUrl)
 {
-    /// <summary>A register record written to Objecten — <c>create</c> on submit, <c>update</c> on
-    /// approval, since the ACL upserts the same object for a registration (§8.6).</summary>
+    /// <summary>
+    /// A register record written to Objecten — <c>create</c> on submit and <c>partial_update</c> on
+    /// approval, since the ACL upserts the same object for a registration (§8.6).
+    /// </summary>
+    /// <remarks>
+    /// <c>partial_update</c> is what a PATCH actually reports: DRF routes it through the notifying
+    /// <c>update()</c> but names the action <c>partial_update</c>, and that is what Objecten puts in
+    /// the notification. <c>update</c> is accepted too, so a PUT-shaped write would project the same
+    /// way. <c>destroy</c> is deliberately not: removing a registration from the public register is
+    /// its own decision, not a side effect of this one.
+    /// </remarks>
     public bool IsRegisterRecordWritten =>
-        Kanaal == "objecten" && Resource == "object" && Actie is "create" or "update";
+        Kanaal == "objecten" && Resource == "object"
+        && Actie is "create" or "update" or "partial_update";
 
     /// <summary>The object holding the register record. For a <c>resource: object</c> notification
     /// Objecten sends the object as both <c>hoofdObject</c> and <c>resourceUrl</c> — the object is

services/event-subscriber/EventSubscriber.Tests/NotificationProjectorTests.cs

@@ -38,13 +38,17 @@ public sealed class NotificationProjectorTests
         Assert.Equal("REG-2026-0001", entry.Reference);
     }
 
-    [Fact]
-    public async Task approval_updates_the_same_row_from_ingediend_to_ingeschreven()
+    // The ACL PATCHes the same object on approval. DRF routes a PATCH through `update()` but reports
+    // the action as `partial_update`, which is what Objecten puts in the notification — so accepting
+    // only `create`/`update` silently drops every approval.
+    [Theory]
+    [InlineData("partial_update")]
+    [InlineData("update")]
+    public async Task approval_updates_the_same_row_from_ingediend_to_ingeschreven(string actie)
     {
         var projector = Projector();
         await projector.HandleAsync(RecordWritten());
-        // The ACL PATCHes the same object on approval, so Objecten publishes an `update`.
-        await projector.HandleAsync(RecordWritten("update", status: RegistrationStatus.Ingeschreven));
+        await projector.HandleAsync(RecordWritten(actie, status: RegistrationStatus.Ingeschreven));
 
         var entry = Assert.Single(await _store.AllAsync());
         Assert.Equal(ZaakId, entry.Id);
@@ -112,7 +116,7 @@ public sealed class NotificationProjectorTests
     {
         var projector = Projector();
         await projector.HandleAsync(RecordWritten());
-        await projector.HandleAsync(RecordWritten("update", status: RegistrationStatus.Ingeschreven));
+        await projector.HandleAsync(RecordWritten("partial_update", status: RegistrationStatus.Ingeschreven));
         var callsAfterProjection = _acl.CallCount;
 
         await projector.RebuildAsync();