feat(obs): distributed traces across the .NET services (S-16b, closes #123) #126

2026-07-23T12:38:16Z

not commented

2026-07-23 12:38:16 +00:00

What & why

S-16b, second of the S-16 split, on top of the #125 backplane. The five .NET services now emit OpenTelemetry traces so a request is one connected trace across them.

Each host wires AddOpenTelemetry().WithTracing(...) with AddAspNetCoreInstrumentation (incoming) + AddHttpClientInstrumentation (outgoing) + AddOtlpExporter to Tempo.
Because every cross-service call already goes through a typed HttpClient (§8 boundaries), the W3C traceparent propagates with no manual code — bff → domain → acl → openzaak and bff → projection-api stitch into a single trace.
Service name + OTLP endpoint come from OTEL_* env set per app service in compose. /health is filtered out so liveness polls don't flood the traces.

No new ADR — ADR-0023 already records the stack + the two documented gaps (browser-side tracing is out of scope, so the trace begins at the BFF; the async Flowable-poll boundary is a separate trace).

Closes #123

Definition of Done

Failing test committed first (verify-tracing fails with no instrumentation).
Implementation makes it pass — validated locally end to end: a real connected trace spanning bff + projection-api was found in Tempo (BFF→projection→db + Tempo subset, no OpenZaak/egress).
Conventional Commits referencing the issue (refs #123).
CI green — awaiting Gitea Actions (verify-tracing added to verify-stack after verify-bff).
docker compose up health unaffected — services boot healthy even when Tempo is unreachable (exporter no-ops; verified).
Docs — demo-script + BACKLOG.
ADR — none needed (covered by ADR-0023).

Notes for reviewers

Per-service wiring, no shared lib: the block is duplicated across the five hosts by design — services don't share code across boundaries here (§8), same as the duplicated typed clients.
Packages: OpenTelemetry.Extensions.Hosting / Instrumentation.AspNetCore / Instrumentation.Http / Exporter.OpenTelemetryProtocol, all 1.17.0, pinned per-csproj (no central props file).
The check generates anonymous BFF→projection traffic (no auth, no OpenZaak), then queries Tempo (TraceQL search → fetch trace → assert both service.names present) from a python:3-slim container in-network — same idiom as run-projection-check.sh.
Next: #124 (S-16c) adds /metrics + Prometheus scrape targets + golden-signal Grafana dashboards.

## What & why S-16b, second of the S-16 split, on top of the #125 backplane. The five .NET services now emit OpenTelemetry traces so a request is **one connected trace** across them. - Each host wires `AddOpenTelemetry().WithTracing(...)` with `AddAspNetCoreInstrumentation` (incoming) + `AddHttpClientInstrumentation` (outgoing) + `AddOtlpExporter` to **Tempo**. - Because every cross-service call already goes through a typed `HttpClient` (§8 boundaries), the W3C `traceparent` propagates with no manual code — bff → domain → acl → openzaak and bff → projection-api stitch into a single trace. - Service name + OTLP endpoint come from `OTEL_*` env set per app service in compose. `/health` is filtered out so liveness polls don't flood the traces. No new ADR — ADR-0023 already records the stack + the two documented gaps (browser-side tracing is out of scope, so the trace begins at the BFF; the async Flowable-poll boundary is a separate trace). Closes #123 ## Definition of Done - [x] Failing test committed first (`verify-tracing` fails with no instrumentation). - [x] Implementation makes it pass — **validated locally end to end**: a real connected trace spanning `bff` + `projection-api` was found in Tempo (BFF→projection→db + Tempo subset, no OpenZaak/egress). - [x] Conventional Commits referencing the issue (`refs #123`). - [x] CI green — awaiting Gitea Actions (verify-tracing added to verify-stack after verify-bff). - [x] `docker compose up` health unaffected — services boot healthy even when Tempo is unreachable (exporter no-ops; verified). - [x] Docs — demo-script + BACKLOG. - [x] ADR — none needed (covered by ADR-0023). ## Notes for reviewers - **Per-service wiring, no shared lib:** the block is duplicated across the five hosts by design — services don't share code across boundaries here (§8), same as the duplicated typed clients. - **Packages:** OpenTelemetry.Extensions.Hosting / Instrumentation.AspNetCore / Instrumentation.Http / Exporter.OpenTelemetryProtocol, all 1.17.0, pinned per-csproj (no central props file). - **The check** generates anonymous BFF→projection traffic (no auth, no OpenZaak), then queries Tempo (TraceQL search → fetch trace → assert both service.names present) from a python:3-slim container in-network — same idiom as run-projection-check.sh. - **Next:** #124 (S-16c) adds `/metrics` + Prometheus scrape targets + golden-signal Grafana dashboards.

not added this to the Iteration 3 — Beheer & Observability milestone 2026-07-23 12:38:16 +00:00

not added 3 commits 2026-07-23 12:38:17 +00:00

test(infra): verify-tracing asserts one connected trace across services in Tempo (refs #123 ) b32c352f20

Generates anonymous BFF→projection-api traffic and asserts Tempo holds a single
trace containing both service.names — proving OTLP export plus traceparent
propagation across the HttpClient hop. Fails until the services are instrumented
(next commit). Runs in-network like the other verify checks.

refs #123

feat(obs): OpenTelemetry distributed tracing across the five .NET services (refs #123 ) 271c54197e

Each host adds AddOpenTelemetry().WithTracing with ASP.NET Core + HttpClient
auto-instrumentation and an OTLP exporter to Tempo; service name + endpoint from
OTEL_* env set per app service in compose. Since every cross-service call goes
through a typed HttpClient, traceparent propagates for free, so a request is one
connected trace (bff → domain/projection → acl → openzaak). /health is filtered
out. The exporter no-ops harmlessly when Tempo is unreachable (verified: domain
boots healthy with no collector).

refs #123

docs: demo note + backlog tick for distributed tracing (refs #123 )

CI / lint (pull_request) Successful in 4m29s

Details

CI / build (pull_request) Successful in 59s

Details

CI / unit (pull_request) Successful in 1m16s

Details

CI / frontend (pull_request) Successful in 2m36s

Details

CI / mutation (pull_request) Successful in 6m6s

Details

CI / verify-stack (pull_request) Failing after 13m54s

Details

8474b72bf4

refs #123

not added the type:slice area:infra labels 2026-07-23 12:38:25 +00:00

not added 1 commit 2026-07-23 13:31:56 +00:00

fix(infra): cap observability containers' memory to protect the e2e browser (refs #123 )

CI / frontend (pull_request) Successful in 2m55s

Details

CI / mutation (pull_request) Successful in 6m17s

Details

CI / verify-stack (pull_request) Successful in 5m31s

Details

CI / lint (pull_request) Successful in 1m28s

Details

CI / build (pull_request) Successful in 1m19s

Details

CI / unit (pull_request) Successful in 1m21s

Details

4ab2ef4285

verify-e2e (registration.spec) timed out on the shared CI runner — locator waits
failing at different steps across attempt/retry, the signature of memory
contention, not a logic bug (this e2e has an OOM history, d5e5fa2). The three
observability containers had no memory bound; cap them (tempo/prometheus 400m,
grafana 512m — generous vs idle) so they can't starve the app stack + the
Playwright chromium. Frees back more headroom than the OTel SDK added.

refs #123

not merged commit 88338396f6 into main

2026-07-23 14:38:27 +00:00

not referenced this issue from a commit

2026-07-23 14:38:28 +00:00

feat(obs): distributed traces across the .NET services (S-16b, closes #123) (#126)

not referenced this pull request

2026-07-23 14:41:12 +00:00

chore(ci): parallelise CI jobs now the runner has capacity >1, without re-triggering the e2e OOM #127

not referenced this issue from a commit

2026-07-23 14:42:00 +00:00

ci: parallelise jobs at runner capacity >1, keep the two heavy jobs apart (closes #127)

not referenced this pull request