feat(obs): distributed traces across the .NET services (S-16b, closes #123) (#126)

## What & why

S-16b, second of the S-16 split, on top of the #125 backplane. The five .NET services now emit OpenTelemetry traces so a request is **one connected trace** across them.

- Each host wires `AddOpenTelemetry().WithTracing(...)` with `AddAspNetCoreInstrumentation` (incoming) + `AddHttpClientInstrumentation` (outgoing) + `AddOtlpExporter` to **Tempo**.
- Because every cross-service call already goes through a typed `HttpClient` (§8 boundaries), the W3C `traceparent` propagates with no manual code — bff → domain → acl → openzaak and bff → projection-api stitch into a single trace.
- Service name + OTLP endpoint come from `OTEL_*` env set per app service in compose. `/health` is filtered out so liveness polls don't flood the traces.

No new ADR — ADR-0023 already records the stack + the two documented gaps (browser-side tracing is out of scope, so the trace begins at the BFF; the async Flowable-poll boundary is a separate trace).

Closes #123

## Definition of Done

- [x] Failing test committed first (`verify-tracing` fails with no instrumentation).
- [x] Implementation makes it pass — **validated locally end to end**: a real connected trace spanning `bff` + `projection-api` was found in Tempo (BFF→projection→db + Tempo subset, no OpenZaak/egress).
- [x] Conventional Commits referencing the issue (`refs #123`).
- [ ] CI green — awaiting Gitea Actions (verify-tracing added to verify-stack after verify-bff).
- [x] `docker compose up` health unaffected — services boot healthy even when Tempo is unreachable (exporter no-ops; verified).
- [x] Docs — demo-script + BACKLOG.
- [x] ADR — none needed (covered by ADR-0023).

## Notes for reviewers

- **Per-service wiring, no shared lib:** the block is duplicated across the five hosts by design — services don't share code across boundaries here (§8), same as the duplicated typed clients.
- **Packages:** OpenTelemetry.Extensions.Hosting / Instrumentation.AspNetCore / Instrumentation.Http / Exporter.OpenTelemetryProtocol, all 1.17.0, pinned per-csproj (no central props file).
- **The check** generates anonymous BFF→projection traffic (no auth, no OpenZaak), then queries Tempo (TraceQL search → fetch trace → assert both service.names present) from a python:3-slim container in-network — same idiom as run-projection-check.sh.
- **Next:** #124 (S-16c) adds `/metrics` + Prometheus scrape targets + golden-signal Grafana dashboards.

Reviewed-on: #126

infra/tracing-check.py

@@ -0,0 +1,81 @@
+#!/usr/bin/env python3
+"""S-16b (#123): prove distributed tracing works end to end.
+
+Generate anonymous BFF traffic (GET /openbaar/register, which the BFF serves by
+calling projection-api — no auth, no OpenZaak egress), then query Tempo and assert
+that ONE trace contains spans from both `bff` and `projection-api`. That proves the
+services export OTLP to Tempo AND that the W3C traceparent propagates across the
+HttpClient hop, stitching the request into a single connected trace.
+
+Stdlib only (urllib/json) so it runs in a bare python:3-slim container in-network.
+"""
+import json
+import os
+import sys
+import time
+import urllib.error
+import urllib.parse
+import urllib.request
+
+BFF = os.environ["BFF"]        # http://<bff-ip>:8080
+TEMPO = os.environ["TEMPO"]    # http://<tempo-ip>:3200
+TIMEOUT = int(os.environ.get("TRACING_TIMEOUT", "90"))
+WANT = {"bff", "projection-api"}  # the two services that must share one trace
+
+
+def _get(url):
+    with urllib.request.urlopen(url, timeout=10) as r:
+        return r.read()
+
+
+def generate_traffic():
+    # A non-2xx still produces spans; only total unreachability of the BFF is fatal.
+    for _ in range(3):
+        try:
+            _get(f"{BFF}/openbaar/register")
+        except urllib.error.HTTPError:
+            pass
+
+
+def search_trace_ids():
+    q = urllib.parse.quote('{ resource.service.name = "bff" }')
+    try:
+        data = json.loads(_get(f"{TEMPO}/api/search?q={q}&limit=50"))
+    except Exception:
+        return []
+    return [t["traceID"] for t in data.get("traces", [])]
+
+
+def services_in_trace(trace_id):
+    try:
+        data = json.loads(_get(f"{TEMPO}/api/traces/{trace_id}"))
+    except Exception:
+        return set()
+    names = set()
+    for batch in data.get("batches", []):
+        for attr in batch.get("resource", {}).get("attributes", []):
+            if attr.get("key") == "service.name":
+                names.add(attr.get("value", {}).get("stringValue"))
+    return names
+
+
+def main():
+    deadline = time.time() + TIMEOUT
+    generate_traffic()
+    seen = set()
+    while time.time() < deadline:
+        for tid in search_trace_ids():
+            names = services_in_trace(tid)
+            seen |= names
+            if WANT.issubset(names):
+                print(f"OK — trace {tid} spans {sorted(names)}")
+                return 0
+        time.sleep(3)
+        generate_traffic()
+    print(f"FAIL — no single trace spanned {sorted(WANT)}; services seen: {sorted(seen)}",
+          file=sys.stderr)
+    return 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())