main
2
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 not
|
d5dfbdc0b2 |
feat(obs): Prometheus metrics on /metrics + golden-signal Grafana dashboard (closes #124) (#129)
## What & why S-16c, the last of the S-16 (#17) split, on top of the backplane (#122) and distributed tracing (#123). The five .NET services now expose OpenTelemetry **metrics** in Prometheus format at `/metrics`; Prometheus scrapes each (one job per service); and Grafana ships a pre-built **Request path — golden signals** dashboard (traffic / errors / latency / saturation), split by service. Closes #124 ### How - Each service adds `.WithMetrics(AddAspNetCoreInstrumentation + AddHttpClientInstrumentation + AddMeter("System.Runtime") + AddPrometheusExporter)` and maps `/metrics`. Same shape as the S-16b tracing wiring already in these `Program.cs` files. - `infra/observability/prometheus/prometheus.yml`: one scrape job per service (`acl`, `domain`, `bff`, `event-subscriber`, `projection-api`), reached by compose service name. - `infra/observability/grafana/provisioning/dashboards/`: dashboard provider + `golden-signals.json` (baked into the Grafana image by the existing `COPY provisioning/`). - `verify-metrics` (new CI verify-stack step + Makefile target): generates BFF traffic and asserts Prometheus scraped the golden-signal metric from every service. Mirrors `verify-tracing`. ### Dependency (CLAUDE.md §13/§14) Adds `OpenTelemetry.Exporter.Prometheus.AspNetCore` `1.17.0-beta.1` (matched to the `1.17.0` core already in use). It gives the OTel-native `/metrics` pull endpoint; replacing it would mean hand-rolling Prometheus exposition over a `MeterListener`; the risk is that it is a **prerelease** package (the whole OTel .NET Prometheus line is `-beta`) — pinned, wired only in `Program.cs`, and gated by `verify-metrics`. Recorded in **ADR-0024**. ## Definition of Done - [x] Linked Gitea issue (#124). - [x] Failing test committed before the implementation (`test(bff): /metrics exposes http-server request duration`). - [x] Implementation makes the test pass. - [ ] CI green — pending Gitea Actions run. - [x] `docker compose up` reaches green health within 3 min (backplane images unchanged in shape; not on the health gate, ADR-0023). - [x] Docs updated — demo-script S-16c entry. - [x] ADR added — ADR-0024. - [x] Demo note in `docs/demo-script.md`. ## Notes for reviewers - `/health` polls are counted as traffic (metrics aren't path-filtered, unlike traces). Fine for a demo dashboard and honest — real load stacks on top. - `projection-api` has no Stryker config (unchanged); the four mutated services carry the metrics wiring in `Program.cs`, same as the merged S-16b tracing code. - Metric names verified against a live service: `http_server_request_duration_seconds{,_bucket,_count}`, label `http_response_status_code`, `dotnet_process_cpu_time_seconds_total`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Reviewed-on: #129 |
||
|
not
|
4274fd30d1 |
feat(infra): observability backplane — Tempo + Prometheus + Grafana (S-16a, closes #122) (#125)
## What & why S-16a, the first of the **S-16 split** (#17 closed → #122/#123/#124, §13). Stands up a local, CI-friendly observability backplane so traces (S-16b) and metrics (S-16c) have somewhere to land, viewable in one Grafana. - **Grafana Tempo** — OTLP trace ingest (gRPC 4317 / HTTP 4318), local storage. - **Prometheus** — scrapes itself for now; service `/metrics` targets arrive in S-16c. - **Grafana** — Tempo + Prometheus datasources auto-provisioned with fixed uids (`tempo`, `prometheus`), exposed on :3000. All three are small **built images** with config baked in (`infra/observability/`), on the existing `cg` network. **No OTLP collector** (Tempo ingests OTLP directly; Prometheus scrapes) and **no config-volume seeding** — the tools aren't verbatim CG peer modules, so a 3-line `COPY` Dockerfile is the simpler path that still reaches sibling containers on the CI runner (**ADR-0023**). ### Verified, not assumed `make verify-observability` (new CI `verify-stack` step, run early) asks Grafana to reach both datasources — Prometheus via its health method, Tempo via the datasource proxy (Tempo's plugin implements no health method) — so it proves the datasources are wired, not merely that containers booted. Validated locally against the three containers (no external egress): Grafana healthy, both datasources reachable. Closes #122 ## Definition of Done - [x] Failing test committed first (`verify-observability` fails with no backplane). - [x] Implementation makes it pass; verified locally. - [x] Conventional Commits referencing the issue (`refs #122`). - [ ] CI green — awaiting Gitea Actions (verify-stack now includes the observability step; `docker compose config` validates locally). - [ ] `docker compose up` reaches green health within 3 min — new containers are lightweight and off the health-gate list. - [x] Docs — ADR-0023, demo-script, BACKLOG sync. - [x] ADR added — `docs/architecture/adr-0023-observability-stack.md`. - [x] Demo note in `docs/demo-script.md`. ## Notes for reviewers - **No app changes** — this is pure infra; the five services are untouched (instrumentation is #123/#124). - **Ports:** Grafana 3000 (admin/admin, anonymous viewer on), Prometheus 9090; Tempo internal to `cg`. - **CI:** the three containers are added to the failure log-dump list; deliberately **not** added to `WAIT_SVCS` (the check polls Grafana itself, so no in-image healthcheck tool is needed). Trades ~3 small image builds per run. - **Next:** #123 wires OTLP export + `AddAspNetCoreInstrumentation`/`AddHttpClientInstrumentation` into the five hosts so a request becomes one connected trace in Tempo. Reviewed-on: #125 |