arch(acl): ADR-0005 adopt Stryker.NET for mutation testing (refs #47)
Record the decision to adopt Stryker.NET (pinned local tool, solution mode on Acl.slnx) and to set the first repo-wide mutation baseline on the ACL: observed 95%, enforced break threshold 90%. Document the ratchet, local run, and report location in the CI runbook; add the ADR to the docs nav. Proposed in #51 (adr-proposal). Refs #47. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
68
docs/architecture/adr-0005-mutation-testing.md
Normal file
68
docs/architecture/adr-0005-mutation-testing.md
Normal file
@@ -0,0 +1,68 @@
|
||||
# ADR-0005: Stryker.NET for mutation testing, baseline on the ACL
|
||||
|
||||
- **Status:** Accepted
|
||||
- **Date:** 2026-06-25
|
||||
- **Deciders:** Respellion engineering
|
||||
- **Relates to:** S-04b (#47); proposed in #51; supports CLAUDE.md §5 (mutation ratchet) and §3 (Definition of Done)
|
||||
|
||||
## Context
|
||||
|
||||
CLAUDE.md §5 mandates Stryker on every PR with a **ratchet**: CI fails on a regression
|
||||
below the established baseline, and the baseline only ever moves up. §3 lists "mutation
|
||||
(ratchet)" as a Definition-of-Done gate for **every** slice. Yet no baseline existed — so,
|
||||
strictly, no slice could satisfy that gate. S-04b establishes it.
|
||||
|
||||
The ACL is the natural place to set the first baseline: it is the first service with real
|
||||
branching logic — `OpenZaakGateway` (HTTP contract, geo CRS headers, error handling),
|
||||
`ZgwToken` (HS256 JWT minting), and the `AclService` default-fill mapping. We need a tool
|
||||
that:
|
||||
|
||||
- mutates C# and runs the existing xUnit suite per mutant,
|
||||
- is reproducible (same version locally and in CI, no global install),
|
||||
- understands this repo's `.slnx` solution format (used repo-wide),
|
||||
- emits a break threshold CI can gate on.
|
||||
|
||||
## Decision
|
||||
|
||||
**Use [Stryker.NET](https://stryker-mutator.io/docs/stryker-net/) (`dotnet-stryker`),
|
||||
pinned as a local dotnet tool**, configured in solution mode against `Acl.slnx`.
|
||||
|
||||
- Pinned in `.config/dotnet-tools.json` (v4.15.0); `dotnet tool restore` makes
|
||||
`make mutation` reproducible from a fresh clone, locally and in CI — no global install.
|
||||
- **Solution mode** (`stryker-config.json` → `solution: Acl.slnx`) mutates the two projects
|
||||
under test (`Acl.Application`, `Acl.Infrastructure`); `Acl.Api` is untested and skipped.
|
||||
Stryker 4.15 reads `.slnx` directly, so no throwaway `.sln` shim is needed.
|
||||
- A `mutation` make target runs it; it is wired into `make ci` and a parallel Gitea Actions
|
||||
`mutation` job, keeping `make ci` an exact mirror of the pipeline.
|
||||
|
||||
**Baseline:** writing S-04b's tests surfaced that the ACL suite was thin — the initial
|
||||
score was **35%** (survivors: unasserted CRS headers, null guards, error paths, and JWT
|
||||
claims). Those tests were strengthened (killing the mutants honestly rather than lowering
|
||||
the bar), raising the score to **95%**. The enforced `break` threshold is set to **90%** —
|
||||
one-mutant headroom over the ~20-mutant surface, since a single mutant is ≈5%.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Positive:** test *strength* is gated, not just coverage; the ratchet protects the ACL's
|
||||
ZGW contract logic; the baseline is repo-wide and ratchets upward per §5.
|
||||
- **Cost:** a new dependency (`dotnet-stryker`) and a slower CI job than unit tests (~25 s on
|
||||
the small ACL). Pinned + tool-restored, so reproducible.
|
||||
- **One accepted survivor:** a mutation of the empty-response *exception message string*.
|
||||
Asserting exception message text is brittle and the behaviour (type + control flow) is
|
||||
unchanged — treated as an equivalent mutant, not a test gap.
|
||||
- **Commitment:** later slices ratchet the threshold up deliberately, never down (§5). New
|
||||
services add their own mutation run as they gain branching logic (BFF, Domain, …).
|
||||
- **Replaceable by:** no realistic .NET alternative — Stryker.NET is the tool §5 already
|
||||
names; the fallback is no mutation testing, which §5 forbids.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Global `dotnet tool install -g`** — rejected: not reproducible/pinned per clone; the
|
||||
local manifest gives every checkout and the CI runner the same version.
|
||||
- **Mutate the whole `register-referentie.slnx`** — rejected for this slice: scopes the
|
||||
baseline to services with no logic yet (BFF skeleton), diluting the signal. Each service
|
||||
opts in as it gains logic.
|
||||
- **Application-only scope** — rejected: would leave `Acl.Infrastructure`'s HTTP/JWT logic —
|
||||
the riskiest code — unguarded by the ratchet.
|
||||
- **Coverage gate instead of mutation** — rejected: line coverage does not measure whether
|
||||
tests would *catch* a regression; that is the whole point of §5.
|
||||
@@ -16,6 +16,7 @@ and CI cannot drift:
|
||||
| `lint` | `make lint` → `dotnet format … --verify-no-changes` | .NET 10 SDK |
|
||||
| `build` | `make build` → `dotnet build … -c Release` | .NET 10 SDK |
|
||||
| `unit` | `make unit` → `dotnet test … -c Release` | .NET 10 SDK |
|
||||
| `mutation` | `make mutation` → `dotnet tool restore` → `dotnet stryker` (ACL) | .NET 10 SDK |
|
||||
| `compose-smoke` | `make smoke` → seed config volumes → `up -d` (full stack) → `up --wait` durable services → `down` | container engine + compose v2 |
|
||||
|
||||
All `uses:` references are absolute, tag-pinned URLs (`https://github.com/actions/checkout@v4`,
|
||||
@@ -30,6 +31,30 @@ Actions resolves them from GitHub.
|
||||
> bare `docker compose up` no longer self-seeds; use `make up`. See
|
||||
> [gitea-actions-gotchas.md](gitea-actions-gotchas.md).
|
||||
|
||||
## Mutation testing (the ratchet)
|
||||
|
||||
The `mutation` job enforces test *strength*, not just coverage (CLAUDE.md §5).
|
||||
[Stryker.NET](https://stryker-mutator.io/docs/stryker-net/) is pinned as a local
|
||||
dotnet tool (`.config/dotnet-tools.json`), so it runs identically locally and in CI:
|
||||
|
||||
```bash
|
||||
make mutation # dotnet tool restore + dotnet stryker on the ACL
|
||||
```
|
||||
|
||||
Config lives in [`services/acl/stryker-config.json`](../../services/acl/stryker-config.json).
|
||||
It runs in **solution mode** against `Acl.slnx`, mutating the two projects under test
|
||||
(`Acl.Application`, `Acl.Infrastructure`); `Acl.Api` has no tests and is skipped.
|
||||
|
||||
**Baseline (the ratchet):** the ACL is the first service with branching logic, so it
|
||||
sets the repo-wide baseline. Observed score **95%**; enforced `break` threshold **90%**
|
||||
(one-mutant headroom over the ~20-mutant surface). Stryker exits non-zero — failing the
|
||||
job — when the score drops below `break`. Per §5 the baseline only moves **up**, and only
|
||||
as a slice's stated outcome; never lower it. New services add their own mutation run as
|
||||
they gain logic.
|
||||
|
||||
The HTML report is written to `services/acl/StrykerOutput/<timestamp>/reports/` (git-ignored);
|
||||
open it to see survived vs. killed mutants.
|
||||
|
||||
## Running the stack locally without `make` (Windows / Docker Desktop)
|
||||
|
||||
`make` and the bash helpers assume a Unix shell. To bring the whole stack up on a
|
||||
@@ -55,8 +80,9 @@ the containerized CI runner). Keep the two files in sync.
|
||||
Until the runner exists, run the full pipeline yourself before pushing:
|
||||
|
||||
```bash
|
||||
make ci # lint + build + unit + smoke — what the pipeline runs
|
||||
make ci # lint + build + unit + mutation + smoke — what the pipeline runs
|
||||
make lint # or a single stage
|
||||
make mutation # Stryker.NET ratchet on the ACL
|
||||
make smoke # compose up --wait, curl /health, tear down
|
||||
```
|
||||
|
||||
|
||||
Reference in New Issue
Block a user