eho/register-referentie
1
0
Fork 0
Code Issues 10 Pull Requests 1 Actions Packages Projects Releases 1 Wiki Activity

verify-tracing is flaky — Tempo intermittently drops spans under runner load #156

New Issue
Closed
opened 2026-08-28 12:30:22 +00:00 by not · 0 comments
not commented 2026-08-28 12:30:22 +00:00
Contributor
Copy Link
Copy Source

Observed on verify-stack run 722 (PR #155). verify-tracing failed:

FAIL — no single trace spanned ['bff', 'projection-api']; services seen: ['bff']

An identical run on the same branch 20 minutes earlier (721) passed, and the tracing path was
untouched by that PR. A plain re-run of the same job on the same commit went green, so it is
intermittent, not a regression.

It is not a broken trace chain — Tempo could not ingest. From the same run's container logs:

tempo-1 | level=warn  msg="removing distributor_pool failing healthcheck" addr=127.0.0.1:9095 \
                      reason="rpc error: code = DeadlineExceeded desc = context de..."
tempo-1 | level=error msg="pusher failed to consume trace data" err="context canceled"   (x18)

No OOM kill and no restart — Tempo stayed up but dropped spans during the window, so
projection-api's never landed and the check saw only bff.

Why this is worth fixing rather than tolerating

CLAUDE.md §15: flaky tests are fixed, not retried. The check gates merges, so every occurrence
costs a full ~20-minute verify-stack cycle, and it trains people to re-run on red — which is
exactly how a real tracing regression would get waved through.

Notes for whoever picks this up

  • tempo runs with mem_limit: 400m (set in commit d5e5fa2 to stop the backplane starving the
    app stack + Playwright on the memory-tight runner). Worth checking whether that is now too
    tight: the stack has gained containers since, most recently objecten-celery (#152).
  • CELERY_WORKER_CONCURRENCY is a dead end. The Maykin image already defaults it to 1, so
    objecten-celery is a single-process worker and there is nothing to turn down. (The #152 PR
    notes name it as "the next knob" — that is wrong; this issue supersedes it.)
  • run-tracing-check.sh polls with TRACING_TIMEOUT (CI passes 120s). If the real cause is
    ingestion lag under load rather than dropped spans, the fix is on Tempo's side — a longer
    timeout would just paper over dropped data.
  • The err="context canceled" suggests the exporter gave up on the push, so the .NET OTLP
    exporter timeout is also worth a look alongside Tempo's limits.
Observed on `verify-stack` run 722 (PR #155). `verify-tracing` failed: ``` FAIL — no single trace spanned ['bff', 'projection-api']; services seen: ['bff'] ``` An identical run on the same branch 20 minutes earlier (721) passed, and the tracing path was untouched by that PR. A plain re-run of the same job on the same commit went green, so it is intermittent, not a regression. **It is not a broken trace chain — Tempo could not ingest.** From the same run's container logs: ``` tempo-1 | level=warn msg="removing distributor_pool failing healthcheck" addr=127.0.0.1:9095 \ reason="rpc error: code = DeadlineExceeded desc = context de..." tempo-1 | level=error msg="pusher failed to consume trace data" err="context canceled" (x18) ``` No OOM kill and no restart — Tempo stayed up but dropped spans during the window, so projection-api's never landed and the check saw only `bff`. ## Why this is worth fixing rather than tolerating CLAUDE.md §15: flaky tests are fixed, not retried. The check gates merges, so every occurrence costs a full ~20-minute `verify-stack` cycle, and it trains people to re-run on red — which is exactly how a real tracing regression would get waved through. ## Notes for whoever picks this up - `tempo` runs with `mem_limit: 400m` (set in commit d5e5fa2 to stop the backplane starving the app stack + Playwright on the memory-tight runner). Worth checking whether that is now too tight: the stack has gained containers since, most recently `objecten-celery` (#152). - **`CELERY_WORKER_CONCURRENCY` is a dead end.** The Maykin image already defaults it to 1, so `objecten-celery` is a single-process worker and there is nothing to turn down. (The #152 PR notes name it as "the next knob" — that is wrong; this issue supersedes it.) - `run-tracing-check.sh` polls with `TRACING_TIMEOUT` (CI passes 120s). If the real cause is ingestion lag under load rather than dropped spans, the fix is on Tempo's side — a longer timeout would just paper over dropped data. - The `err="context canceled"` suggests the exporter gave up on the push, so the .NET OTLP exporter timeout is also worth a look alongside Tempo's limits.
not added the area:infratype:bug labels 2026-08-28 12:30:28 +00:00
not closed this issue 2026-09-01 08:31:37 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area:acl
area:bff
area:docs
area:domain
area:event-subscriber
area:infra
area:portal-behandel
area:portal-beheer
area:portal-openbaar
area:portal-self-service
area:projection
area:workflow
type:adr-proposal
type:bug
type:chore
type:slice
No labels area:infra type:bug
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
eho (Edwin van den Houdt)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: eho/register-referentie#156