register-referentie

eho/register-referentie

Fork 0

Author	SHA1	Message	Date
not	60d556b46e	docs(architecture): record why single-binary Tempo has its ingester health check off (refs #156 ) CI / lint (pull_request) Successful in 1m41s Details CI / build (pull_request) Successful in 1m8s Details CI / unit (pull_request) Successful in 1m23s Details CI / frontend (pull_request) Successful in 3m21s Details CI / mutation (pull_request) Successful in 6m32s Details CI / verify-stack (pull_request) Successful in 9m31s Details	2026-09-01 09:45:28 +02:00
not	e54dbe9d5d	fix(observability): stop Tempo evicting its only ingester under load (refs #156 ) `verify-tracing` flaked on run 722: `FAIL — no single trace spanned ['bff', 'projection-api']`, green on a plain re-run of the same commit. Not a broken trace chain — Tempo could not ingest: removing distributor_pool failing healthcheck addr=127.0.0.1:9095 reason="rpc error: code = DeadlineExceeded" pusher failed to consume trace data err="context canceled" (x18) Root cause is the mechanism of the data loss, not whatever caused the stall. Tempo runs single-binary, so distributor and ingester are the same process and the distributor's ingester pool holds exactly one, in-process, member. dskit still health-checks it over loopback gRPC with a 1s deadline; on the shared runner a transient stall blows that, the only ingester is dropped from the pool, and every push fails until the next 15s check interval — spans silently lost. With one in-process ingester the check can never route around a failure, so it can only ever discard data. Fix: `ingester_client.pool_config.healthcheckenabled: false`. This addresses both candidate triggers (GC pressure near `mem_limit`, CPU contention) at the point where they turn into lost data, so `mem_limit: 400m` stays untouched — raising it would risk reintroducing the verify-e2e OOM of #144 on a memory-tight runner. Also print `tempo_distributor_ingester_clients` on the check's failure path: a recurrence then names Tempo-dropped-spans instead of costing another container-log dive, since from the check's side that is indistinguishable from missing instrumentation. Verified against the built image: config parses (`-config.verify`), the effective `/status/config` reports `healthcheckenabled: false`, and the diagnostic reads the metric off a live Tempo.	2026-09-01 09:45:07 +02:00

Author

SHA1

Message

Date

not

60d556b46e

docs(architecture): record why single-binary Tempo has its ingester health check off (refs #156 )

CI / lint (pull_request) Successful in 1m41s

Details

CI / build (pull_request) Successful in 1m8s

Details

CI / unit (pull_request) Successful in 1m23s

Details

CI / frontend (pull_request) Successful in 3m21s

Details

CI / mutation (pull_request) Successful in 6m32s

Details

CI / verify-stack (pull_request) Successful in 9m31s

Details

2026-09-01 09:45:28 +02:00

not

e54dbe9d5d

fix(observability): stop Tempo evicting its only ingester under load (refs #156 )

`verify-tracing` flaked on run 722: `FAIL — no single trace spanned ['bff',
'projection-api']`, green on a plain re-run of the same commit. Not a broken trace
chain — Tempo could not ingest:

    removing distributor_pool failing healthcheck addr=127.0.0.1:9095
      reason="rpc error: code = DeadlineExceeded"
    pusher failed to consume trace data  err="context canceled"   (x18)

Root cause is the mechanism of the data loss, not whatever caused the stall. Tempo
runs single-binary, so distributor and ingester are the *same process* and the
distributor's ingester pool holds exactly one, in-process, member. dskit still
health-checks it over loopback gRPC with a 1s deadline; on the shared runner a
transient stall blows that, the only ingester is dropped from the pool, and every
push fails until the next 15s check interval — spans silently lost. With one
in-process ingester the check can never route around a failure, so it can only ever
discard data.

Fix: `ingester_client.pool_config.healthcheckenabled: false`. This addresses both
candidate triggers (GC pressure near `mem_limit`, CPU contention) at the point where
they turn into lost data, so `mem_limit: 400m` stays untouched — raising it would
risk reintroducing the verify-e2e OOM of #144 on a memory-tight runner.

Also print `tempo_distributor_ingester_clients` on the check's failure path: a
recurrence then names Tempo-dropped-spans instead of costing another container-log
dive, since from the check's side that is indistinguishable from missing
instrumentation.

Verified against the built image: config parses (`-config.verify`), the effective
`/status/config` reports `healthcheckenabled: false`, and the diagnostic reads the
metric off a live Tempo.

2026-09-01 09:45:07 +02:00

Compare commits

Diff Content Not Available