eho/register-referentie
1
0
Fork 0
Code Issues 10 Pull Requests 2 Actions 1 Packages Projects Releases 1 Wiki Activity

verify-stack killed by OOM: celery workers fork one process per CPU on the shared node #182

New Issue
Closed
opened 2026-09-25 10:08:42 +00:00 by not · 0 comments
not commented 2026-09-25 10:08:42 +00:00
Contributor
Copy Link
Copy Source

Bug

verify-stack fails on main (run 827) and on PR #180 (run 830). The logs end mid-step with no error.

Cause

The Gitea runner runs on the same single Talos node (15 GB) as the deployed stack, and the node is out of memory:

  • Talos' OOM controller SIGKILLed the runner pod (dind exit 137) at 09:52:59, the moment job 3194 died. It also keeps killing oz-celery (36 restarts) and nrc-celery (32).
  • oz-celery and nrc-celery run 49 celery processes between them at about 225 MB each. CELERY_WORKER_CONCURRENCY is unset, so each worker forks one process per CPU (22 on this node). verify-stack starts the compose stack inside dind, with the same defaults, on top of that.

Fix

Set CELERY_WORKER_CONCURRENCY: "2" in the oz and nrc env groups (compose anchors and chart). This is the same lever as the uWSGI caps in #144/#145/#147. Objecten's worker already defaults to 1.

## Bug `verify-stack` fails on main (run 827) and on PR #180 (run 830). The logs end mid-step with no error. ## Cause The Gitea runner runs on the same single Talos node (15 GB) as the deployed stack, and the node is out of memory: - Talos' OOM controller SIGKILLed the runner pod (`dind` exit 137) at 09:52:59, the moment job 3194 died. It also keeps killing `oz-celery` (36 restarts) and `nrc-celery` (32). - `oz-celery` and `nrc-celery` run **49 celery processes** between them at about 225 MB each. `CELERY_WORKER_CONCURRENCY` is unset, so each worker forks one process per CPU (22 on this node). `verify-stack` starts the compose stack inside `dind`, with the same defaults, on top of that. ## Fix Set `CELERY_WORKER_CONCURRENCY: "2"` in the oz and nrc env groups (compose anchors and chart). This is the same lever as the uWSGI caps in #144/#145/#147. Objecten's worker already defaults to 1.
not closed this issue 2026-09-25 11:11:49 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
area:acl
area:bff
area:docs
area:domain
area:event-subscriber
area:infra
area:portal-behandel
area:portal-beheer
area:portal-openbaar
area:portal-self-service
area:projection
area:workflow
type:adr-proposal
type:bug
type:chore
type:slice
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
eho (Edwin van den Houdt)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: eho/register-referentie#182