fix(infra): cap celery workers at 2 so the shared node stops OOM-killing CI (closes #182) #183

2026-09-25T10:09:11Z

not commented

2026-09-25 10:09:11 +00:00

What & why

verify-stack is being killed by the OOM controller on the shared Talos node, on main (run 827) and on #180 (run 830). The cause is Celery: with CELERY_WORKER_CONCURRENCY unset, oz-celery and nrc-celery each fork one worker per CPU. That's 22 each on the lab node, 49 Celery processes at about 225 MB apiece. Details and the kernel log evidence are in #182.

This sets CELERY_WORKER_CONCURRENCY: "2" in the oz and nrc env groups:

compose (&oz-env, &nrc-env): what verify-stack starts inside dind.
chart (envGroups.oz / .nrc): the deployed demo on the same node.

Both images' /celery_worker.sh honour the variable; I checked in the running pods. Web, init and beat containers share the anchors and ignore it. objecten-celery already defaults to 1.

Closes #182

Definition of Done

Linked Gitea issue (above).
Failing test committed before the implementation. (Resource setting; the evidence is the OOM log in #182.)
Conventional Commits referencing the issue (refs #NN).
CI green. This PR's verify-stack run is the check.
docker compose config renders the variable into all 7 services on the two anchors.
Docs: comments next to the setting, following the existing uWSGI notes.

Notes for reviewers

make k8s-lint and make k8s-drift pass.
Not applied live. I couldn't patch the running cluster from my session. After merge, the deploy updates the oz-env / nrc-env ConfigMaps. The celery pods only pick that up on restart, and the deploy step restarts only this repo's nine services. So run once:
kubectl -n big rollout restart deploy/oz-celery deploy/nrc-celery
Why 2 and not 1: this matches UWSGI_THREADS: "2", and it keeps one notification delivery from blocking behind a slow task. It cuts roughly 40 processes, about 9 GB RSS (less in practice, because forked workers share pages).
Longer term: CI and the demo share one 15 GB VM. Resource requests on the runner, or moving the runner off the node, would stop one from starving the other.

🤖 Generated with Claude Code

## What & why `verify-stack` is being killed by the OOM controller on the shared Talos node, on main (run 827) and on #180 (run 830). The cause is Celery: with `CELERY_WORKER_CONCURRENCY` unset, `oz-celery` and `nrc-celery` each fork one worker per CPU. That's 22 each on the lab node, 49 Celery processes at about 225 MB apiece. Details and the kernel log evidence are in #182. This sets `CELERY_WORKER_CONCURRENCY: "2"` in the oz and nrc env groups: - **compose** (`&oz-env`, `&nrc-env`): what `verify-stack` starts inside `dind`. - **chart** (`envGroups.oz` / `.nrc`): the deployed demo on the same node. Both images' `/celery_worker.sh` honour the variable; I checked in the running pods. Web, init and beat containers share the anchors and ignore it. `objecten-celery` already defaults to 1. Closes #182 ## Definition of Done - [x] Linked Gitea issue (above). - [ ] Failing test committed before the implementation. *(Resource setting; the evidence is the OOM log in #182.)* - [x] Conventional Commits referencing the issue (`refs #NN`). - [x] CI green. This PR's `verify-stack` run is the check. - [x] `docker compose config` renders the variable into all 7 services on the two anchors. - [x] Docs: comments next to the setting, following the existing uWSGI notes. ## Notes for reviewers - `make k8s-lint` and `make k8s-drift` pass. - **Not applied live.** I couldn't patch the running cluster from my session. After merge, the deploy updates the `oz-env` / `nrc-env` ConfigMaps. The celery pods only pick that up on restart, and the deploy step restarts only this repo's nine services. So run once: `kubectl -n big rollout restart deploy/oz-celery deploy/nrc-celery` - **Why 2 and not 1:** this matches `UWSGI_THREADS: "2"`, and it keeps one notification delivery from blocking behind a slow task. It cuts roughly 40 processes, about 9 GB RSS (less in practice, because forked workers share pages). - **Longer term:** CI and the demo share one 15 GB VM. Resource requests on the runner, or moving the runner off the node, would stop one from starving the other. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

not added 1 commit 2026-09-25 10:09:12 +00:00

fix(infra): cap celery workers at 2 so the shared node stops OOM-killing CI (refs #182 )

CI / lint (pull_request) Successful in 1m55s

Details

CI / k8s (pull_request) Successful in 11s

Details

CI / build (pull_request) Successful in 1m20s

Details

CI / unit (pull_request) Successful in 1m27s

Details

CI / frontend (pull_request) Successful in 3m29s

Details

CI / mutation (pull_request) Successful in 5m51s

Details

CI / verify-stack (pull_request) Successful in 11m37s

Details

c7f06b35fa

Unset CELERY_WORKER_CONCURRENCY makes oz-celery and nrc-celery fork one
process per CPU — 22 each on the lab node, ~225 MB apiece — and Talos'
OOM controller was killing the runner mid-verify-stack. Same lever as the
uWSGI caps (#144/#145/#147); objecten-celery already defaults to 1.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

not merged commit 594fdde227 into main

2026-09-25 11:11:47 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: eho/register-referentie#183