fix(infra): cap celery workers at 2 so the shared node stops OOM-killing CI (closes #182) (#183)
CI / k8s (push) Successful in 9s
CI / build (push) Successful in 1m39s
CI / lint (push) Successful in 2m0s
CI / mutation (push) Canceled after 0s
CI / verify-stack (push) Canceled after 0s
CI / unit (push) Canceled after 1m9s
CI / frontend (push) Canceled after 1m16s
Deploy to Talos / deploy (push) Successful in 2m31s

## What & why

`verify-stack` is being killed by the OOM controller on the shared Talos node, on main (run 827) and on #180 (run 830). The cause is Celery: with `CELERY_WORKER_CONCURRENCY` unset, `oz-celery` and `nrc-celery` each fork one worker per CPU. That's 22 each on the lab node, 49 Celery processes at about 225 MB apiece. Details and the kernel log evidence are in #182.

This sets `CELERY_WORKER_CONCURRENCY: "2"` in the oz and nrc env groups:
- **compose** (`&oz-env`, `&nrc-env`): what `verify-stack` starts inside `dind`.
- **chart** (`envGroups.oz` / `.nrc`): the deployed demo on the same node.

Both images' `/celery_worker.sh` honour the variable; I checked in the running pods. Web, init and beat containers share the anchors and ignore it. `objecten-celery` already defaults to 1.

Closes #182

## Definition of Done

- [x] Linked Gitea issue (above).
- [ ] Failing test committed before the implementation. *(Resource setting; the evidence is the OOM log in #182.)*
- [x] Conventional Commits referencing the issue (`refs #NN`).
- [ ] CI green. This PR's `verify-stack` run is the check.
- [x] `docker compose config` renders the variable into all 7 services on the two anchors.
- [x] Docs: comments next to the setting, following the existing uWSGI notes.

## Notes for reviewers

- `make k8s-lint` and `make k8s-drift` pass.
- **Not applied live.** I couldn't patch the running cluster from my session. After merge, the deploy updates the `oz-env` / `nrc-env` ConfigMaps. The celery pods only pick that up on restart, and the deploy step restarts only this repo's nine services. So run once:
  `kubectl -n big rollout restart deploy/oz-celery deploy/nrc-celery`
- **Why 2 and not 1:** this matches `UWSGI_THREADS: "2"`, and it keeps one notification delivery from blocking behind a slow task. It cuts roughly 40 processes, about 9 GB RSS (less in practice, because forked workers share pages).
- **Longer term:** CI and the demo share one 15 GB VM. Resource requests on the runner, or moving the runner off the node, would stop one from starving the other.

🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #183
This commit was merged in pull request #183.
This commit is contained in:
not
2026-09-25 11:11:45 +00:00
parent 804031eeb8
commit 594fdde227
2 changed files with 7 additions and 0 deletions
+2
View File
@@ -76,6 +76,7 @@ envGroups:
oz:
UWSGI_PROCESSES: "1"
UWSGI_THREADS: "2"
CELERY_WORKER_CONCURRENCY: "2"
DJANGO_SETTINGS_MODULE: openzaak.conf.docker
SECRET_KEY: dev-only-not-for-production
DB_HOST: oz-db
@@ -98,6 +99,7 @@ envGroups:
nrc:
UWSGI_PROCESSES: "1"
UWSGI_THREADS: "2"
CELERY_WORKER_CONCURRENCY: "2"
DJANGO_SETTINGS_MODULE: nrc.conf.docker
SECRET_KEY: dev-only-not-for-production
DB_HOST: nrc-db