From 594fdde2277afb6e040b4710ed2a53cf3c61d774 Mon Sep 17 00:00:00 2001
From: Niek Otten <not@respellion.nl>
Date: Fri, 25 Sep 2026 11:11:45 +0000
Subject: [PATCH] fix(infra): cap celery workers at 2 so the shared node stops
 OOM-killing CI (closes #182) (#183)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

## What & why

`verify-stack` is being killed by the OOM controller on the shared Talos node, on main (run 827) and on #180 (run 830). The cause is Celery: with `CELERY_WORKER_CONCURRENCY` unset, `oz-celery` and `nrc-celery` each fork one worker per CPU. That's 22 each on the lab node, 49 Celery processes at about 225 MB apiece. Details and the kernel log evidence are in #182.

This sets `CELERY_WORKER_CONCURRENCY: "2"` in the oz and nrc env groups:
- **compose** (`&oz-env`, `&nrc-env`): what `verify-stack` starts inside `dind`.
- **chart** (`envGroups.oz` / `.nrc`): the deployed demo on the same node.

Both images' `/celery_worker.sh` honour the variable; I checked in the running pods. Web, init and beat containers share the anchors and ignore it. `objecten-celery` already defaults to 1.

Closes #182

## Definition of Done

- [x] Linked Gitea issue (above).
- [ ] Failing test committed before the implementation. *(Resource setting; the evidence is the OOM log in #182.)*
- [x] Conventional Commits referencing the issue (`refs #NN`).
- [ ] CI green. This PR's `verify-stack` run is the check.
- [x] `docker compose config` renders the variable into all 7 services on the two anchors.
- [x] Docs: comments next to the setting, following the existing uWSGI notes.

## Notes for reviewers

- `make k8s-lint` and `make k8s-drift` pass.
- **Not applied live.** I couldn't patch the running cluster from my session. After merge, the deploy updates the `oz-env` / `nrc-env` ConfigMaps. The celery pods only pick that up on restart, and the deploy step restarts only this repo's nine services. So run once:
  `kubectl -n big rollout restart deploy/oz-celery deploy/nrc-celery`
- **Why 2 and not 1:** this matches `UWSGI_THREADS: "2"`, and it keeps one notification delivery from blocking behind a slow task. It cuts roughly 40 processes, about 9 GB RSS (less in practice, because forked workers share pages).
- **Longer term:** CI and the demo share one 15 GB VM. Resource requests on the runner, or moving the runner off the node, would stop one from starving the other.

🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: https://git.labs.respellion.tech/eho/register-referentie/pulls/183
---
 infra/docker-compose.yml             | 5 +++++
 infra/helm/big-reference/values.yaml | 2 ++
 2 files changed, 7 insertions(+)

diff --git a/infra/docker-compose.yml b/infra/docker-compose.yml
index 3a5a6da..6ac6fae 100644
--- a/infra/docker-compose.yml
+++ b/infra/docker-compose.yml
@@ -57,6 +57,9 @@ services:
       # share this anchor and ignore it — they don't run uwsgi.
       UWSGI_PROCESSES: "1"
       UWSGI_THREADS: "2"
+      # Same lever for oz-celery: unset, the worker forks one process per CPU (22 on the lab node,
+      # ~225 MB each), which OOM-killed the shared runner mid-verify-stack. Only celery reads it.
+      CELERY_WORKER_CONCURRENCY: "2"
       DJANGO_SETTINGS_MODULE: openzaak.conf.docker
       SECRET_KEY: ${OZ_SECRET_KEY:-dev-only-not-for-production}
       DB_HOST: oz-db
@@ -144,6 +147,8 @@ services:
       # 1 uWSGI worker, not the image default of 4×4 (#147) — see the oz-env note above.
       UWSGI_PROCESSES: "1"
       UWSGI_THREADS: "2"
+      # Two celery workers, not one per CPU — see the oz-env note above.
+      CELERY_WORKER_CONCURRENCY: "2"
       DJANGO_SETTINGS_MODULE: nrc.conf.docker
       SECRET_KEY: ${NRC_SECRET_KEY:-dev-only-not-for-production}
       DB_HOST: nrc-db
diff --git a/infra/helm/big-reference/values.yaml b/infra/helm/big-reference/values.yaml
index 20e7361..31f4df4 100644
--- a/infra/helm/big-reference/values.yaml
+++ b/infra/helm/big-reference/values.yaml
@@ -76,6 +76,7 @@ envGroups:
   oz:
     UWSGI_PROCESSES: "1"
     UWSGI_THREADS: "2"
+    CELERY_WORKER_CONCURRENCY: "2"
     DJANGO_SETTINGS_MODULE: openzaak.conf.docker
     SECRET_KEY: dev-only-not-for-production
     DB_HOST: oz-db
@@ -98,6 +99,7 @@ envGroups:
   nrc:
     UWSGI_PROCESSES: "1"
     UWSGI_THREADS: "2"
+    CELERY_WORKER_CONCURRENCY: "2"
     DJANGO_SETTINGS_MODULE: nrc.conf.docker
     SECRET_KEY: dev-only-not-for-production
     DB_HOST: nrc-db