feat(workflow): beoordeling escalation to teamlead after 14 days (S-14, closes #15) (#99)

## What & why

S-14: a beoordeling a behandelaar does not pick up within **14 days** escalates to the **teamlead**.

A non-interrupting `P14D` boundary timer on the `Beoordelen` user task fires an external-worker task
(`BeoordelingEscaleren`); the domain's escalation worker reassigns the still-open task's candidate group
from `behandelaar` to `teamlead`. The task keeps its identity — only who may claim it changes. The
escalation-via-external-worker decision is recorded in **ADR-0015** (proposal #98); it upholds §8.2
(the Workflow Client stays the only code that talks to Flowable) and keeps Flowable a stock image.

Closes #15

## Definition of Done

- [x] Linked Gitea issue (above).
- [x] Failing test committed before the implementation.
- [x] Implementation makes the test pass; refactor commit if structure improved.
- [x] Conventional Commits referencing the issue (`refs #NN`).
- [x] CI green — all Gitea Actions jobs.
- [x] `docker compose up` from a fresh clone reaches green health checks within 3 minutes (no new services; escalation is additive to the domain worker).
- [x] Docs updated (ADR-0015, demo note).
- [x] ADR added (`docs/architecture/adr-0015-beoordeling-escalation.md`).
- [x] Demo note in `docs/demo-script.md`.

## How it was built (TDD)

- **Workflow Client** (`IBeoordelingEscalatieClient`): acquire `BeoordelingEscaleren` jobs → find the open `Beoordelen` task in the instance → add `teamlead`/remove `behandelaar` candidate group → complete the job. Red → green.
- **Escalation drain loop** (`BeoordelingEscalatieProcessor`) + hosted `BeoordelingEscalatiePump`, mirroring the OpenZaak worker. Red → green.
- **BPMN**: non-interrupting `P14D` boundary timer on `Beoordelen` → external task → escalation end.
- **Both branches** (escalate after timeout; no-op when completed in time) covered by the `Een beoordeling escaleren` acceptance scenarios + Workflow Client unit tests.
- **Live integration**: `verify-domain` fires the timer early via Flowable's management API and asserts the reassignment to teamlead.

## Notes for reviewers

- Interface segregation: escalation is on `IBeoordelingEscalatieClient`, separate from the OpenZaak worker's `IExternalWorkerClient`.
- Reassignment is two REST hops (add teamlead, remove behandelaar); idempotent on redelivery — see ADR-0015 consequences.
- Local checks green: domain unit tests (104), acceptance (13), `dotnet format --verify-no-changes`, Release build (0 errors), **domain mutation 96.69%** (break 90). The `run-domain-check.sh` escalation path is CI-verified on verify-stack (local full-stack run is constrained here).
- `BeoordelingEscalatiePump` excluded from mutation, mirroring the existing `OpenZaakJobPump` exclusion.

Reviewed-on: #99

services/domain/Big.Infrastructure/BeoordelingEscalatieProcessor.cs

@@ -0,0 +1,37 @@
+using Microsoft.Extensions.Logging;
+
+namespace Big.Infrastructure;
+
+/// <summary>
+/// One poll tick of the beoordeling-escalation worker (S-14, ADR-0015): acquire the parked
+/// <c>BeoordelingEscaleren</c> jobs — the tokens the 14-day boundary timer on <c>Beoordelen</c> spawns
+/// — reassign each instance's still-open <c>Beoordelen</c> task to the teamlead, and complete the job.
+/// A job that fails is logged and left un-completed so Flowable redelivers it (§8.6). Split out from
+/// the hosted pump so the acquire→reassign→complete logic is unit-testable without a running host.
+/// </summary>
+public sealed class BeoordelingEscalatieProcessor(
+    IBeoordelingEscalatieClient client,
+    ILogger<BeoordelingEscalatieProcessor> logger)
+{
+    /// <summary>Acquire and process up to <paramref name="maxJobs"/> escalations. Returns the number acquired.</summary>
+    public async Task<int> PumpOnceAsync(int maxJobs, CancellationToken ct = default)
+    {
+        var jobs = await client.AcquireBeoordelingEscalatieJobsAsync(maxJobs, ct);
+
+        foreach (var job in jobs)
+        {
+            try
+            {
+                await client.ReassignBeoordelingToTeamleadAsync(job.ProcessInstanceId, ct);
+                await client.CompleteBeoordelingEscalatieJobAsync(job.JobId, ct);
+            }
+            catch (Exception ex)
+            {
+                // Leave the job un-completed: its lock expires and Flowable redelivers it (§8.6).
+                logger.LogError(ex, "BeoordelingEscaleren job {JobId} failed; leaving it for redelivery.", job.JobId);
+            }
+        }
+
+        return jobs.Count;
+    }
+}

services/domain/Big.Infrastructure/BeoordelingEscalatiePump.cs

@@ -0,0 +1,49 @@
+using Microsoft.Extensions.DependencyInjection;
+using Microsoft.Extensions.Hosting;
+using Microsoft.Extensions.Logging;
+
+namespace Big.Infrastructure;
+
+/// <summary>
+/// The hosted polling loop of the beoordeling-escalation worker (S-14, ADR-0015): on an interval it
+/// resolves a scoped <see cref="BeoordelingEscalatieProcessor"/> and asks it to drain the parked
+/// <c>BeoordelingEscaleren</c> jobs. A deliberately thin shell — all acquire/reassign/complete logic
+/// lives in the processor, which is unit-tested; this class only owns the timer, the per-tick scope,
+/// and loop resilience. Structurally identical to <see cref="OpenZaakJobPump"/>.
+/// </summary>
+public sealed class BeoordelingEscalatiePump(
+    IServiceScopeFactory scopeFactory,
+    FlowableOptions options,
+    ILogger<BeoordelingEscalatiePump> logger) : BackgroundService
+{
+    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
+    {
+        while (!stoppingToken.IsCancellationRequested)
+        {
+            try
+            {
+                using var scope = scopeFactory.CreateScope();
+                var processor = scope.ServiceProvider.GetRequiredService<BeoordelingEscalatieProcessor>();
+                await processor.PumpOnceAsync(options.MaxJobsPerPoll, stoppingToken);
+            }
+            catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
+            {
+                break;
+            }
+            catch (Exception ex)
+            {
+                // A transient fault (e.g. Flowable briefly unreachable) must not kill the loop.
+                logger.LogError(ex, "BeoordelingEscaleren job poll failed; retrying after the poll interval.");
+            }
+
+            try
+            {
+                await Task.Delay(options.PollInterval, stoppingToken);
+            }
+            catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
+            {
+                break;
+            }
+        }
+    }
+}

services/domain/Big.Infrastructure/FlowableWorkflowClient.cs

@@ -15,11 +15,14 @@ namespace Big.Infrastructure;
 /// The REST contract here is the one verified against a live flowable-rest engine (ADR-0009).
 /// </summary>
 public sealed class FlowableWorkflowClient(HttpClient http, FlowableOptions options)
-    : IWorkflowClient, IExternalWorkerClient, IUserTaskClient
+    : IWorkflowClient, IExternalWorkerClient, IUserTaskClient, IBeoordelingEscalatieClient
 {
     private const string Topic = "OpenZaakAanmaken";
+    private const string EscalatieTopic = "BeoordelingEscaleren";
     private const string ProcessDefinitionKey = "registratie";
     private const string BeoordelenTaskKey = "Beoordelen";
+    private const string BehandelaarGroup = "behandelaar";
+    private const string TeamleadGroup = "teamlead";
     private const string RegistrationIdVariable = "registrationId";
     private const string ZaakUrlVariable = "zaakUrl";
     private const string BesluitVariable = "besluit";
@@ -106,6 +109,47 @@ public sealed class FlowableWorkflowClient(HttpClient http, FlowableOptions opti
         response.EnsureSuccessStatusCode();
     }
 
+    public async Task<IReadOnlyList<EscalatieJob>> AcquireBeoordelingEscalatieJobsAsync(int maxJobs, CancellationToken ct = default)
+    {
+        var request = new AcquireJobsRequest(EscalatieTopic, options.LockDuration, maxJobs, options.WorkerId);
+
+        var jobs = await PostAsync<AcquireJobsRequest, List<AcquiredEscalatieJob>>(
+            "external-job-api/acquire/jobs", request, ct) ?? [];
+
+        return [.. jobs.Select(job => new EscalatieJob(job.Id, job.ProcessInstanceId))];
+    }
+
+    public async Task ReassignBeoordelingToTeamleadAsync(string processInstanceId, CancellationToken ct = default)
+    {
+        // The escalation token runs in parallel to the still-open Beoordelen task (non-interrupting
+        // boundary timer); find that task in this instance so we can move it to the teamlead. If the
+        // behandelaar completed it just before the timer fired there is nothing to reassign — a
+        // best-effort no-op (the timer/completion race, cf. §8.6).
+        var query = new TaskByInstanceQueryRequest(processInstanceId, BeoordelenTaskKey);
+        var page = await PostAsync<TaskByInstanceQueryRequest, TaskQueryResult>(
+            "service/query/tasks", query, ct);
+
+        var task = page?.Data?.FirstOrDefault();
+        if (task is null)
+            return;
+
+        // Add teamlead, then drop behandelaar: the task now belongs to the teamlead group.
+        using (var added = await SendAsync(
+            $"service/runtime/tasks/{task.Id}/identitylinks",
+            new IdentityLinkRequest(TeamleadGroup, "candidate"), ct))
+            added.EnsureSuccessStatusCode();
+
+        await DeleteAsync(
+            $"service/runtime/tasks/{task.Id}/identitylinks/groups/{BehandelaarGroup}/candidate", ct);
+    }
+
+    public async Task CompleteBeoordelingEscalatieJobAsync(string jobId, CancellationToken ct = default)
+    {
+        using var response = await SendAsync(
+            $"external-job-api/acquire/jobs/{jobId}/complete", new CompleteJobRequest(options.WorkerId, []), ct);
+        response.EnsureSuccessStatusCode();
+    }
+
     private async Task<TResponse?> GetAsync<TResponse>(string path, CancellationToken ct)
     {
         var message = new HttpRequestMessage(HttpMethod.Get, new Uri(options.BaseUrl, path));
@@ -115,6 +159,14 @@ public sealed class FlowableWorkflowClient(HttpClient http, FlowableOptions opti
         return await response.Content.ReadFromJsonAsync<TResponse>(ct);
     }
 
+    private async Task DeleteAsync(string path, CancellationToken ct)
+    {
+        var message = new HttpRequestMessage(HttpMethod.Delete, new Uri(options.BaseUrl, path));
+        message.Headers.Authorization = new AuthenticationHeaderValue("Basic", BasicCredentials());
+        using var response = await http.SendAsync(message, ct);
+        response.EnsureSuccessStatusCode();
+    }
+
     private async Task<TResponse?> PostAsync<TRequest, TResponse>(string path, TRequest body, CancellationToken ct)
     {
         using var response = await SendAsync(path, body, ct);
@@ -154,6 +206,14 @@ public sealed class FlowableWorkflowClient(HttpClient http, FlowableOptions opti
         [property: JsonPropertyName("taskDefinitionKey")] string TaskDefinitionKey,
         [property: JsonPropertyName("includeProcessVariables")] bool IncludeProcessVariables);
 
+    private sealed record TaskByInstanceQueryRequest(
+        [property: JsonPropertyName("processInstanceId")] string ProcessInstanceId,
+        [property: JsonPropertyName("taskDefinitionKey")] string TaskDefinitionKey);
+
+    private sealed record IdentityLinkRequest(
+        [property: JsonPropertyName("group")] string Group,
+        [property: JsonPropertyName("type")] string Type);
+
     private sealed record ClaimTaskRequest(
         [property: JsonPropertyName("action")] string Action,
         [property: JsonPropertyName("assignee")] string Assignee);
@@ -193,6 +253,10 @@ public sealed class FlowableWorkflowClient(HttpClient http, FlowableOptions opti
 
     private sealed record ExecutionDto([property: JsonPropertyName("id")] string Id);
 
+    private sealed record AcquiredEscalatieJob(
+        [property: JsonPropertyName("id")] string Id,
+        [property: JsonPropertyName("processInstanceId")] string ProcessInstanceId);
+
     private sealed record AcquiredJob(
         [property: JsonPropertyName("id")] string Id,
         [property: JsonPropertyName("variables")] IReadOnlyList<Variable> Variables)

services/domain/Big.Infrastructure/IExternalWorkerClient.cs

@@ -16,3 +16,23 @@ public interface IExternalWorkerClient
     /// <summary>Complete an acquired job, passing the opened zaak URL back into the process.</summary>
     Task CompleteOpenZaakJobAsync(string jobId, Uri zaakUrl, CancellationToken ct = default);
 }
+
+/// <summary>
+/// The escalation side of the Workflow Client (S-14): the <c>BeoordelingEscaleren</c> external-worker
+/// jobs parked by the 14-day boundary timer on <c>Beoordelen</c>, and the reassignment they drive.
+/// Kept separate from <see cref="IExternalWorkerClient"/> (interface segregation) so the OpenZaak
+/// worker never sees escalation. Implemented by <see cref="FlowableWorkflowClient"/> — the only code
+/// that talks to Flowable (§8.2, ADR-0015).
+/// </summary>
+public interface IBeoordelingEscalatieClient
+{
+    /// <summary>Acquire and lock up to <paramref name="maxJobs"/> <c>BeoordelingEscaleren</c> jobs.</summary>
+    Task<IReadOnlyList<EscalatieJob>> AcquireBeoordelingEscalatieJobsAsync(int maxJobs, CancellationToken ct = default);
+
+    /// <summary>Reassign the still-open <c>Beoordelen</c> task in the given process instance from the
+    /// behandelaar group to teamlead. Best-effort no-op if the task is no longer open.</summary>
+    Task ReassignBeoordelingToTeamleadAsync(string processInstanceId, CancellationToken ct = default);
+
+    /// <summary>Complete an acquired escalation job so its token reaches the escalation end event.</summary>
+    Task CompleteBeoordelingEscalatieJobAsync(string jobId, CancellationToken ct = default);
+}