learning-platform/AI_PIPELINE_HARDENING_PLAN.md

AI Pipeline Hardening — Implementation Plan

Audience: an AI agent executing this plan against the Respellion Learning Platform. Owner before this work: Raymond Verhoef (rve@respellion.nl). Source of truth for repo conventions: AI_AGENT.md. Read it before starting.

This plan upgrades the platform's interaction with the Anthropic API: how prompts are built, how responses are parsed, how the model is retried, and how outputs are validated. It is broken into six phases that can be implemented and shipped independently. Each phase ends with verifiable acceptance criteria.

---

0. Operating principles

These rules govern every phase. Re-read them before you commit.

1. PocketBase is the source of truth. No persistent state in localStorage (see AI_AGENT.md §2). The Anthropic API is proxied via Caddy; never add x-api-key headers in frontend code.
2. No behaviour regressions. Existing UI flows (extraction, weekly learning, weekly quiz, R42 chat, handbook sync, analyze-graph) must keep working after every phase. Phases are additive.
3. Schema-first. Where the model produces structured output, define a JSON Schema (Zod) and validate every response. Reject (don't paper over) malformed output.
4. Single LLM entry point. After Phase 1 there is exactly one module that talks to /api/anthropic/v1/messages. All callers go through it.
5. No silent truncation. If stop_reason === 'max_tokens' on a structured-output call, throw — never persist a partial parse.
6. Cache what is stable, vary what is dynamic. Use prompt caching for system prompts and KB context. User messages are never cached.
7. Comments and docs: follow the repo's terse style. Don't add explanatory comments unless the why is non-obvious.
8. Migrations: when changing a PocketBase schema, add a migration in pb_migrations/ following the existing timestamp prefix convention. Never edit shipped migrations.
9. Stop and ask if you encounter a decision the plan doesn't cover (e.g. a model deprecation, a missing collection, a failing test that looks pre-existing).

Files you will touch (or create) across all phases

Path	Purpose
src/lib/llm.js (new, Phase 1)	Single Anthropic client wrapper
src/lib/llmSchemas.js (new, Phase 1)	Zod schemas for every structured task
src/lib/llmRetry.js (new, Phase 1)	Retry + backoff + abort policy
src/lib/random.js (new, Phase 4)	Fisher–Yates shuffle + RNG helpers
src/lib/api.js	Becomes a thin re-export from llm.js (back-compat)
src/lib/extractionPipeline.js	Migrated to llm.js + tool use + overlap chunking
src/lib/learningService.js	Migrated to llm.js + tool use + patch-refine
src/lib/testService.js	Migrated to llm.js + tool use + dedup + shuffle fix
src/components/admin/KnowledgeGraph.jsx	analyzeGraph → tool use + dry-run preview
src/components/chat/rag.js	Retrieval (TF-IDF) + lookup_topic tool
src/components/chat/prompts.js	Split system prompt into cacheable + dynamic
src/components/chat/useChat.js	Wire retrieval + truncation
pb_migrations/*	Schema additions for llm_calls, question difficulty, topic relevance_locked
evals/ (new, Phase 6)	Golden-set eval harness

---

Phase 1 — Foundation (single LLM client + robust parsing)

Goal: every LLM call goes through one module that handles retry, timeout, abort, JSON extraction, and schema validation. No behaviour change visible to the user.

1.1 Create src/lib/llmRetry.js

Implements the retry policy used by llm.js.

Behaviour:

• Exponential backoff with full jitter, base 1000ms, cap 16000ms.
• Retries only on these HTTP statuses: 408, 425, 429, 500, 502, 503, 504, 529.
• Honours Retry-After header (seconds or HTTP date). If present and ≤ 60s, use it; if > 60s, fail fast.
• Default maxRetries = 4.
• Does not retry on AbortError.

Exported interface:

// withRetry: (fn: (attempt:number) => Promise<T>, opts?) => Promise<T>
// RetryableError(status, retryAfterMs)
export async function withRetry(fn, { maxRetries = 4, signal } = {}) { ... }
export class RetryableError extends Error { constructor(status, retryAfterMs) { ... } }

1.2 Create src/lib/llmSchemas.js

One Zod schema per structured task. Install Zod (npm i zod).

Required schemas (names + shape match what callers already produce — do not change field names):

• extractionResultSchema — { topics: Topic[], relations: Relation[] } matching the existing SYSTEM_PROMPT in extractionPipeline.js.
• handbookResultSchema — same shape, but relation.type enum unified to related_to | depends_on | part_of | executed_by (see Phase 3 task 3.5 — for now the schema accepts both executes and executed_by, normalize executes → executed_by post-validation).
• learningArticleSchema, learningSlidesSchema, learningInfographicSchema, learningAllSchema matching learningService.js.
• quizQuestionsSchema — { questions: Question[] } with options.length === 4 and correctIndex ∈ [0,3].
• customTopicSchema — { label, type: 'concept'|'role'|'process', description }.
• graphActionsSchema — { merges, deletions, newRelations, relevanceUpdates } matching KnowledgeGraph.jsx:329.
• proposeGraphDeltaSchema — matches PROPOSE_GRAPH_DELTA_TOOL.input_schema in prompts.js.

Acceptance: every schema has at least one happy-path Vitest test in src/lib/__tests__/llmSchemas.test.js (add vitest if not present).

1.3 Create src/lib/llm.js

The single Anthropic client. All other modules must call only this one.

Public interface:

// Task tier — used to pick a model from settings.
// 'fast'      → admin:model:fast      (default: claude-haiku-4-5-20251001)
// 'standard'  → admin:model:standard  (default: claude-sonnet-4-6)
// 'reasoning' → admin:model:reasoning (default: claude-opus-4-7)
export async function callLLM({
  task,            // string, e.g. 'extract.source' — used for logging only
  tier = 'standard',
  system,          // string OR Array<{ type:'text', text:string, cache_control?:{type:'ephemeral'} }>
  messages,        // [{ role, content }] OR omitted (use `user`)
  user,            // shorthand for [{role:'user', content: user}]
  tools,           // optional Anthropic tool definitions
  toolChoice,      // optional, e.g. { type:'tool', name:'emit_knowledge_graph' }
  schema,          // optional Zod schema for text→JSON path (used only when no tool)
  maxTokens = 4096,
  temperature = 0,
  signal,          // AbortSignal
}): Promise<{
  text: string,
  toolUses: Array<{ name, input }>,
  stopReason: string,
  usage: { input_tokens, output_tokens, cache_creation_input_tokens, cache_read_input_tokens },
  requestId: string | null,
  model: string,
  durationMs: number,
}>

Key requirements:

1. Simulation mode preserved: if storage.get('admin:use_simulation') === true, return a deterministic stub (use existing simulateResponse payload for backward compatibility — branch on task prefix to return a matching stub).
2. Fetch with AbortController; default 60-second timeout if caller didn't pass a signal.
3. Retry through withRetry (Phase 1.1).
4. Auth-portal detection preserved: if response is not application/json, throw Your session has expired. Please refresh the page and log in again. exactly as today.
5. No truncation acceptance: if stop_reason === 'max_tokens' AND caller passed schema OR toolChoice requested a tool, throw LLMTruncatedError.
6. Robust JSON extraction when caller passed schema (and no tool was used): use parseStructuredText(text) that
   • strips ```json and ``` fences,
   • finds the outermost balanced JSON value (object or array) via a tiny brace-matching scan, not regex,
   • throws LLMOutputError if no balanced JSON found,
   • runs Zod schema.parse on the result.
7. Tool path: when tools is provided and the model emits tool_use, return them under toolUses. Validate each tool's input against the corresponding Zod schema if the caller wired one in (via toolSchemas: { [toolName]: ZodSchema }).
8. Logging: after every call, append a row to a new PocketBase collection llm_calls (best-effort — never block on this; catch and console.debug failures). Fields: task, model, tier, duration_ms, input_tokens, output_tokens, cache_read_tokens, cache_create_tokens, stop_reason, ok, error_msg. See Phase 5 task 5.6 for the migration.
9. Custom errors: LLMHttpError, LLMTruncatedError, LLMOutputError, LLMValidationError. All extend Error and set name for instanceof checks.

1.4 Make src/lib/api.js a thin shim

Replace the existing anthropicApi.generateContent and anthropicApi.chat implementations with calls into llm.js. Preserve the exact exported names and return shapes so no caller breaks.

// api.js after Phase 1
export const anthropicApi = {
  async generateContent(systemPrompt, userMessage, maxRetries = 1) {
    const { text } = await callLLM({
      task: 'legacy.generateContent',
      tier: 'standard',
      system: systemPrompt,
      user: userMessage,
      maxTokens: 8192,
      temperature: 0,
    });
    return text;
  },
  async chat(systemPrompt, messages, opts = {}) {
    const r = await callLLM({
      task: 'legacy.chat',
      tier: 'standard',
      system: systemPrompt,
      messages,
      tools: opts.tools,
      maxTokens: 1024,
      temperature: 0.3,   // chat default — see Phase 5
    });
    return {
      content: [
        ...(r.text ? [{ type:'text', text: r.text }] : []),
        ...r.toolUses.map(tu => ({ type:'tool_use', name: tu.name, input: tu.input })),
      ],
      stop_reason: r.stopReason,
    };
  },
};

1.5 Update default model + tiered settings

• Replace DEFAULT_MODEL = 'claude-sonnet-4-20250514' with the three tier defaults above.
• In Admin → Settings, add three model selects (fast, standard, reasoning). Read existing admin:model as a legacy fallback for standard (so existing users don't lose their override).

Phase 1 acceptance criteria

• npm run lint passes; npm run test passes (Vitest).
• Every existing user flow (extraction, weekly content, weekly quiz, R42, handbook sync, analyze graph) still works against the live API.
• grep -r "fetch.*anthropic" src/ returns only src/lib/llm.js.
• Simulation mode toggle still returns stubbed responses for all flows.
• Manually verify: kill the network mid-call → request aborts within 60s and surfaces a clear error message.
• Manually verify: rate-limit the proxy (429 + Retry-After: 5) → call retries once after ~5s and then succeeds.

---

Phase 2 — Prompt caching & tool-based structured outputs

Goal: structured-output tasks no longer parse JSON out of prose. Large stable prompts are cached.

2.1 Migrate extraction to tool use

In extractionPipeline.js:

• Replace the "Return JSON only" instruction with a tool: emit_knowledge_graph whose input_schema mirrors extractionResultSchema.
• Replace anthropicApi.generateContent(...) with callLLM({ ..., tools:[emitKnowledgeGraphTool], toolChoice:{ type:'tool', name:'emit_knowledge_graph' } }).
• Read the validated object from toolUses[0].input.
• Same migration for analyzeHandbookDelta (tool emit_handbook_delta).
• Delete every responseText.match(/\{[\s\S]*\}/) site.

2.2 Migrate learning, quiz, custom-topic, graph-actions to tool use

Same pattern, in:

• learningService.js: tools emit_learning_article, emit_learning_slides, emit_learning_infographic, emit_learning_all, emit_custom_topic.
• testService.js: tool emit_quiz_questions.
• KnowledgeGraph.jsx:297: tool emit_graph_actions.

2.3 Prompt caching

Pass system as an array of blocks so the stable parts can be cached:

system: [
  { type:'text', text: STABLE_SYSTEM_HEADER, cache_control: { type:'ephemeral' } },
  { type:'text', text: dynamicPart }, // not cached
],

Apply caching to:

• Extraction SYSTEM_PROMPT and HANDBOOK_SYSTEM_PROMPT (both fully stable → cache the whole block).
• R42 system prompt — split into three blocks: stable preamble (cached), KB context (cached only while the graph hasn't changed; bust by appending a short hash of the topic IDs+labels — Phase 5 details), and per-turn role line (not cached).

2.4 Patch-based learning refinement

Refactor refineLearningContent (learningService.js:147) from "return the full updated JSON" to patch operations via tools:

• set_section(heading: string, body: string) — replace one section by heading match.
• add_section(heading: string, body: string, position: 'start'|'end').
• remove_section(heading: string).
• replace_takeaways(items: string[]).
• set_intro(intro: string).

Apply patches client-side to the cached object. Re-validate against learningArticleSchema after patching; reject the whole turn if invalid.

Phase 2 acceptance criteria

• No regex JSON extraction left in src/: grep -rn "match(/\\\\{\\[\\\\s\\\\S\\]\\*\\\\}/)" src/ returns nothing.
• Token usage telemetry shows cache_read_input_tokens > 0 on the second extraction call within 5 minutes (cache hit).
• Re-running extraction on a known source produces the same topic count ±10% as before this phase.
• refineLearningContent round-trip ("make the intro shorter") produces only the changed section in the diff against the prior cached content.

---

Phase 3 — Extraction quality

Goal: fewer near-duplicate topics, no silent truncation, adaptive throttling, unified vocabulary.

3.1 Sentence-aware chunking with overlap

Replace chunkText in extractionPipeline.js:87:

• Target ~2000 input tokens per chunk. Approximate as chars / 4. Configurable via MAX_CHUNK_CHARS = 8000.
• 200-token overlap between chunks (OVERLAP_CHARS = 800).
• Split on sentence boundaries (/(?<=[.!?])\s+/) first; fall back to paragraph boundary if a sentence is too long; never produce a chunk larger than MAX_CHUNK_CHARS.
• Add a guard: if a single sentence exceeds MAX_CHUNK_CHARS, hard-split at character boundary and log a warning.

3.2 Stateful extraction

Before each chunk after the first, prepend to the user message:

Already-extracted topic IDs (do NOT create new IDs for these — reuse them if the same concept appears here):
- software-engineer
- onboarding-buddy
...

Cap the list at 200 IDs by recency to keep token cost bounded. The model will then reuse IDs instead of inventing variants like software-developer.

3.3 Adaptive throttling

Replace the hard setTimeout(r, 12000) in extractionPipeline.js:127 and the 15s sleep in KnowledgeGraph.jsx:274 with a shared token-bucket limiter in src/lib/llmRetry.js:

export const extractionLimiter = createLimiter({ rps: 5/60, burst: 1 }); // 5 req/min
// usage: await extractionLimiter.acquire();

callLLM accepts an optional limiter param that it awaits before fetch. On 429 with Retry-After, the limiter is paused for that duration.

3.4 Preserve admin-edited relevance

Add a migration introducing relevance_locked: bool on topics. Set it to true whenever an admin edits learning_relevance via the UI (KnowledgeGraph.jsx edit handler — locate by searching for setLearningRelevance or the relevance form field).

In mergeKnowledgeGraph (extractionPipeline.js:167), when relevance_locked, never overwrite learning_relevance.

3.5 Unify relation vocabulary

Pick one set: related_to | depends_on | part_of | executed_by. Migrate:

• HANDBOOK_SYSTEM_PROMPT (extractionPipeline.js:42) — change executes to executed_by and swap the source/target in the prompt example.
• Write a one-shot migration script pb_migrations/<timestamp>_normalize_relation_types.js that rewrites any existing executes relation to executed_by and swaps source ↔ target.
• Verify R42's validateDelta (rag.js:108) already enforces this set (it does) — no change needed there.

3.6 Cancellation

Add a "Cancel" button to the source-processing UI in ContentManager.jsx / UploadZone.jsx (locate the one that displays extraction progress). Wire it to abort the in-flight callLLM via the signal it receives. On cancel, set source status to cancelled (add to status enum migration).

Phase 3 acceptance criteria

• Running extraction twice on the same sources/ROLES.md produces zero new topics on the second run (idempotency through reused IDs).
• Locked-relevance topics survive re-extraction.
• No fixed setTimeout ≥ 5s anywhere in src/ (grep -rn "setTimeout" src/).
• Cancelling an extraction mid-run leaves the source in cancelled state, not processing.
• pb_migrations includes the relation-vocabulary normalization and the relevance_locked column.

---

Phase 4 — Quiz & content quality

Goal: quiz questions are positionally unbiased, deduped, and difficulty-tagged. Random helpers are correct.

4.1 Random helpers

Create src/lib/random.js:

export function shuffle(arr) { /* Fisher–Yates, returns NEW array */ }
export function sample(arr, n) { /* unbiased sample without replacement */ }
export function pickInt(min, maxInclusive) { /* uniform integer */ }

Replace every .sort(() => 0.5 - Math.random()) with shuffle(arr):

• testService.js:122 and testService.js:163.
• Any other site found by grep -rn "0.5 - Math.random()" src/.

4.2 Debias correctIndex in quiz prompt

In testService.js:81:

• Change the example in the prompt to use "correctIndex": 2 (not 0).
• Add to the prompt: "Distribute correctIndex roughly evenly across 0, 1, 2, and 3. Do not place the correct answer at the same position more than 4 out of 10 times."
• After parsing, run a check: if more than 50% of the batch share the same correctIndex, log a warning and re-roll up to 2 times.

4.3 Difficulty field

• Add to quizQuestionsSchema: difficulty: 'easy'|'medium'|'hard'.
• Update the prompt to require difficulty on every question (current prompt says "4 easy, 4 medium, 2 hard" but never tagged — now tag it).
• Migration: add difficulty to the quiz_banks.questions[] element. PocketBase stores questions as JSON, so the migration is a no-op at the column level; older records get difficulty: 'medium' on read (add a normalizer in db.js).

4.4 Question dedup

In forceGenerateTopicQuestions (testService.js:65):

• Normalize question text (lowercase, strip punctuation, collapse whitespace) → normKey.
• Before persisting, drop any new question whose normKey matches an existing bank question.
• Log dropped duplicates with console.debug('[quiz] dropped duplicate:', text).

4.5 Quality gate

In the same function, after schema validation:

• Reject the whole batch if any question has fewer than 4 distinct options.
• Reject if any option contains "all of the above", "none of the above", "both A and B" (case-insensitive).
• Reject if explanation.trim().length < 20.
• Surface the rejection to the admin UI with a "Retry" button.

4.6 Custom topic ID hygiene

In generateCustomTopic (learningService.js:177):

• Generate kebab-case ID from the polished label, not Date.now().
• Collision check against existing topics (append -2, -3, … if needed).
• Default learning_relevance: 'standard' when the model omits it.

Phase 4 acceptance criteria

• No .sort(() => 0.5 - Math.random()) anywhere in src/.
• Sample of 50 fresh quiz questions across 5 topics: no position holds >40% of correct answers.
• Re-running quiz generation for the same topic does not grow the bank with semantic duplicates.
• Custom topics created via R42 use kebab-case IDs and pass schema validation.

---

Phase 5 — R42 retrieval & telemetry

Goal: R42 stops shipping the entire KG every turn; conversations are bounded; every call is logged.

5.1 TF-IDF retrieval in the browser

Create src/lib/retrieval.js:

export function buildIndex(topics) { /* TF-IDF over label + description */ }
export function retrieveTopK(index, query, k = 10) { /* returns Topic[] */ }

Implementation: a small dependency-free TF-IDF — tokenize on /[a-zA-Z0-9-]+/, lowercase, drop stopwords (Dutch + English short list). Cache the index on the topics array reference. About 100 lines.

5.2 Rewrite buildKbContext

In rag.js:11:

• Use retrieveTopK(index, userMessage, 10) to pick which topics go into the system prompt.
• Always include any topic whose ID or label is mentioned verbatim (existing behaviour).
• Drop the full "every topic" dump.
• Return { context, retrievedTopics, allTopics } — validateDelta continues to use allTopics.

5.3 lookup_topic tool

Add a second R42 tool in prompts.js:

export const LOOKUP_TOPIC_TOOL = {
  name: 'lookup_topic',
  description: 'Fetch the full description and any deeper learning content for a topic. Use when the retrieved context does not contain enough to answer.',
  input_schema: { type:'object', properties: { id:{type:'string'} }, required:['id'] },
};

In useChat.js, when the model emits a lookup_topic, fetch via db.getTopics() + db.getContent(id) and append a tool_result block, then call callLLM again with the extended messages. Cap to 3 lookup hops per turn to avoid loops.

5.4 R42 prompt cache busting

KB-context block is cached as ephemeral. The cache key is automatic per text, so any change busts it. Append a stable suffix to the KB block: "\n[kb_hash: <first-8-chars-of-sha256-of-sorted-topic-ids>]". This guarantees the block content changes the moment a topic is added/removed, even if the included topic list looks similar.

5.5 Conversation truncation

In useChat.js:

• Keep last 12 turns verbatim in apiMessages.
• Older messages: if more than 12 turns exist, summarize the older ones with a cheap (tier: 'fast') call and prepend a single { role:'system'-equivalent inside the user history is not allowed by Anthropic — instead prepend a user/assistant pair } block, OR simply drop older turns. Default: drop older turns and prepend a one-line assistant message saying "(earlier conversation truncated)". Summarization is an optional follow-up.

5.6 llm_calls collection

Add a migration pb_migrations/<timestamp>_created_llm_calls.js for collection llm_calls:

Field	Type	Notes
task	text	e.g. extract.source
model	text	resolved model ID
tier	text	fast / standard / reasoning
duration_ms	number
input_tokens	number
output_tokens	number
cache_read_tokens	number
cache_create_tokens	number
stop_reason	text
ok	bool
error_msg	text	nullable

Wire callLLM to write to it (best-effort, never throws). Add a minimal Admin → Diagnostics view that shows the last 100 calls and aggregate cost (using public Anthropic prices in a constant; refresh manually).

5.7 R42 defaults

• temperature: 0.3 for R42 chat in callLLM.
• maxTokens: 2048 for R42 (text + tool budget).

Phase 5 acceptance criteria

• R42 system prompt is ≤ 4000 tokens regardless of KG size (verify on a graph with 200+ topics).
• Adding a topic in the admin UI causes the next R42 call to show cache_read_tokens === 0 for the KB block, then subsequent calls to show non-zero.
• R42 successfully answers a question about a topic whose ID was not in the user's message by emitting lookup_topic (manual verification).
• Admin → Diagnostics shows recent LLM calls with model, tokens, duration.

---

Phase 6 — Eval harness (optional, high-leverage)

Goal: prompt or model changes can be measured before they ship.

6.1 Golden sets

Create evals/ at the repo root:

evals/
  extraction/
    cases/
      roles-handbook.txt
      governance-handbook.txt
    expected/
      roles-handbook.json       # { mustContain: ['software-engineer', ...], minTopics: 20 }
      governance-handbook.json
  quiz/
    cases/
      software-engineer.json    # topic to generate quiz for
    rubric.md                   # human-readable quality rubric
  chat/
    scripts/
      ask-about-known-topic.json
      ask-about-unknown-topic.json
      propose-new-role.json

10 extraction cases, 5 quiz topics, 10 chat scripts is enough to start.

6.2 Runner

evals/run.mjs — Node script that:

1. Loads each case.
2. Invokes the same code path the app uses (import from src/lib/* directly; reuse the simulation toggle off).
3. Compares against expectations:
   • Extraction: mustContain IDs present? minTopics met? No stop_reason: 'max_tokens'?
   • Quiz: distribution of correctIndex, mean explanation length, banned phrases.
   • Chat: for each script, did the expected tool fire? Did the reply contain expected anchors?
4. Writes evals/results/<ISO-timestamp>.json and a Markdown diff against the previous baseline.

Add npm run eval to package.json.

6.3 Prompt versioning

In each prompt module, export PROMPT_VERSION = '2026-05-20-001'. Persist it on the artifact (content.data.prompt_version, quiz_banks.questions[i].prompt_version, topics.metadata.prompt_version). Add an admin button "Mark stale content for regeneration" that lists artifacts whose version is older than the current.

Phase 6 acceptance criteria

• npm run eval runs end-to-end against the live API and produces a result file.
• CI (or a manual check) runs evals on every change to src/lib/llm.js, prompts, or schemas.
• Each AI-generated artifact in PocketBase carries a prompt_version and is filterable by it.

---

Cross-phase verification checklist

After every phase, run this short checklist before merging:

1. Build: npm run build succeeds.
2. Lint: npm run lint clean.
3. Tests: npm run test green.
4. Smoke flows in dev (simulation off, real API key):
   • Add a source via Admin → Sources, extract, verify topics appear.
   • Visit Leren for the current week, generate article. Then slides.
   • Visit Testen, generate weekly quiz. Submit. Score lands on leaderboard.
   • Open R42, ask a known and an unknown question; propose a new topic; admin accepts it.
   • Run Admin → Knowledge Graph → Analyze & Optimize Graph.
   • Run Admin → Knowledge Graph → Sync Handbook (small repo or mock).
5. Simulation toggle: flip simulation mode on and confirm no real API calls happen (Network tab).

---

Rollback strategy

Every phase is shippable on its own. If a phase introduces a regression:

• Phase 1–2: git revert the merge commit. api.js retains the legacy interface, so callers that haven't migrated still work.
• Phase 3: revert + the relation-vocabulary migration is reversible (rerun the reverse swap).
• Phase 4: revert; quiz schema additions are forward-compatible (older readers ignore difficulty).
• Phase 5: revert + drop the llm_calls collection if undesired.
• Phase 6: purely additive; remove the evals/ folder if abandoned.

Never git push --force to main. PR-per-phase.

---

Out of scope (do not do as part of this plan)

• Replacing PocketBase. Stays as-is.
• Server-side embeddings or a vector store. Phase 5 deliberately uses in-browser TF-IDF.
• Streaming responses. Mentioned as a future improvement; not in this plan.
• Multi-tenant changes. The platform serves one company.
• UI redesign of the Admin pages beyond what each phase requires.

---

Glossary

• Tier — coarse model class (fast/standard/reasoning) mapped to a concrete Anthropic model ID via admin settings.
• Tool use — Anthropic's structured-output mechanism. The model emits a tool_use content block whose input is schema-valid JSON.
• Prompt caching — Anthropic feature where blocks marked cache_control: { type:'ephemeral' } are reused across requests at lower input cost. 5-minute TTL.
• TF-IDF — Term-frequency / inverse-document-frequency. A classic IR scoring function used here as a cheap retrieval signal.
• KB context — The block in R42's system prompt that lists topics and relations from the knowledge graph.
• Delta — A proposed addition to the knowledge graph emitted by R42 via the propose_graph_delta tool.