R42 was missing knowledge-graph information (e.g. pension questions)
because retrieval and context-building dropped relevant facts:
- retrieval: exact-token TF-IDF could not match Dutch compound words,
so a "pensioen" query scored 0 against "pensioenregeling" /
"partnerpensioen" and never retrieved them. Add a compound-word
fallback (shared >=6-char stem or containment, 0.4x weight) alongside
exact matching.
- rag: deep article content was only injected for verbatim-mentioned
topics; retrieved topics contributed just a 200-char description.
Inject ~1000 chars of content for up to 5 topics (mentions first,
then top-ranked retrieved) and widen the description snippet to 320.
- prompts: add a NAUWKEURIGHEID block (use all relevant facts, call
lookup_topic before giving up) and relax the 4-sentence cap for
detail/list answers so complete facts aren't summarised away.
Also add a clear-history control: a trash button in the chat header
(confirm dialog) wipes chat🧵{userId} and reseeds the greeting
via clearThread() in useChat.
Tests: compound-word matching + rag deep-content injection. Spec updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Add dependency-free TF-IDF retrieval (src/lib/retrieval.js) with NL+EN
stopwords and a WeakMap-cached index.
- Rewrite buildKbContext to ship the top-K relevant topics + verbatim-
mentioned ids only, filter relations to the included set, and append a
[kb_hash: <8 hex>] suffix so the ephemeral prompt cache busts when the
graph changes. Returns { context, retrievedTopics, allTopics }.
- Add LOOKUP_TOPIC_TOOL and drive useChat through callLLM directly with a
multi-hop tool_result loop capped at 3 hops; preserve Anthropic-provided
tool_use ids through callLLM so the loop can echo correct tool_use_id.
- Truncate R42 history to the last 12 turns and prepend a single
"(earlier conversation truncated)" assistant message.
- Set R42 chat defaults: temperature 0.3, maxTokens 2048.
- Add pb_migrations/1780500002_created_llm_calls.js (the best-effort
logger in callLLM was already wired) and a new Admin → Diagnostics
view showing the last 100 calls with token usage, cache-hit rate, and
USD cost from a local Anthropic price table.
- Finalize AI_PIPELINE_HARDENING_PLAN.md: mark Phases 1–5 shipped and
Phase 6 (eval harness) explicitly out of scope.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
