feat(r42): improve KB grounding accuracy and add clear-history
R42 was missing knowledge-graph information (e.g. pension questions) because retrieval and context-building dropped relevant facts: - retrieval: exact-token TF-IDF could not match Dutch compound words, so a "pensioen" query scored 0 against "pensioenregeling" / "partnerpensioen" and never retrieved them. Add a compound-word fallback (shared >=6-char stem or containment, 0.4x weight) alongside exact matching. - rag: deep article content was only injected for verbatim-mentioned topics; retrieved topics contributed just a 200-char description. Inject ~1000 chars of content for up to 5 topics (mentions first, then top-ranked retrieved) and widen the description snippet to 320. - prompts: add a NAUWKEURIGHEID block (use all relevant facts, call lookup_topic before giving up) and relax the 4-sentence cap for detail/list answers so complete facts aren't summarised away. Also add a clear-history control: a trash button in the chat header (confirm dialog) wipes chat🧵{userId} and reseeds the greeting via clearThread() in useChat. Tests: compound-word matching + rag deep-content injection. Spec updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -63,6 +63,29 @@ export function buildIndex(topics) {
|
||||
return index;
|
||||
}
|
||||
|
||||
// Compound-word matching. Dutch is heavily compounding, so a user's word
|
||||
// (`pensioenafspraken`) is a *different* token than the graph's labels
|
||||
// (`pensioenregeling`, `partnerpensioen`), even though they share the stem
|
||||
// `pensioen`. Exact TF-IDF scores those pairs at 0, so the relevant topics are
|
||||
// never retrieved. These heuristics recover that recall at a reduced weight,
|
||||
// so exact matches still dominate the ranking.
|
||||
const PARTIAL_MIN_QUERY_LEN = 6; // only expand meaty query tokens
|
||||
const PARTIAL_MIN_OVERLAP = 6; // shared stem / substring must be this long
|
||||
const PARTIAL_WEIGHT = 0.4; // discount vs. an exact term hit
|
||||
|
||||
/** True when two distinct tokens share a long stem or one contains the other. */
|
||||
function partialMatch(q, d) {
|
||||
if (q === d) return false;
|
||||
const shorter = q.length <= d.length ? q : d;
|
||||
const longer = q.length <= d.length ? d : q;
|
||||
if (shorter.length < PARTIAL_MIN_OVERLAP) return false;
|
||||
if (longer.includes(shorter)) return true;
|
||||
let n = 0;
|
||||
const m = shorter.length;
|
||||
while (n < m && q[n] === d[n]) n++;
|
||||
return n >= PARTIAL_MIN_OVERLAP;
|
||||
}
|
||||
|
||||
export function retrieveTopK(index, query, k = 10) {
|
||||
if (!index || !index.N || !query) return [];
|
||||
const qTokens = tokenize(query);
|
||||
@@ -80,8 +103,19 @@ export function retrieveTopK(index, query, k = 10) {
|
||||
let s = 0;
|
||||
for (const t of qTokens) {
|
||||
const f = tf.get(t);
|
||||
if (!f) continue;
|
||||
s += (1 + Math.log(f)) * idf(t);
|
||||
if (f) {
|
||||
s += (1 + Math.log(f)) * idf(t);
|
||||
continue;
|
||||
}
|
||||
// No exact hit — try a compound-word match against this doc's terms.
|
||||
if (t.length < PARTIAL_MIN_QUERY_LEN) continue;
|
||||
let best = 0;
|
||||
for (const [term, tf2] of tf) {
|
||||
if (!partialMatch(t, term)) continue;
|
||||
const w = PARTIAL_WEIGHT * (1 + Math.log(tf2)) * idf(term);
|
||||
if (w > best) best = w;
|
||||
}
|
||||
s += best;
|
||||
}
|
||||
scores[i] = s;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user