R42 was missing knowledge-graph information (e.g. pension questions)
because retrieval and context-building dropped relevant facts:
- retrieval: exact-token TF-IDF could not match Dutch compound words,
so a "pensioen" query scored 0 against "pensioenregeling" /
"partnerpensioen" and never retrieved them. Add a compound-word
fallback (shared >=6-char stem or containment, 0.4x weight) alongside
exact matching.
- rag: deep article content was only injected for verbatim-mentioned
topics; retrieved topics contributed just a 200-char description.
Inject ~1000 chars of content for up to 5 topics (mentions first,
then top-ranked retrieved) and widen the description snippet to 320.
- prompts: add a NAUWKEURIGHEID block (use all relevant facts, call
lookup_topic before giving up) and relax the 4-sentence cap for
detail/list answers so complete facts aren't summarised away.
Also add a clear-history control: a trash button in the chat header
(confirm dialog) wipes chat🧵{userId} and reseeds the greeting
via clearThread() in useChat.
Tests: compound-word matching + rag deep-content injection. Spec updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Opening an onboarding theme could fail with "LLM output failed schema
validation": the fast-tier model occasionally emits the key_points and
topics_covered tool fields as JSON-encoded strings instead of real
arrays. Same failure class the quiz schema already guards against with
z.preprocess.
- onboardingOverviewSchema: parse "[...]" strings back to arrays before
validation; split a bullet/newline string of key points as a fallback;
keep the first 5 key points on overage instead of failing the user.
Genuinely bad output (too few points, non-JSON strings, wrong types)
still fails validation.
- emit_onboarding_overview tool: descriptions now state the fields must
be JSON arrays, never strings (prompt-side nudge).
- 5 regression tests reproducing the exact #32 payload shapes
(npm test 139/139).
Closes#32
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A self-paced onboarding track that introduces a new employee to every KB
theme in breadth (not depth), so they grasp how Respellion works day to
day and week to week. Offered as a CTA inside the Dashboard "New here?"
explainer card; always available regardless of enrollment.
Design:
- Theme is the trackable unit; the 5 "days" are a read-time presentation
grouping, so re-chunking never loses progress. Completion is stored per
theme in onboarding_completions.
- Per-theme overview generated lazily on first open (fast-tier
emit_onboarding_overview tool), cached in onboarding_overviews keyed by
theme + a topics_fingerprint that triggers regeneration when the theme's
topic set changes.
- Reachable via /onboarding-track using the existing skipEnrollmentGate
prop, decoupled from the 26-week curriculum (distinct from /onboarding,
the enrollment page).
Backend:
- pb_migrations/1781200000_created_onboarding.js: two collections with
authenticated-only rules and unique indexes; TEXT team_member_id (no
relation) per the post-#18/#27 convention. Mirrored in
scripts/setup-pb-collections.mjs.
- src/lib/onboardingService.js: pure helpers (orderThemes,
distributeThemesIntoDays, computeTopicsFingerprint,
computeOnboardingProgress, buildOnboardingPlan) + generation + I/O.
- db.js onboarding helpers use pb.filter() bindings (theme is free text).
- LLM tool + Zod schema + registry + simulation stub.
Frontend:
- src/pages/OnboardingTrack.jsx (day list, per-theme overview, completion
banner, progress ring/day bar).
- Dashboard "New here?" card CTA + X/5-days progress chip (hidden when the
KB has no themes).
Docs: data-model, generation-spec (§D), frontend-spec updated.
Verified: 22 new unit tests (npm test 134/134), eslint clean on changed
files, npm run build OK, PocketBase v0.30.4 boot applies the migration
(collections + unique indexes + authed rules confirmed), and a backend
contract check (upsert idempotency, unique-index guard, special-char
theme filtering).
Closes#30
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces the client-side PIN login with real authentication against Azure
Entra ID via PocketBase's built-in OAuth2 (OIDC) flow. Every Azure user is
auto-provisioned as a team_member on first login.
- pb_migrations: team_members becomes a PocketBase auth collection (OIDC
provider configured from env); all collections require @request.auth.id
- pb_hooks: provisioning of role (ENTRA_ADMIN_EMAILS allow-list) and
enrollment_status, with admin-role re-sync on every login
- frontend: "Sign in with Microsoft" login, pb.authStore-based session,
TeamManager manages roster/roles (no PIN/manual create)
- infra: Entra env wiring + pb_hooks mount for dev (Labs) and prod
- src/lib/azureAuth.js + tests for the canonical role/allow-list logic
- docs/auth-spec.md
Knowledge base, generated tests and micro-learnings are left untouched.
Existing PIN users are dropped (no migration required, per sign-off).
Closes#16
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Topics met learning_relevance="exclude" of relevance_locked=true werden
door de Full Analysis stilletjes verwijderd: het model zag exclude als
"irrelevant" en stelde ze voor in actions.deletions / actions.merges,
en analyzeGraph paste die zonder filter toe. Het locked-vlaggetje werd
in full-scope payload niet eens meegestuurd, dus zelfs een nieuwe prompt
kon niet helpen.
Defense-in-depth fixes:
1. Pure graphGuard.filterAiActions strips elke deletion/merge waarvan de
target excluded of locked is, vóór bulkSave. Merges waarin een
protected topic juist de keepId is (canonical survivor) blijven door.
2. SYSTEM_PROMPTS.full krijgt een expliciete "PROTECTED TOPICS" sectie
die exclude en locked topics als nooit-te-verwijderen markeert.
3. relevance_locked wordt nu ook in de full-scope compactTopics payload
meegestuurd, zodat het model de vlag überhaupt ziet.
4. UI-feedback: GraphControls toont een ShieldCheck banner met aantal
geblokkeerde acties na de analyze, en het excluded-aantal naast de
"Show Excluded Nodes" toggle (visible/hidden).
5. 13 nieuwe Vitest cases dekken protected detection en alle drop/keep
paden van filterAiActions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When the KB has more than 26 themes the curriculum AI is required to
merge — so theme labels not appearing as week names are expected, not
warnings. The previous validation surfaced this as a console.warn that
read like an error in the learning station console.
- validateSchedule now only flags missing theme labels when themes_kb <= 26
- adds a real coverage check: warns when learning topics are absent from
every week (the actual signal we care about)
- adds vitest coverage for both behaviours
Closes#12
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a /topic-test route where learners can take a 5-question test on any
eligible topic at any time. Questions come from a shared, admin-curated
question_bank — no LLM calls on the user path. Points feed the existing
leaderboard with a 10pt/topic/week cap (per ISO week) so the bank can't be
farmed, and repeats from a thin bank yield 0 points.
- New PB collections: question_bank, on_demand_attempts (+ migrations and
setup-pb-collections.mjs entries).
- db.js: un-deprecates getQuizBank/setQuizBank against question_bank so the
existing admin TestManager panel becomes functional again; adds
saveOnDemandAttempt, getOnDemandPointsThisWeek, getUserSeenQuestionIds,
getAllOnDemandAttempts, isoWeekKey.
- testService.js: getEligibleTopicsForOnDemand, startOnDemandTest (unseen-
first draw), finishOnDemandTest (cap enforcement + leaderboard upsert).
- New TopicTest page, route, and nav entry; weekly test flow untouched.
- Leaderboard folds on-demand 100%s into perfect-score count and merges
on-demand attempts into the recent activity feed.
- Spec: docs/on-demand-tests-spec.md with ADRs and extension points.
- Tests: 5 new vitest cases covering eligibility, draw fallback, scoring,
partial cap, and exhausted cap (85/85 total).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add dependency-free TF-IDF retrieval (src/lib/retrieval.js) with NL+EN
stopwords and a WeakMap-cached index.
- Rewrite buildKbContext to ship the top-K relevant topics + verbatim-
mentioned ids only, filter relations to the included set, and append a
[kb_hash: <8 hex>] suffix so the ephemeral prompt cache busts when the
graph changes. Returns { context, retrievedTopics, allTopics }.
- Add LOOKUP_TOPIC_TOOL and drive useChat through callLLM directly with a
multi-hop tool_result loop capped at 3 hops; preserve Anthropic-provided
tool_use ids through callLLM so the loop can echo correct tool_use_id.
- Truncate R42 history to the last 12 turns and prepend a single
"(earlier conversation truncated)" assistant message.
- Set R42 chat defaults: temperature 0.3, maxTokens 2048.
- Add pb_migrations/1780500002_created_llm_calls.js (the best-effort
logger in callLLM was already wired) and a new Admin → Diagnostics
view showing the last 100 calls with token usage, cache-hit rate, and
USD cost from a local Anthropic price table.
- Finalize AI_PIPELINE_HARDENING_PLAN.md: mark Phases 1–5 shipped and
Phase 6 (eval harness) explicitly out of scope.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- src/lib/random.js: Fisher–Yates shuffle/sample/pickInt; replace every
biased .sort(() => 0.5 - Math.random()) site in testService.
- testService: debias correctIndex via prompt + runtime re-roll (up to 2x
when one position holds >50%); quality gate rejecting <4 distinct
options, banned filler ("all of the above" etc) and explanations
shorter than 20 chars; dedup new questions against the existing bank
via normalised question text.
- Quiz schema/tool/prompt require difficulty ('easy'|'medium'|'hard');
db.getQuizBank defaults legacy records to 'medium' on read.
- learningService.generateCustomTopic: kebab-case slug ID from the
polished label with collision suffixes; default learning_relevance
'standard' when the model omits it.
- Tests for random helpers, dedup/quality-gate behaviour and the
extended quiz schema.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace stateless one-shot extraction with a stateful, paced, cancellable
pipeline. Six subtasks:
- 3.1 Sentence-aware chunking with 800-char overlap (was paragraph-only
at 4000 chars). Hard-split fallback for runaway sentences.
- 3.2 Stateful extraction: chunks 2+ receive an "already-extracted topic
IDs" hint capped at 200 IDs, so the model reuses IDs instead of
inventing variants like software-developer vs software-engineer.
- 3.3 Token-bucket limiter in llmRetry.js (extractionLimiter, 5 req/min).
callLLM awaits the limiter before fetch; 429+Retry-After calls
pauseUntil. Replaces hard setTimeout(12000) and setTimeout(15000).
- 3.4 relevance_locked column on topics — admin edits to relevance are
sticky across re-extraction. Migration + merge respects the flag +
unlock checkbox in KnowledgeGraph edit form.
- 3.5 Unify relation vocabulary — handbook prompt no longer mentions
legacy "executes"; one-shot migration rewrites existing executes rows
to executed_by with source/target swapped.
- 3.6 Cancellation — Cancel button on UploadZone wired to an
AbortController threaded into callLLM; aborted runs persist status =
"cancelled" rather than "failed".
Tests: 16 new unit tests for chunkText, buildKnownIdsHint, and
createLimiter. All 61 tests pass, 0 lint errors, build clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Every structured-output call now uses an Anthropic tool instead of
parsing JSON out of free-form prose, and stable system prompts are
sent as cacheable blocks. Behaviour-equivalent to phase 1 from the
caller's point of view; the savings show up in token usage and in the
absence of "AI returned non-JSON response" failure modes.
* src/lib/llmTools.js — single source of truth for tool definitions:
emit_knowledge_graph, emit_handbook_delta, emit_learning_article /
_slides / _infographic / _all, emit_custom_topic, emit_quiz_questions,
emit_graph_actions, plus five article-patch tools (set_intro,
set_section, add_section, remove_section, replace_takeaways).
* src/lib/articlePatches.js — pure applyArticlePatches +
applyAndValidate; rebuilds the article from a sequence of patch tool
calls and re-validates against learningArticleSchema. set_section
falls back to appending when no matching heading exists so the
model's intent is preserved rather than silently dropped.
* src/lib/llmSchemas.js — Zod schemas for the five patch ops,
registered in toolSchemaRegistry so callLLM validates them
automatically.
* src/lib/llm.js — simulation mode now returns a tool_use stub matching
toolChoice.name, so the UI keeps working with Simulation Mode on
after the structured-output migration.
* src/lib/extractionPipeline.js — processSourceText and
analyzeHandbookDelta migrated to callLLM + tool use. System prompts
sent as { cache_control: ephemeral } blocks. Handbook results pass
through normalizeHandbookResult to collapse legacy "executes"
relations into executed_by with swapped source/target.
* src/lib/learningService.js — generateLearningContent picks the right
tool per selectedType; generateCustomTopic uses emit_custom_topic;
refineLearningContent now drives the five patch tools with
toolChoice 'any' and rejects the whole turn if the patched article
fails validation. Article-only refinement is intentional for phase 2;
refining a topic without an article surfaces a clear error.
* src/lib/testService.js — quiz generation via emit_quiz_questions.
* src/components/admin/KnowledgeGraph.jsx — analyzeGraph routed through
the reasoning tier (Opus) since graph-wide consolidation benefits
from a stronger reasoner.
* src/components/chat/prompts.js — buildSystemPrompt now returns three
text blocks: stable preamble (cached), KB context (cached, hash-bust
deferred to phase 5), per-turn user/admin tail (uncached).
* src/lib/__tests__/ — 13 new tests covering each patch op, multi-op
sequencing, post-patch validation failure, and tool/registry shape.
Acceptance: lint and 45/45 tests green; build succeeds; no
`match(/\{[\s\S]*\}/)` JSON extraction left in src/. Live verification
of cache hits on a second extraction within 5 minutes is deferred to
manual smoke testing — needs real `/api/anthropic` traffic.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Implements phase 1 of AI_PIPELINE_HARDENING_PLAN.md. Every Anthropic call
now goes through one module that owns retry, timeout, abort, structured-
output parsing, schema validation, and best-effort call telemetry.
* src/lib/llm.js — single callLLM entry point. Resolves model per tier
(fast / standard / reasoning) with admin:model legacy fallback for the
standard tier; 60s default timeout via AbortController; balanced-brace
JSON extraction; LLMHttpError, LLMTruncatedError, LLMOutputError, and
LLMValidationError surface clearly distinct failure modes.
* src/lib/llmRetry.js — exponential backoff with full jitter, retries
only on transient HTTP statuses, honours Retry-After up to 60s, never
retries on AbortError.
* src/lib/llmSchemas.js — Zod schemas for every structured task plus
normalizeHandbookResult (collapses legacy "executes" relations into
the canonical "executed_by" vocabulary).
* src/lib/api.js — thin shim over callLLM so existing callers (extraction
pipeline, learning, quiz, R42, knowledge graph) keep working unchanged.
* src/lib/__tests__/ — 32 Vitest cases covering parse paths, error
surfaces, simulation mode, model resolution, and schema validation.
* src/pages/Admin/index.jsx — three model inputs (fast / standard /
reasoning) replacing the single legacy field; legacy value falls back
for the standard tier so existing overrides survive.
Adds Zod and Vitest, plus an "npm run test" script.
Also cleans up the pre-existing repo-wide ESLint failures so phase 1's
"npm run lint passes" acceptance criterion can be checked: drops unused
React imports across the JSX tree (React 19 JSX runtime auto-imports),
attaches cause to rethrown errors in the service modules, ignores
pb_migrations in the ESLint config (PocketBase JSVM globals), and
removes one dead handleCreateCustom function in Leren.jsx. A real
behaviour bug surfaced in Testen.jsx — the quiz timer captured a stale
finishQuiz via setInterval closure; now updated via finishQuizRef so the
timer always invokes the latest callback.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
