feat: phase 2 of AI pipeline hardening — tool-based structured outputs + prompt caching
All checks were successful
On Pull Request to Main / test (pull_request) Successful in 31s
On Pull Request to Main / publish (pull_request) Successful in 1m1s
On Pull Request to Main / deploy-dev (pull_request) Successful in 1m31s

Every structured-output call now uses an Anthropic tool instead of
parsing JSON out of free-form prose, and stable system prompts are
sent as cacheable blocks. Behaviour-equivalent to phase 1 from the
caller's point of view; the savings show up in token usage and in the
absence of "AI returned non-JSON response" failure modes.

* src/lib/llmTools.js — single source of truth for tool definitions:
  emit_knowledge_graph, emit_handbook_delta, emit_learning_article /
  _slides / _infographic / _all, emit_custom_topic, emit_quiz_questions,
  emit_graph_actions, plus five article-patch tools (set_intro,
  set_section, add_section, remove_section, replace_takeaways).
* src/lib/articlePatches.js — pure applyArticlePatches +
  applyAndValidate; rebuilds the article from a sequence of patch tool
  calls and re-validates against learningArticleSchema. set_section
  falls back to appending when no matching heading exists so the
  model's intent is preserved rather than silently dropped.
* src/lib/llmSchemas.js — Zod schemas for the five patch ops,
  registered in toolSchemaRegistry so callLLM validates them
  automatically.
* src/lib/llm.js — simulation mode now returns a tool_use stub matching
  toolChoice.name, so the UI keeps working with Simulation Mode on
  after the structured-output migration.
* src/lib/extractionPipeline.js — processSourceText and
  analyzeHandbookDelta migrated to callLLM + tool use. System prompts
  sent as { cache_control: ephemeral } blocks. Handbook results pass
  through normalizeHandbookResult to collapse legacy "executes"
  relations into executed_by with swapped source/target.
* src/lib/learningService.js — generateLearningContent picks the right
  tool per selectedType; generateCustomTopic uses emit_custom_topic;
  refineLearningContent now drives the five patch tools with
  toolChoice 'any' and rejects the whole turn if the patched article
  fails validation. Article-only refinement is intentional for phase 2;
  refining a topic without an article surfaces a clear error.
* src/lib/testService.js — quiz generation via emit_quiz_questions.
* src/components/admin/KnowledgeGraph.jsx — analyzeGraph routed through
  the reasoning tier (Opus) since graph-wide consolidation benefits
  from a stronger reasoner.
* src/components/chat/prompts.js — buildSystemPrompt now returns three
  text blocks: stable preamble (cached), KB context (cached, hash-bust
  deferred to phase 5), per-turn user/admin tail (uncached).
* src/lib/__tests__/ — 13 new tests covering each patch op, multi-op
  sequencing, post-patch validation failure, and tool/registry shape.

Acceptance: lint and 45/45 tests green; build succeeds; no
`match(/\{[\s\S]*\}/)` JSON extraction left in src/. Live verification
of cache hits on a second extraction within 5 minutes is deferred to
manual smoke testing — needs real `/api/anthropic` traffic.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
RaymondVerhoef
2026-05-20 15:47:20 +02:00
parent 8a8745fad2
commit f838755991
11 changed files with 872 additions and 291 deletions

View File

@@ -0,0 +1,104 @@
import { describe, expect, it } from 'vitest';
import { applyArticlePatches, applyAndValidate } from '../articlePatches';
const article = () => ({
title: 'Onboarding',
intro: 'Old intro.',
sections: [
{ heading: 'Day one', body: 'First day body, three sentences long. Welcome. Read the handbook.' },
{ heading: 'Day two', body: 'Second day body. Three sentences. Meet your team.' },
],
keyTakeaways: ['Show up', 'Ask questions'],
});
describe('applyArticlePatches', () => {
it('does not mutate the input article', () => {
const original = article();
const snapshot = JSON.parse(JSON.stringify(original));
applyArticlePatches(original, [
{ name: 'set_intro', input: { intro: 'New intro.' } },
]);
expect(original).toEqual(snapshot);
});
it('set_intro replaces the intro', () => {
const result = applyArticlePatches(article(), [
{ name: 'set_intro', input: { intro: 'Punchier intro.' } },
]);
expect(result.intro).toBe('Punchier intro.');
});
it('set_section replaces the matching section body (case-insensitive)', () => {
const result = applyArticlePatches(article(), [
{ name: 'set_section', input: { heading: 'DAY ONE', body: 'Rewritten body. With several sentences. Indeed.' } },
]);
expect(result.sections[0].body).toMatch(/Rewritten body/);
expect(result.sections[1].body).toMatch(/Second day body/);
});
it('add_section position=start prepends a new section', () => {
const result = applyArticlePatches(article(), [
{ name: 'add_section', input: { heading: 'Before', body: 'New intro section. Three sentences. Indeed.', position: 'start' } },
]);
expect(result.sections[0].heading).toBe('Before');
expect(result.sections).toHaveLength(3);
});
it('add_section position=end appends a new section', () => {
const result = applyArticlePatches(article(), [
{ name: 'add_section', input: { heading: 'After', body: 'Closing section. Three sentences. Indeed.', position: 'end' } },
]);
expect(result.sections[2].heading).toBe('After');
});
it('remove_section drops the matching section', () => {
const result = applyArticlePatches(article(), [
{ name: 'remove_section', input: { heading: 'Day one' } },
]);
expect(result.sections).toHaveLength(1);
expect(result.sections[0].heading).toBe('Day two');
});
it('replace_takeaways swaps the key takeaways', () => {
const result = applyArticlePatches(article(), [
{ name: 'replace_takeaways', input: { items: ['First', 'Second', 'Third'] } },
]);
expect(result.keyTakeaways).toEqual(['First', 'Second', 'Third']);
});
it('applies multiple patches in order', () => {
const result = applyArticlePatches(article(), [
{ name: 'set_intro', input: { intro: 'Brand new intro.' } },
{ name: 'remove_section', input: { heading: 'Day one' } },
{ name: 'add_section', input: { heading: 'New', body: 'Body of the new section. Three sentences. Yes.', position: 'end' } },
]);
expect(result.intro).toBe('Brand new intro.');
expect(result.sections.map(s => s.heading)).toEqual(['Day two', 'New']);
});
it('falls back to appending when set_section cannot find a matching heading', () => {
const result = applyArticlePatches(article(), [
{ name: 'set_section', input: { heading: 'Nonexistent', body: 'New body, with three sentences. Yes indeed. Foo.' } },
]);
expect(result.sections).toHaveLength(3);
expect(result.sections[2].heading).toBe('Nonexistent');
});
});
describe('applyAndValidate', () => {
it('returns the patched article when valid', () => {
const patched = applyAndValidate(article(), [
{ name: 'set_intro', input: { intro: 'Tighter intro.' } },
]);
expect(patched.intro).toBe('Tighter intro.');
});
it('throws when patches strip the article to invalid', () => {
expect(() =>
applyAndValidate(article(), [
{ name: 'remove_section', input: { heading: 'Day one' } },
{ name: 'remove_section', input: { heading: 'Day two' } },
]),
).toThrow(/invalid article/i);
});
});