feat: add specification for micro learning generation and remove deprecated pipeline planning files
This commit is contained in:
RaymondVerhoef
@@ -1,573 +0,0 @@
|
|||||||
# AI Pipeline Hardening — Implementation Plan
|
|
||||||
|
|
||||||
> **Audience:** an AI agent executing this plan against the Respellion Learning Platform.
|
|
||||||
> **Owner before this work:** Raymond Verhoef (rve@respellion.nl).
|
|
||||||
> **Source of truth for repo conventions:** [`AI_AGENT.md`](AI_AGENT.md). Read it before starting.
|
|
||||||
|
|
||||||
This plan upgrades the platform's interaction with the Anthropic API: how prompts are built, how responses are parsed, how the model is retried, and how outputs are validated. It is broken into six phases that can be implemented and shipped independently. Each phase ends with verifiable acceptance criteria.
|
|
||||||
|
|
||||||
> **Status (2026-05-20):** Phases 1–5 implemented and shipped. Phase 6 (eval harness) is intentionally **out of scope** for this initiative — the production pipeline is hardened to the level the platform needs, and a golden-set runner can be reopened later as a stand-alone task if regression risk grows. The repo no longer carries any TODO from this plan.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 0. Operating principles
|
|
||||||
|
|
||||||
These rules govern every phase. Re-read them before you commit.
|
|
||||||
|
|
||||||
1. **PocketBase is the source of truth.** No persistent state in localStorage (see [`AI_AGENT.md`](AI_AGENT.md) §2). The Anthropic API is proxied via Caddy; **never** add `x-api-key` headers in frontend code.
|
|
||||||
2. **No behaviour regressions.** Existing UI flows (extraction, weekly learning, weekly quiz, R42 chat, handbook sync, analyze-graph) must keep working after every phase. Phases are additive.
|
|
||||||
3. **Schema-first.** Where the model produces structured output, define a JSON Schema (Zod) and validate every response. Reject (don't paper over) malformed output.
|
|
||||||
4. **Single LLM entry point.** After Phase 1 there is exactly one module that talks to `/api/anthropic/v1/messages`. All callers go through it.
|
|
||||||
5. **No silent truncation.** If `stop_reason === 'max_tokens'` on a structured-output call, throw — never persist a partial parse.
|
|
||||||
6. **Cache what is stable, vary what is dynamic.** Use prompt caching for system prompts and KB context. User messages are never cached.
|
|
||||||
7. **Comments and docs:** follow the repo's terse style. Don't add explanatory comments unless the *why* is non-obvious.
|
|
||||||
8. **Migrations:** when changing a PocketBase schema, add a migration in `pb_migrations/` following the existing timestamp prefix convention. Never edit shipped migrations.
|
|
||||||
9. **Stop and ask** if you encounter a decision the plan doesn't cover (e.g. a model deprecation, a missing collection, a failing test that looks pre-existing).
|
|
||||||
|
|
||||||
### Files you will touch (or create) across all phases
|
|
||||||
|
|
||||||
| Path | Purpose |
|
|
||||||
|---|---|
|
|
||||||
| `src/lib/llm.js` *(new, Phase 1)* | Single Anthropic client wrapper |
|
|
||||||
| `src/lib/llmSchemas.js` *(new, Phase 1)* | Zod schemas for every structured task |
|
|
||||||
| `src/lib/llmRetry.js` *(new, Phase 1)* | Retry + backoff + abort policy |
|
|
||||||
| `src/lib/random.js` *(new, Phase 4)* | Fisher–Yates shuffle + RNG helpers |
|
|
||||||
| `src/lib/api.js` | Becomes a thin re-export from `llm.js` (back-compat) |
|
|
||||||
| `src/lib/extractionPipeline.js` | Migrated to `llm.js` + tool use + overlap chunking |
|
|
||||||
| `src/lib/learningService.js` | Migrated to `llm.js` + tool use + patch-refine |
|
|
||||||
| `src/lib/testService.js` | Migrated to `llm.js` + tool use + dedup + shuffle fix |
|
|
||||||
| `src/components/admin/KnowledgeGraph.jsx` | `analyzeGraph` → tool use + dry-run preview |
|
|
||||||
| `src/components/chat/rag.js` | Retrieval (TF-IDF) + `lookup_topic` tool |
|
|
||||||
| `src/components/chat/prompts.js` | Split system prompt into cacheable + dynamic |
|
|
||||||
| `src/components/chat/useChat.js` | Wire retrieval + truncation |
|
|
||||||
| `pb_migrations/*` | Schema additions for `llm_calls`, question `difficulty`, topic `relevance_locked` |
|
|
||||||
| `evals/` *(new, Phase 6)* | Golden-set eval harness |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 1 — Foundation (single LLM client + robust parsing)
|
|
||||||
|
|
||||||
**Goal:** every LLM call goes through one module that handles retry, timeout, abort, JSON extraction, and schema validation. No behaviour change visible to the user.
|
|
||||||
|
|
||||||
### 1.1 Create `src/lib/llmRetry.js`
|
|
||||||
|
|
||||||
Implements the retry policy used by `llm.js`.
|
|
||||||
|
|
||||||
**Behaviour:**
|
|
||||||
- Exponential backoff with full jitter, base 1000ms, cap 16000ms.
|
|
||||||
- Retries only on these HTTP statuses: `408, 425, 429, 500, 502, 503, 504, 529`.
|
|
||||||
- Honours `Retry-After` header (seconds or HTTP date). If present and ≤ 60s, use it; if > 60s, fail fast.
|
|
||||||
- Default `maxRetries = 4`.
|
|
||||||
- Does **not** retry on `AbortError`.
|
|
||||||
|
|
||||||
**Exported interface:**
|
|
||||||
|
|
||||||
```js
|
|
||||||
// withRetry: (fn: (attempt:number) => Promise<T>, opts?) => Promise<T>
|
|
||||||
// RetryableError(status, retryAfterMs)
|
|
||||||
export async function withRetry(fn, { maxRetries = 4, signal } = {}) { ... }
|
|
||||||
export class RetryableError extends Error { constructor(status, retryAfterMs) { ... } }
|
|
||||||
```
|
|
||||||
|
|
||||||
### 1.2 Create `src/lib/llmSchemas.js`
|
|
||||||
|
|
||||||
One Zod schema per structured task. Install Zod (`npm i zod`).
|
|
||||||
|
|
||||||
Required schemas (names + shape match what callers already produce — do not change field names):
|
|
||||||
|
|
||||||
- `extractionResultSchema` — `{ topics: Topic[], relations: Relation[] }` matching the existing `SYSTEM_PROMPT` in [`extractionPipeline.js`](src/lib/extractionPipeline.js).
|
|
||||||
- `handbookResultSchema` — same shape, but `relation.type` enum unified to `related_to | depends_on | part_of | executed_by` (see Phase 3 task 3.5 — for now the schema accepts both `executes` and `executed_by`, normalize `executes → executed_by` post-validation).
|
|
||||||
- `learningArticleSchema`, `learningSlidesSchema`, `learningInfographicSchema`, `learningAllSchema` matching [`learningService.js`](src/lib/learningService.js).
|
|
||||||
- `quizQuestionsSchema` — `{ questions: Question[] }` with `options.length === 4` and `correctIndex ∈ [0,3]`.
|
|
||||||
- `customTopicSchema` — `{ label, type: 'concept'|'role'|'process', description }`.
|
|
||||||
- `graphActionsSchema` — `{ merges, deletions, newRelations, relevanceUpdates }` matching [`KnowledgeGraph.jsx:329`](src/components/admin/KnowledgeGraph.jsx).
|
|
||||||
- `proposeGraphDeltaSchema` — matches `PROPOSE_GRAPH_DELTA_TOOL.input_schema` in [`prompts.js`](src/components/chat/prompts.js).
|
|
||||||
|
|
||||||
**Acceptance:** every schema has at least one happy-path Vitest test in `src/lib/__tests__/llmSchemas.test.js` (add `vitest` if not present).
|
|
||||||
|
|
||||||
### 1.3 Create `src/lib/llm.js`
|
|
||||||
|
|
||||||
The single Anthropic client. All other modules must call only this one.
|
|
||||||
|
|
||||||
**Public interface:**
|
|
||||||
|
|
||||||
```js
|
|
||||||
// Task tier — used to pick a model from settings.
|
|
||||||
// 'fast' → admin:model:fast (default: claude-haiku-4-5-20251001)
|
|
||||||
// 'standard' → admin:model:standard (default: claude-sonnet-4-6)
|
|
||||||
// 'reasoning' → admin:model:reasoning (default: claude-opus-4-7)
|
|
||||||
export async function callLLM({
|
|
||||||
task, // string, e.g. 'extract.source' — used for logging only
|
|
||||||
tier = 'standard',
|
|
||||||
system, // string OR Array<{ type:'text', text:string, cache_control?:{type:'ephemeral'} }>
|
|
||||||
messages, // [{ role, content }] OR omitted (use `user`)
|
|
||||||
user, // shorthand for [{role:'user', content: user}]
|
|
||||||
tools, // optional Anthropic tool definitions
|
|
||||||
toolChoice, // optional, e.g. { type:'tool', name:'emit_knowledge_graph' }
|
|
||||||
schema, // optional Zod schema for text→JSON path (used only when no tool)
|
|
||||||
maxTokens = 4096,
|
|
||||||
temperature = 0,
|
|
||||||
signal, // AbortSignal
|
|
||||||
}): Promise<{
|
|
||||||
text: string,
|
|
||||||
toolUses: Array<{ name, input }>,
|
|
||||||
stopReason: string,
|
|
||||||
usage: { input_tokens, output_tokens, cache_creation_input_tokens, cache_read_input_tokens },
|
|
||||||
requestId: string | null,
|
|
||||||
model: string,
|
|
||||||
durationMs: number,
|
|
||||||
}>
|
|
||||||
```
|
|
||||||
|
|
||||||
**Key requirements:**
|
|
||||||
|
|
||||||
1. **Simulation mode** preserved: if `storage.get('admin:use_simulation') === true`, return a deterministic stub (use existing `simulateResponse` payload for backward compatibility — branch on `task` prefix to return a matching stub).
|
|
||||||
2. **Fetch** with `AbortController`; default **60-second timeout** if caller didn't pass a signal.
|
|
||||||
3. **Retry** through `withRetry` (Phase 1.1).
|
|
||||||
4. **Auth-portal detection** preserved: if response is not `application/json`, throw `Your session has expired. Please refresh the page and log in again.` exactly as today.
|
|
||||||
5. **No truncation acceptance**: if `stop_reason === 'max_tokens'` AND caller passed `schema` OR `toolChoice` requested a tool, throw `LLMTruncatedError`.
|
|
||||||
6. **Robust JSON extraction** when caller passed `schema` (and no tool was used): use `parseStructuredText(text)` that
|
|
||||||
- strips ```` ```json ```` and ```` ``` ```` fences,
|
|
||||||
- finds the outermost balanced JSON value (object **or** array) via a tiny brace-matching scan, not regex,
|
|
||||||
- throws `LLMOutputError` if no balanced JSON found,
|
|
||||||
- runs Zod `schema.parse` on the result.
|
|
||||||
7. **Tool path:** when `tools` is provided and the model emits `tool_use`, return them under `toolUses`. Validate each tool's input against the corresponding Zod schema if the caller wired one in (via `toolSchemas: { [toolName]: ZodSchema }`).
|
|
||||||
8. **Logging:** after every call, append a row to a new PocketBase collection `llm_calls` (best-effort — never block on this; catch and console.debug failures). Fields: `task, model, tier, duration_ms, input_tokens, output_tokens, cache_read_tokens, cache_create_tokens, stop_reason, ok, error_msg`. See Phase 5 task 5.6 for the migration.
|
|
||||||
9. **Custom errors**: `LLMHttpError`, `LLMTruncatedError`, `LLMOutputError`, `LLMValidationError`. All extend `Error` and set `name` for `instanceof` checks.
|
|
||||||
|
|
||||||
### 1.4 Make `src/lib/api.js` a thin shim
|
|
||||||
|
|
||||||
Replace the existing `anthropicApi.generateContent` and `anthropicApi.chat` implementations with calls into `llm.js`. Preserve the exact exported names and return shapes so no caller breaks.
|
|
||||||
|
|
||||||
```js
|
|
||||||
// api.js after Phase 1
|
|
||||||
export const anthropicApi = {
|
|
||||||
async generateContent(systemPrompt, userMessage, maxRetries = 1) {
|
|
||||||
const { text } = await callLLM({
|
|
||||||
task: 'legacy.generateContent',
|
|
||||||
tier: 'standard',
|
|
||||||
system: systemPrompt,
|
|
||||||
user: userMessage,
|
|
||||||
maxTokens: 8192,
|
|
||||||
temperature: 0,
|
|
||||||
});
|
|
||||||
return text;
|
|
||||||
},
|
|
||||||
async chat(systemPrompt, messages, opts = {}) {
|
|
||||||
const r = await callLLM({
|
|
||||||
task: 'legacy.chat',
|
|
||||||
tier: 'standard',
|
|
||||||
system: systemPrompt,
|
|
||||||
messages,
|
|
||||||
tools: opts.tools,
|
|
||||||
maxTokens: 1024,
|
|
||||||
temperature: 0.3, // chat default — see Phase 5
|
|
||||||
});
|
|
||||||
return {
|
|
||||||
content: [
|
|
||||||
...(r.text ? [{ type:'text', text: r.text }] : []),
|
|
||||||
...r.toolUses.map(tu => ({ type:'tool_use', name: tu.name, input: tu.input })),
|
|
||||||
],
|
|
||||||
stop_reason: r.stopReason,
|
|
||||||
};
|
|
||||||
},
|
|
||||||
};
|
|
||||||
```
|
|
||||||
|
|
||||||
### 1.5 Update default model + tiered settings
|
|
||||||
|
|
||||||
- Replace `DEFAULT_MODEL = 'claude-sonnet-4-20250514'` with the three tier defaults above.
|
|
||||||
- In **Admin → Settings**, add three model selects (`fast`, `standard`, `reasoning`). Read existing `admin:model` as a legacy fallback for `standard` (so existing users don't lose their override).
|
|
||||||
|
|
||||||
### Phase 1 acceptance criteria
|
|
||||||
|
|
||||||
- [ ] `npm run lint` passes; `npm run test` passes (Vitest).
|
|
||||||
- [ ] Every existing user flow (extraction, weekly content, weekly quiz, R42, handbook sync, analyze graph) still works against the live API.
|
|
||||||
- [ ] `grep -r "fetch.*anthropic" src/` returns only `src/lib/llm.js`.
|
|
||||||
- [ ] Simulation mode toggle still returns stubbed responses for all flows.
|
|
||||||
- [ ] Manually verify: kill the network mid-call → request aborts within 60s and surfaces a clear error message.
|
|
||||||
- [ ] Manually verify: rate-limit the proxy (429 + `Retry-After: 5`) → call retries once after ~5s and then succeeds.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 2 — Prompt caching & tool-based structured outputs
|
|
||||||
|
|
||||||
**Goal:** structured-output tasks no longer parse JSON out of prose. Large stable prompts are cached.
|
|
||||||
|
|
||||||
### 2.1 Migrate extraction to tool use
|
|
||||||
|
|
||||||
In [`extractionPipeline.js`](src/lib/extractionPipeline.js):
|
|
||||||
|
|
||||||
- Replace the "Return JSON only" instruction with a tool: `emit_knowledge_graph` whose `input_schema` mirrors `extractionResultSchema`.
|
|
||||||
- Replace `anthropicApi.generateContent(...)` with `callLLM({ ..., tools:[emitKnowledgeGraphTool], toolChoice:{ type:'tool', name:'emit_knowledge_graph' } })`.
|
|
||||||
- Read the validated object from `toolUses[0].input`.
|
|
||||||
- Same migration for `analyzeHandbookDelta` (tool `emit_handbook_delta`).
|
|
||||||
- Delete every `responseText.match(/\{[\s\S]*\}/)` site.
|
|
||||||
|
|
||||||
### 2.2 Migrate learning, quiz, custom-topic, graph-actions to tool use
|
|
||||||
|
|
||||||
Same pattern, in:
|
|
||||||
|
|
||||||
- [`learningService.js`](src/lib/learningService.js): tools `emit_learning_article`, `emit_learning_slides`, `emit_learning_infographic`, `emit_learning_all`, `emit_custom_topic`.
|
|
||||||
- [`testService.js`](src/lib/testService.js): tool `emit_quiz_questions`.
|
|
||||||
- [`KnowledgeGraph.jsx:297`](src/components/admin/KnowledgeGraph.jsx): tool `emit_graph_actions`.
|
|
||||||
|
|
||||||
### 2.3 Prompt caching
|
|
||||||
|
|
||||||
Pass `system` as an array of blocks so the stable parts can be cached:
|
|
||||||
|
|
||||||
```js
|
|
||||||
system: [
|
|
||||||
{ type:'text', text: STABLE_SYSTEM_HEADER, cache_control: { type:'ephemeral' } },
|
|
||||||
{ type:'text', text: dynamicPart }, // not cached
|
|
||||||
],
|
|
||||||
```
|
|
||||||
|
|
||||||
Apply caching to:
|
|
||||||
- Extraction `SYSTEM_PROMPT` and `HANDBOOK_SYSTEM_PROMPT` (both fully stable → cache the whole block).
|
|
||||||
- R42 system prompt — split into three blocks: stable preamble (cached), KB context (cached *only* while the graph hasn't changed; bust by appending a short hash of the topic IDs+labels — Phase 5 details), and per-turn role line (not cached).
|
|
||||||
|
|
||||||
### 2.4 Patch-based learning refinement
|
|
||||||
|
|
||||||
Refactor `refineLearningContent` ([`learningService.js:147`](src/lib/learningService.js:147)) from "return the full updated JSON" to **patch operations** via tools:
|
|
||||||
|
|
||||||
- `set_section(heading: string, body: string)` — replace one section by heading match.
|
|
||||||
- `add_section(heading: string, body: string, position: 'start'|'end')`.
|
|
||||||
- `remove_section(heading: string)`.
|
|
||||||
- `replace_takeaways(items: string[])`.
|
|
||||||
- `set_intro(intro: string)`.
|
|
||||||
|
|
||||||
Apply patches client-side to the cached object. Re-validate against `learningArticleSchema` after patching; reject the whole turn if invalid.
|
|
||||||
|
|
||||||
### Phase 2 acceptance criteria
|
|
||||||
|
|
||||||
- [ ] No regex JSON extraction left in `src/`: `grep -rn "match(/\\\\{\\[\\\\s\\\\S\\]\\*\\\\}/)" src/` returns nothing.
|
|
||||||
- [ ] Token usage telemetry shows `cache_read_input_tokens > 0` on the second extraction call within 5 minutes (cache hit).
|
|
||||||
- [ ] Re-running extraction on a known source produces the same topic count ±10% as before this phase.
|
|
||||||
- [ ] `refineLearningContent` round-trip ("make the intro shorter") produces only the changed section in the diff against the prior cached content.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 3 — Extraction quality
|
|
||||||
|
|
||||||
**Goal:** fewer near-duplicate topics, no silent truncation, adaptive throttling, unified vocabulary.
|
|
||||||
|
|
||||||
### 3.1 Sentence-aware chunking with overlap
|
|
||||||
|
|
||||||
Replace `chunkText` in [`extractionPipeline.js:87`](src/lib/extractionPipeline.js:87):
|
|
||||||
|
|
||||||
- Target **~2000 input tokens per chunk**. Approximate as `chars / 4`. Configurable via `MAX_CHUNK_CHARS = 8000`.
|
|
||||||
- **200-token overlap** between chunks (`OVERLAP_CHARS = 800`).
|
|
||||||
- Split on sentence boundaries (`/(?<=[.!?])\s+/`) first; fall back to paragraph boundary if a sentence is too long; never produce a chunk larger than `MAX_CHUNK_CHARS`.
|
|
||||||
- Add a guard: if a single sentence exceeds `MAX_CHUNK_CHARS`, hard-split at character boundary and log a warning.
|
|
||||||
|
|
||||||
### 3.2 Stateful extraction
|
|
||||||
|
|
||||||
Before each chunk after the first, prepend to the user message:
|
|
||||||
|
|
||||||
```text
|
|
||||||
Already-extracted topic IDs (do NOT create new IDs for these — reuse them if the same concept appears here):
|
|
||||||
- software-engineer
|
|
||||||
- onboarding-buddy
|
|
||||||
...
|
|
||||||
```
|
|
||||||
|
|
||||||
Cap the list at 200 IDs by recency to keep token cost bounded. The model will then reuse IDs instead of inventing variants like `software-developer`.
|
|
||||||
|
|
||||||
### 3.3 Adaptive throttling
|
|
||||||
|
|
||||||
Replace the hard `setTimeout(r, 12000)` in [`extractionPipeline.js:127`](src/lib/extractionPipeline.js:127) and the 15s sleep in [`KnowledgeGraph.jsx:274`](src/components/admin/KnowledgeGraph.jsx:274) with a shared token-bucket limiter in `src/lib/llmRetry.js`:
|
|
||||||
|
|
||||||
```js
|
|
||||||
export const extractionLimiter = createLimiter({ rps: 5/60, burst: 1 }); // 5 req/min
|
|
||||||
// usage: await extractionLimiter.acquire();
|
|
||||||
```
|
|
||||||
|
|
||||||
`callLLM` accepts an optional `limiter` param that it `await`s before fetch. On 429 with `Retry-After`, the limiter is paused for that duration.
|
|
||||||
|
|
||||||
### 3.4 Preserve admin-edited relevance
|
|
||||||
|
|
||||||
Add a migration introducing `relevance_locked: bool` on `topics`. Set it to `true` whenever an admin edits `learning_relevance` via the UI ([`KnowledgeGraph.jsx`](src/components/admin/KnowledgeGraph.jsx) edit handler — locate by searching for `setLearningRelevance` or the relevance form field).
|
|
||||||
|
|
||||||
In `mergeKnowledgeGraph` ([`extractionPipeline.js:167`](src/lib/extractionPipeline.js:167)), when `relevance_locked`, never overwrite `learning_relevance`.
|
|
||||||
|
|
||||||
### 3.5 Unify relation vocabulary
|
|
||||||
|
|
||||||
Pick **one** set: `related_to | depends_on | part_of | executed_by`. Migrate:
|
|
||||||
|
|
||||||
- `HANDBOOK_SYSTEM_PROMPT` ([`extractionPipeline.js:42`](src/lib/extractionPipeline.js:42)) — change `executes` to `executed_by` and swap the source/target in the prompt example.
|
|
||||||
- Write a one-shot migration script `pb_migrations/<timestamp>_normalize_relation_types.js` that rewrites any existing `executes` relation to `executed_by` and swaps `source ↔ target`.
|
|
||||||
- Verify R42's `validateDelta` ([`rag.js:108`](src/components/chat/rag.js:108)) already enforces this set (it does) — no change needed there.
|
|
||||||
|
|
||||||
### 3.6 Cancellation
|
|
||||||
|
|
||||||
Add a "Cancel" button to the source-processing UI in `ContentManager.jsx` / `UploadZone.jsx` (locate the one that displays extraction progress). Wire it to abort the in-flight `callLLM` via the `signal` it receives. On cancel, set source status to `cancelled` (add to status enum migration).
|
|
||||||
|
|
||||||
### Phase 3 acceptance criteria
|
|
||||||
|
|
||||||
- [ ] Running extraction twice on the same `sources/ROLES.md` produces zero new topics on the second run (idempotency through reused IDs).
|
|
||||||
- [ ] Locked-relevance topics survive re-extraction.
|
|
||||||
- [ ] No fixed `setTimeout` ≥ 5s anywhere in `src/` (`grep -rn "setTimeout" src/`).
|
|
||||||
- [ ] Cancelling an extraction mid-run leaves the source in `cancelled` state, not `processing`.
|
|
||||||
- [ ] `pb_migrations` includes the relation-vocabulary normalization and the `relevance_locked` column.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 4 — Quiz & content quality
|
|
||||||
|
|
||||||
**Goal:** quiz questions are positionally unbiased, deduped, and difficulty-tagged. Random helpers are correct.
|
|
||||||
|
|
||||||
### 4.1 Random helpers
|
|
||||||
|
|
||||||
Create `src/lib/random.js`:
|
|
||||||
|
|
||||||
```js
|
|
||||||
export function shuffle(arr) { /* Fisher–Yates, returns NEW array */ }
|
|
||||||
export function sample(arr, n) { /* unbiased sample without replacement */ }
|
|
||||||
export function pickInt(min, maxInclusive) { /* uniform integer */ }
|
|
||||||
```
|
|
||||||
|
|
||||||
Replace every `.sort(() => 0.5 - Math.random())` with `shuffle(arr)`:
|
|
||||||
|
|
||||||
- [`testService.js:122`](src/lib/testService.js:122) and [`testService.js:163`](src/lib/testService.js:163).
|
|
||||||
- Any other site found by `grep -rn "0.5 - Math.random()" src/`.
|
|
||||||
|
|
||||||
### 4.2 Debias `correctIndex` in quiz prompt
|
|
||||||
|
|
||||||
In [`testService.js:81`](src/lib/testService.js:81):
|
|
||||||
|
|
||||||
- Change the example in the prompt to use `"correctIndex": 2` (not 0).
|
|
||||||
- Add to the prompt: *"Distribute correctIndex roughly evenly across 0, 1, 2, and 3. Do not place the correct answer at the same position more than 4 out of 10 times."*
|
|
||||||
- After parsing, run a check: if more than 50% of the batch share the same `correctIndex`, log a warning and re-roll up to 2 times.
|
|
||||||
|
|
||||||
### 4.3 Difficulty field
|
|
||||||
|
|
||||||
- Add to `quizQuestionsSchema`: `difficulty: 'easy'|'medium'|'hard'`.
|
|
||||||
- Update the prompt to require difficulty on every question (current prompt says "4 easy, 4 medium, 2 hard" but never tagged — now tag it).
|
|
||||||
- Migration: add `difficulty` to the `quiz_banks.questions[]` element. PocketBase stores `questions` as JSON, so the migration is a no-op at the column level; older records get `difficulty: 'medium'` on read (add a normalizer in `db.js`).
|
|
||||||
|
|
||||||
### 4.4 Question dedup
|
|
||||||
|
|
||||||
In `forceGenerateTopicQuestions` ([`testService.js:65`](src/lib/testService.js:65)):
|
|
||||||
|
|
||||||
- Normalize question text (lowercase, strip punctuation, collapse whitespace) → `normKey`.
|
|
||||||
- Before persisting, drop any new question whose `normKey` matches an existing bank question.
|
|
||||||
- Log dropped duplicates with `console.debug('[quiz] dropped duplicate:', text)`.
|
|
||||||
|
|
||||||
### 4.5 Quality gate
|
|
||||||
|
|
||||||
In the same function, after schema validation:
|
|
||||||
|
|
||||||
- Reject the whole batch if any question has fewer than 4 distinct options.
|
|
||||||
- Reject if any option contains `"all of the above"`, `"none of the above"`, `"both A and B"` (case-insensitive).
|
|
||||||
- Reject if `explanation.trim().length < 20`.
|
|
||||||
- Surface the rejection to the admin UI with a "Retry" button.
|
|
||||||
|
|
||||||
### 4.6 Custom topic ID hygiene
|
|
||||||
|
|
||||||
In `generateCustomTopic` ([`learningService.js:177`](src/lib/learningService.js:177)):
|
|
||||||
|
|
||||||
- Generate kebab-case ID from the polished label, not `Date.now()`.
|
|
||||||
- Collision check against existing topics (append `-2`, `-3`, … if needed).
|
|
||||||
- Default `learning_relevance: 'standard'` when the model omits it.
|
|
||||||
|
|
||||||
### Phase 4 acceptance criteria
|
|
||||||
|
|
||||||
- [ ] No `.sort(() => 0.5 - Math.random())` anywhere in `src/`.
|
|
||||||
- [ ] Sample of 50 fresh quiz questions across 5 topics: no position holds >40% of correct answers.
|
|
||||||
- [ ] Re-running quiz generation for the same topic does not grow the bank with semantic duplicates.
|
|
||||||
- [ ] Custom topics created via R42 use kebab-case IDs and pass schema validation.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 5 — R42 retrieval & telemetry
|
|
||||||
|
|
||||||
**Goal:** R42 stops shipping the entire KG every turn; conversations are bounded; every call is logged.
|
|
||||||
|
|
||||||
### 5.1 TF-IDF retrieval in the browser
|
|
||||||
|
|
||||||
Create `src/lib/retrieval.js`:
|
|
||||||
|
|
||||||
```js
|
|
||||||
export function buildIndex(topics) { /* TF-IDF over label + description */ }
|
|
||||||
export function retrieveTopK(index, query, k = 10) { /* returns Topic[] */ }
|
|
||||||
```
|
|
||||||
|
|
||||||
Implementation: a small dependency-free TF-IDF — tokenize on `/[a-zA-Z0-9-]+/`, lowercase, drop stopwords (Dutch + English short list). Cache the index on the `topics` array reference. About 100 lines.
|
|
||||||
|
|
||||||
### 5.2 Rewrite `buildKbContext`
|
|
||||||
|
|
||||||
In [`rag.js:11`](src/components/chat/rag.js:11):
|
|
||||||
|
|
||||||
- Use `retrieveTopK(index, userMessage, 10)` to pick which topics go into the system prompt.
|
|
||||||
- Always include any topic whose ID or label is mentioned verbatim (existing behaviour).
|
|
||||||
- Drop the full "every topic" dump.
|
|
||||||
- Return `{ context, retrievedTopics, allTopics }` — `validateDelta` continues to use `allTopics`.
|
|
||||||
|
|
||||||
### 5.3 `lookup_topic` tool
|
|
||||||
|
|
||||||
Add a second R42 tool in [`prompts.js`](src/components/chat/prompts.js):
|
|
||||||
|
|
||||||
```js
|
|
||||||
export const LOOKUP_TOPIC_TOOL = {
|
|
||||||
name: 'lookup_topic',
|
|
||||||
description: 'Fetch the full description and any deeper learning content for a topic. Use when the retrieved context does not contain enough to answer.',
|
|
||||||
input_schema: { type:'object', properties: { id:{type:'string'} }, required:['id'] },
|
|
||||||
};
|
|
||||||
```
|
|
||||||
|
|
||||||
In `useChat.js`, when the model emits a `lookup_topic`, fetch via `db.getTopics()` + `db.getContent(id)` and append a `tool_result` block, then call `callLLM` again with the extended messages. Cap to **3 lookup hops per turn** to avoid loops.
|
|
||||||
|
|
||||||
### 5.4 R42 prompt cache busting
|
|
||||||
|
|
||||||
KB-context block is cached as ephemeral. The cache key is automatic per text, so any change busts it. **Append a stable suffix** to the KB block: `"\n[kb_hash: <first-8-chars-of-sha256-of-sorted-topic-ids>]"`. This guarantees the block content changes the moment a topic is added/removed, even if the included topic list looks similar.
|
|
||||||
|
|
||||||
### 5.5 Conversation truncation
|
|
||||||
|
|
||||||
In `useChat.js`:
|
|
||||||
|
|
||||||
- Keep last **12 turns** verbatim in `apiMessages`.
|
|
||||||
- Older messages: if more than 12 turns exist, summarize the older ones with a cheap (`tier: 'fast'`) call and prepend a single `{ role:'system'-equivalent inside the user history is not allowed by Anthropic — instead prepend a user/assistant pair }` block, OR simply drop older turns. **Default: drop older turns and prepend a one-line `assistant` message saying "(earlier conversation truncated)".** Summarization is an optional follow-up.
|
|
||||||
|
|
||||||
### 5.6 `llm_calls` collection
|
|
||||||
|
|
||||||
Add a migration `pb_migrations/<timestamp>_created_llm_calls.js` for collection `llm_calls`:
|
|
||||||
|
|
||||||
| Field | Type | Notes |
|
|
||||||
|---|---|---|
|
|
||||||
| `task` | text | e.g. `extract.source` |
|
|
||||||
| `model` | text | resolved model ID |
|
|
||||||
| `tier` | text | `fast` / `standard` / `reasoning` |
|
|
||||||
| `duration_ms` | number | |
|
|
||||||
| `input_tokens` | number | |
|
|
||||||
| `output_tokens` | number | |
|
|
||||||
| `cache_read_tokens` | number | |
|
|
||||||
| `cache_create_tokens` | number | |
|
|
||||||
| `stop_reason` | text | |
|
|
||||||
| `ok` | bool | |
|
|
||||||
| `error_msg` | text | nullable |
|
|
||||||
|
|
||||||
Wire `callLLM` to write to it (best-effort, never throws). Add a minimal `Admin → Diagnostics` view that shows the last 100 calls and aggregate cost (using public Anthropic prices in a constant; refresh manually).
|
|
||||||
|
|
||||||
### 5.7 R42 defaults
|
|
||||||
|
|
||||||
- `temperature: 0.3` for R42 chat in `callLLM`.
|
|
||||||
- `maxTokens: 2048` for R42 (text + tool budget).
|
|
||||||
|
|
||||||
### Phase 5 acceptance criteria
|
|
||||||
|
|
||||||
- [ ] R42 system prompt is ≤ 4000 tokens regardless of KG size (verify on a graph with 200+ topics).
|
|
||||||
- [ ] Adding a topic in the admin UI causes the **next** R42 call to show `cache_read_tokens === 0` for the KB block, then subsequent calls to show non-zero.
|
|
||||||
- [ ] R42 successfully answers a question about a topic whose ID was not in the user's message by emitting `lookup_topic` (manual verification).
|
|
||||||
- [ ] `Admin → Diagnostics` shows recent LLM calls with model, tokens, duration.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 6 — Eval harness (NOT IMPLEMENTED — out of scope)
|
|
||||||
|
|
||||||
> Phase 6 was deliberately skipped. The acceptance criteria for Phases 1–5 give enough confidence in extraction, content, quiz, R42, and telemetry that a golden-set harness is not currently load-bearing. Leave this section intact as a starting point for a future, separately-scoped initiative.
|
|
||||||
|
|
||||||
**Goal:** prompt or model changes can be measured before they ship.
|
|
||||||
|
|
||||||
### 6.1 Golden sets
|
|
||||||
|
|
||||||
Create `evals/` at the repo root:
|
|
||||||
|
|
||||||
```
|
|
||||||
evals/
|
|
||||||
extraction/
|
|
||||||
cases/
|
|
||||||
roles-handbook.txt
|
|
||||||
governance-handbook.txt
|
|
||||||
expected/
|
|
||||||
roles-handbook.json # { mustContain: ['software-engineer', ...], minTopics: 20 }
|
|
||||||
governance-handbook.json
|
|
||||||
quiz/
|
|
||||||
cases/
|
|
||||||
software-engineer.json # topic to generate quiz for
|
|
||||||
rubric.md # human-readable quality rubric
|
|
||||||
chat/
|
|
||||||
scripts/
|
|
||||||
ask-about-known-topic.json
|
|
||||||
ask-about-unknown-topic.json
|
|
||||||
propose-new-role.json
|
|
||||||
```
|
|
||||||
|
|
||||||
10 extraction cases, 5 quiz topics, 10 chat scripts is enough to start.
|
|
||||||
|
|
||||||
### 6.2 Runner
|
|
||||||
|
|
||||||
`evals/run.mjs` — Node script that:
|
|
||||||
|
|
||||||
1. Loads each case.
|
|
||||||
2. Invokes the same code path the app uses (import from `src/lib/*` directly; reuse the simulation toggle off).
|
|
||||||
3. Compares against expectations:
|
|
||||||
- **Extraction:** `mustContain` IDs present? `minTopics` met? No `stop_reason: 'max_tokens'`?
|
|
||||||
- **Quiz:** distribution of `correctIndex`, mean explanation length, banned phrases.
|
|
||||||
- **Chat:** for each script, did the expected tool fire? Did the reply contain expected anchors?
|
|
||||||
4. Writes `evals/results/<ISO-timestamp>.json` and a Markdown diff against the previous baseline.
|
|
||||||
|
|
||||||
Add `npm run eval` to `package.json`.
|
|
||||||
|
|
||||||
### 6.3 Prompt versioning
|
|
||||||
|
|
||||||
In each prompt module, export `PROMPT_VERSION = '2026-05-20-001'`. Persist it on the artifact (`content.data.prompt_version`, `quiz_banks.questions[i].prompt_version`, `topics.metadata.prompt_version`). Add an admin button "Mark stale content for regeneration" that lists artifacts whose version is older than the current.
|
|
||||||
|
|
||||||
### Phase 6 acceptance criteria
|
|
||||||
|
|
||||||
- [ ] `npm run eval` runs end-to-end against the live API and produces a result file.
|
|
||||||
- [ ] CI (or a manual check) runs evals on every change to `src/lib/llm.js`, prompts, or schemas.
|
|
||||||
- [ ] Each AI-generated artifact in PocketBase carries a `prompt_version` and is filterable by it.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Cross-phase verification checklist
|
|
||||||
|
|
||||||
After every phase, run this short checklist before merging:
|
|
||||||
|
|
||||||
1. **Build:** `npm run build` succeeds.
|
|
||||||
2. **Lint:** `npm run lint` clean.
|
|
||||||
3. **Tests:** `npm run test` green.
|
|
||||||
4. **Smoke flows** in dev (simulation off, real API key):
|
|
||||||
- Add a source via `Admin → Sources`, extract, verify topics appear.
|
|
||||||
- Visit `Leren` for the current week, generate article. Then slides.
|
|
||||||
- Visit `Testen`, generate weekly quiz. Submit. Score lands on leaderboard.
|
|
||||||
- Open R42, ask a known and an unknown question; propose a new topic; admin accepts it.
|
|
||||||
- Run `Admin → Knowledge Graph → Analyze & Optimize Graph`.
|
|
||||||
- Run `Admin → Knowledge Graph → Sync Handbook` (small repo or mock).
|
|
||||||
5. **Simulation toggle:** flip simulation mode on and confirm no real API calls happen (Network tab).
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Rollback strategy
|
|
||||||
|
|
||||||
Every phase is shippable on its own. If a phase introduces a regression:
|
|
||||||
|
|
||||||
- **Phase 1–2:** `git revert` the merge commit. `api.js` retains the legacy interface, so callers that haven't migrated still work.
|
|
||||||
- **Phase 3:** revert + the relation-vocabulary migration is reversible (rerun the reverse swap).
|
|
||||||
- **Phase 4:** revert; quiz schema additions are forward-compatible (older readers ignore `difficulty`).
|
|
||||||
- **Phase 5:** revert + drop the `llm_calls` collection if undesired.
|
|
||||||
- **Phase 6:** purely additive; remove the `evals/` folder if abandoned.
|
|
||||||
|
|
||||||
Never `git push --force` to `main`. PR-per-phase.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Out of scope (do not do as part of this plan)
|
|
||||||
|
|
||||||
- Replacing PocketBase. Stays as-is.
|
|
||||||
- Server-side embeddings or a vector store. Phase 5 deliberately uses in-browser TF-IDF.
|
|
||||||
- Streaming responses. Mentioned as a future improvement; not in this plan.
|
|
||||||
- Multi-tenant changes. The platform serves one company.
|
|
||||||
- UI redesign of the Admin pages beyond what each phase requires.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Glossary
|
|
||||||
|
|
||||||
- **Tier** — coarse model class (`fast`/`standard`/`reasoning`) mapped to a concrete Anthropic model ID via admin settings.
|
|
||||||
- **Tool use** — Anthropic's structured-output mechanism. The model emits a `tool_use` content block whose `input` is schema-valid JSON.
|
|
||||||
- **Prompt caching** — Anthropic feature where blocks marked `cache_control: { type:'ephemeral' }` are reused across requests at lower input cost. 5-minute TTL.
|
|
||||||
- **TF-IDF** — Term-frequency / inverse-document-frequency. A classic IR scoring function used here as a cheap retrieval signal.
|
|
||||||
- **KB context** — The block in R42's system prompt that lists topics and relations from the knowledge graph.
|
|
||||||
- **Delta** — A proposed addition to the knowledge graph emitted by R42 via the `propose_graph_delta` tool.
|
|
||||||
@@ -1,334 +0,0 @@
|
|||||||
# Curriculum specification
|
|
||||||
|
|
||||||
## Purpose
|
|
||||||
|
|
||||||
This document defines the functional behaviour of a perpetual AI-generated
|
|
||||||
learning curriculum. It is implementation-agnostic — no assumptions are made
|
|
||||||
about programming language, framework, database, or AI provider. Any system
|
|
||||||
that satisfies these functional requirements is a valid implementation.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Concepts
|
|
||||||
|
|
||||||
**Knowledge base (KB)**
|
|
||||||
The source of truth for all learning content. Organised as a two-level
|
|
||||||
hierarchy: Themes (broad subject areas) containing Topics (specific concepts).
|
|
||||||
Topics carry metadata that informs curriculum sequencing.
|
|
||||||
|
|
||||||
**Theme**
|
|
||||||
A broad subject area. One Theme = one weekly session. A Theme contains one
|
|
||||||
or more Topics.
|
|
||||||
|
|
||||||
**Topic**
|
|
||||||
An atomic unit of knowledge within a Theme. Carries:
|
|
||||||
- a complexity weight (1–5 scale, 1 = introductory, 5 = advanced)
|
|
||||||
- a difficulty label (introductory / intermediate / advanced)
|
|
||||||
- relationships to other topics: prerequisite, related, contrast
|
|
||||||
|
|
||||||
**Curriculum**
|
|
||||||
A versioned 26-week schedule mapping one Theme per week. The curriculum is
|
|
||||||
generated by AI from the KB, not manually authored. Admins review and
|
|
||||||
finetune it before it is applied.
|
|
||||||
|
|
||||||
**Curriculum version**
|
|
||||||
Each generation produces a new version. Only one version is active at a time.
|
|
||||||
Previous versions are retained for audit purposes.
|
|
||||||
|
|
||||||
**Cycle**
|
|
||||||
One full pass through the 26-week curriculum. After completing week 26,
|
|
||||||
an employee begins cycle 2. Cycles continue indefinitely.
|
|
||||||
|
|
||||||
**Employee curriculum state**
|
|
||||||
Tracks each employee's position: current cycle, current week, start date,
|
|
||||||
and which curriculum version they are on. Employees have independent start
|
|
||||||
dates — there are no cohorts.
|
|
||||||
|
|
||||||
**Completed week**
|
|
||||||
A week where the employee has recorded at least one session completion.
|
|
||||||
Completed weeks are immutable — no regeneration or version change may alter
|
|
||||||
a completed week's record.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Functional requirements
|
|
||||||
|
|
||||||
### 1. Curriculum generation
|
|
||||||
|
|
||||||
The system must be able to generate a 26-week curriculum from the current
|
|
||||||
published KB.
|
|
||||||
|
|
||||||
**Trigger conditions:**
|
|
||||||
- Manually by an admin
|
|
||||||
- Automatically when new Topics are published to the KB
|
|
||||||
|
|
||||||
**Inputs to the generator:**
|
|
||||||
- All published Themes, each with their ordered Topic list
|
|
||||||
- Per Topic: title, complexity weight, difficulty, prerequisite relationships
|
|
||||||
- If regenerating: reason for regeneration
|
|
||||||
|
|
||||||
**Generation process:**
|
|
||||||
|
|
||||||
Step 1 — Pre-process topic order within each Theme
|
|
||||||
Before any AI involvement, resolve the order of Topics within each Theme
|
|
||||||
using their prerequisite relationships. Build a directed graph of
|
|
||||||
prerequisite edges and apply a topological sort. The result is an ordered
|
|
||||||
topic list per Theme where prerequisites always precede dependent Topics.
|
|
||||||
If a cycle is detected in the graph (which should not occur in a well-formed
|
|
||||||
KB), fall back to ordering by complexity weight ascending.
|
|
||||||
|
|
||||||
This pre-processing offloads prerequisite reasoning from the AI — the AI
|
|
||||||
receives already-ordered topic lists and focuses only on Theme sequencing
|
|
||||||
across weeks.
|
|
||||||
|
|
||||||
Step 2 — AI sequences Themes across 26 weeks
|
|
||||||
The AI receives the pre-processed KB snapshot and produces a 26-week
|
|
||||||
schedule. The AI must follow these sequencing rules:
|
|
||||||
|
|
||||||
- Every Theme must appear at least once across the 26 weeks
|
|
||||||
- Themes with more Topics may span multiple weeks or recur within the 26 weeks
|
|
||||||
- Foundational Themes precede dependent ones
|
|
||||||
- Complexity increases progressively: introductory Themes in the first half,
|
|
||||||
advanced Themes in the second half
|
|
||||||
- If the KB contains more Topics than 26 weeks can cover in depth, prioritise
|
|
||||||
breadth in cycle 1 — every Theme covered, key Topics per Theme
|
|
||||||
- Each week slot has an estimated session duration (15–45 minutes)
|
|
||||||
- Each week slot includes a one-sentence rationale explaining its position
|
|
||||||
|
|
||||||
**Output:**
|
|
||||||
A draft 26-week schedule with one Theme and an ordered Topic subset per week,
|
|
||||||
estimated duration per week, and a rationale per week.
|
|
||||||
|
|
||||||
**Validation:**
|
|
||||||
All Theme and Topic references in the generated schedule must resolve to
|
|
||||||
existing published KB entries. Any reference that does not resolve must
|
|
||||||
cause the generation to be retried or rejected — never persisted.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### 2. Admin review and confirmation
|
|
||||||
|
|
||||||
A generated curriculum is always a draft first. It is not applied to any
|
|
||||||
employee until an admin explicitly confirms it.
|
|
||||||
|
|
||||||
**Admin capabilities before confirmation:**
|
|
||||||
- Preview the full proposed 26-week schedule
|
|
||||||
- View the rationale for each week's Theme assignment
|
|
||||||
- Reorder weeks (swap Theme assignments between weeks)
|
|
||||||
- Add or remove Topics from a week's Topic subset
|
|
||||||
- Annotate any week with free-text notes
|
|
||||||
- Compare the proposed schedule against the currently active version
|
|
||||||
|
|
||||||
**Confirmation:**
|
|
||||||
When admin confirms, the new version becomes active and is applied to all
|
|
||||||
employees whose current week falls within the regenerated range (see
|
|
||||||
section 4 — versioning).
|
|
||||||
|
|
||||||
**Rejection:**
|
|
||||||
If admin rejects the draft, it is discarded and no changes are applied.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### 3. Perpetual cycling
|
|
||||||
|
|
||||||
The curriculum runs indefinitely. After an employee completes week 26, cycle 2
|
|
||||||
begins automatically on the latest active curriculum version.
|
|
||||||
|
|
||||||
**Cycle 2 and beyond are not identical to cycle 1.**
|
|
||||||
The generator must vary subsequent cycles to reinforce rather than repeat.
|
|
||||||
When generating a cycle 2+ schedule for an employee, the generator receives
|
|
||||||
additional input:
|
|
||||||
|
|
||||||
- Which micro learning types the employee has used across their history
|
|
||||||
- Which micro learning types the employee has not yet used
|
|
||||||
- Which Topics the employee engaged with minimally (completed only the
|
|
||||||
minimum required interaction)
|
|
||||||
|
|
||||||
The generator uses this input to:
|
|
||||||
- Vary the Theme sequence from the previous cycle
|
|
||||||
- Surface underused micro learning types as recommended formats
|
|
||||||
- Schedule low-engagement Topics earlier in the new cycle to give them
|
|
||||||
more visibility
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### 4. Versioning and version transitions
|
|
||||||
|
|
||||||
**One active version at all times.**
|
|
||||||
Previous versions are archived, not deleted.
|
|
||||||
|
|
||||||
**Completed weeks are frozen.**
|
|
||||||
When a new curriculum version is applied, only future unstarted weeks are
|
|
||||||
updated. Any week the employee has already started or completed retains its
|
|
||||||
original content. The boundary is the employee's current week at the moment
|
|
||||||
the new version is applied.
|
|
||||||
|
|
||||||
**Version transition logic:**
|
|
||||||
When a new version is confirmed:
|
|
||||||
- For each active employee: identify their current week N
|
|
||||||
- Weeks 1 through N-1 are already completed — untouched
|
|
||||||
- Week N onward is replaced with the new version's schedule
|
|
||||||
- The employee continues from week N on the new version seamlessly
|
|
||||||
|
|
||||||
**Regeneration triggers:**
|
|
||||||
Two events should prompt a regeneration:
|
|
||||||
1. New Topics are published to the KB (new content available for scheduling)
|
|
||||||
2. Admin manually requests regeneration
|
|
||||||
|
|
||||||
In both cases, the new version must be previewed and confirmed by an admin
|
|
||||||
before being applied. Regeneration is never automatic.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### 5. Employee curriculum state
|
|
||||||
|
|
||||||
Each employee has an independent curriculum state:
|
|
||||||
|
|
||||||
| Attribute | Description |
|
|
||||||
|---|---|
|
|
||||||
| Current cycle | Which pass through the 26-week schedule (starts at 1) |
|
|
||||||
| Current week | Which week within the cycle (1–26) |
|
|
||||||
| Start date | When this employee began their curriculum (rolling) |
|
|
||||||
| Active version | Which curriculum version the employee's future weeks draw from |
|
|
||||||
|
|
||||||
**State transitions:**
|
|
||||||
- Week completion: current week increments by 1
|
|
||||||
- Week 26 completion: cycle increments by 1, current week resets to 1,
|
|
||||||
active version updates to latest, cycle variant generated
|
|
||||||
- Version applied: active version updates for weeks not yet started
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### 6. Coverage guarantee
|
|
||||||
|
|
||||||
The curriculum must cover the entire published KB within one cycle.
|
|
||||||
Every published Theme must appear at least once in the 26-week schedule.
|
|
||||||
Coverage is verified after generation and before the draft is made available
|
|
||||||
for admin review. A schedule that fails coverage verification is rejected
|
|
||||||
and regenerated.
|
|
||||||
|
|
||||||
Coverage statistics must be surfaced to the admin during preview:
|
|
||||||
- Total Themes in KB vs Themes scheduled
|
|
||||||
- Total Topics in KB vs Topics scheduled across all weeks
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### 7. Estimated session duration
|
|
||||||
|
|
||||||
Each week slot carries an estimated duration in minutes (15–45 minute range).
|
|
||||||
This is an AI estimate based on Topic count and complexity weights for that
|
|
||||||
week. It is informational — it guides employees on time commitment but does
|
|
||||||
not enforce a time limit.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Data model (logical)
|
|
||||||
|
|
||||||
These are logical entities. Implementation may map them to any storage system.
|
|
||||||
|
|
||||||
**CurriculumVersion**
|
|
||||||
```
|
|
||||||
id
|
|
||||||
version_number integer, increments on each generation
|
|
||||||
status draft | active | superseded
|
|
||||||
generated_at datetime
|
|
||||||
generation_reason string
|
|
||||||
confirmed_by user reference
|
|
||||||
confirmed_at datetime
|
|
||||||
```
|
|
||||||
|
|
||||||
**CurriculumWeek**
|
|
||||||
```
|
|
||||||
id
|
|
||||||
curriculum_version → CurriculumVersion
|
|
||||||
week_number 1–26
|
|
||||||
theme → Theme
|
|
||||||
topics ordered list of → Topic
|
|
||||||
estimated_duration minutes (15–45)
|
|
||||||
week_rationale string (AI-generated, one sentence)
|
|
||||||
admin_notes string (free text, admin-authored)
|
|
||||||
```
|
|
||||||
|
|
||||||
**EmployeeCurriculumState**
|
|
||||||
```
|
|
||||||
id
|
|
||||||
employee → User
|
|
||||||
current_cycle integer (starts at 1)
|
|
||||||
current_week integer (1–26)
|
|
||||||
start_date datetime
|
|
||||||
active_version → CurriculumVersion
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## AI prompt requirements
|
|
||||||
|
|
||||||
The following requirements apply to any AI model used for generation,
|
|
||||||
regardless of provider.
|
|
||||||
|
|
||||||
**Input the AI must receive:**
|
|
||||||
- Full KB snapshot with Themes and pre-ordered Topic lists
|
|
||||||
- Per Topic: complexity weight, difficulty, relationship types
|
|
||||||
- Cycle number (1 for first cycle, 2+ for subsequent)
|
|
||||||
- For cycle 2+: employee history (types used, types not used, low-engagement topics)
|
|
||||||
|
|
||||||
**Output the AI must produce:**
|
|
||||||
- Exactly 26 week slots
|
|
||||||
- Each slot: theme reference, ordered topic references, estimated duration, rationale
|
|
||||||
- Structured format suitable for machine parsing (JSON or equivalent)
|
|
||||||
- No explanatory prose outside the structured output
|
|
||||||
|
|
||||||
**Constraints on AI behaviour:**
|
|
||||||
- Temperature should be set to deterministic or near-deterministic (0 or equivalent)
|
|
||||||
- Output must be validated before persistence — never persist unvalidated AI output
|
|
||||||
- On validation failure: retry once with a stricter prompt, then surface an error
|
|
||||||
to the operator rather than persisting a partial or malformed schedule
|
|
||||||
- The AI must not invent Theme or Topic references — all references must exist
|
|
||||||
in the provided KB snapshot
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Admin capabilities summary
|
|
||||||
|
|
||||||
| Capability | When available |
|
|
||||||
|---|---|
|
|
||||||
| Trigger manual regeneration | Any time |
|
|
||||||
| Preview proposed schedule | After generation, before confirmation |
|
|
||||||
| View week rationale | During preview |
|
|
||||||
| Reorder weeks | During preview |
|
|
||||||
| Edit topic subset per week | During preview |
|
|
||||||
| Add admin notes per week | During preview and after confirmation |
|
|
||||||
| Confirm version | During preview |
|
|
||||||
| Reject draft | During preview |
|
|
||||||
| View coverage statistics | During preview |
|
|
||||||
| View active schedule | Any time |
|
|
||||||
| View version history | Any time |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Behaviours that must never occur
|
|
||||||
|
|
||||||
- A completed week's content changes after the employee has completed it
|
|
||||||
- A curriculum version is applied without admin confirmation
|
|
||||||
- AI output is persisted without validation against the KB
|
|
||||||
- An employee's cycle increments without week 26 being completed
|
|
||||||
- Two versions are active simultaneously
|
|
||||||
- A Theme is absent from a 26-week schedule when it exists in the published KB
|
|
||||||
- The admin is not shown a preview before a new version is applied
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Acceptance criteria
|
|
||||||
|
|
||||||
1. Generate a curriculum from a KB with 5 or more Themes → exactly 26 weeks produced
|
|
||||||
2. All published Themes appear at least once in the schedule
|
|
||||||
3. Topic order within each week respects prerequisite relationships
|
|
||||||
4. Coverage statistics correctly reflect KB vs schedule coverage
|
|
||||||
5. A draft version is available for preview before any employee is affected
|
|
||||||
6. Admin can reorder two weeks and confirm → reordered schedule applied
|
|
||||||
7. New Topics published to KB → regeneration triggered → admin sees preview →
|
|
||||||
confirms → employees at week N see weeks 1 to N unchanged, weeks N+1 onward updated
|
|
||||||
8. Employee completes week 26 → cycle 2 begins → sequence differs from cycle 1
|
|
||||||
9. Cycle 2 schedule surfaces Topics the employee engaged with minimally
|
|
||||||
10. Two employees with different start dates and current weeks both receive
|
|
||||||
correct version transitions independently when a new version is applied
|
|
||||||
556
micro-learning-generation-spec.md
Normal file
556
micro-learning-generation-spec.md
Normal file
@@ -0,0 +1,556 @@
|
|||||||
|
# Micro learning generation specification
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
This document defines the functional behaviour of AI-generated micro learning
|
||||||
|
content. It is implementation-agnostic — no assumptions are made about
|
||||||
|
programming language, framework, database, or AI provider. Any system that
|
||||||
|
satisfies these functional requirements is a valid implementation.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Concepts
|
||||||
|
|
||||||
|
**Topic**
|
||||||
|
An atomic unit of knowledge. The source material for all micro learning
|
||||||
|
generation. A Topic has a title, a body (one or more paragraphs of content),
|
||||||
|
a difficulty level, and a set of key terms. Topics live within a knowledge
|
||||||
|
base organised into broader Themes.
|
||||||
|
|
||||||
|
**Micro learning**
|
||||||
|
A single generated learning artifact derived from one Topic. One micro
|
||||||
|
learning covers exactly one Topic in exactly one format type. Multiple micro
|
||||||
|
learnings can exist per Topic — one per type.
|
||||||
|
|
||||||
|
**Micro learning type**
|
||||||
|
The format and cognitive approach of the artifact. Ten types are defined.
|
||||||
|
Each type targets a different learning mechanism (recall, application,
|
||||||
|
comparison, reflection, and so on). See section — the ten types.
|
||||||
|
|
||||||
|
**Generation**
|
||||||
|
The act of producing micro learning content from a Topic using an AI model.
|
||||||
|
Generation always targets a specific type and produces structured output
|
||||||
|
matching that type's schema.
|
||||||
|
|
||||||
|
**Status lifecycle**
|
||||||
|
Every micro learning passes through a defined status sequence:
|
||||||
|
```
|
||||||
|
queued → generated → published
|
||||||
|
↘ rejected
|
||||||
|
```
|
||||||
|
|
||||||
|
- queued: generation has been requested but not yet executed
|
||||||
|
- generated: AI has produced output, awaiting review
|
||||||
|
- published: content is available to employees
|
||||||
|
- rejected: content was reviewed and discarded; generation can be retriggered
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Functional requirements
|
||||||
|
|
||||||
|
### 1. Generation triggers
|
||||||
|
|
||||||
|
Generation is triggered in two situations:
|
||||||
|
|
||||||
|
**Batch trigger — Theme approval**
|
||||||
|
When an admin approves a batch of Topics (grouped under a Theme), generation
|
||||||
|
is queued for all 10 types for every Topic in that batch. This is the primary
|
||||||
|
trigger. An approved Topic that has not had all 10 types generated is
|
||||||
|
considered incomplete.
|
||||||
|
|
||||||
|
**Individual trigger — manual regeneration**
|
||||||
|
An admin may request regeneration of a specific type for a specific Topic at
|
||||||
|
any time. This replaces the existing generated artifact for that type with a
|
||||||
|
newly generated one. The replacement does not affect other types for the same
|
||||||
|
Topic.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 2. One artifact per topic per type
|
||||||
|
|
||||||
|
The system maintains at most one published artifact per Topic per type at any
|
||||||
|
time. A Topic with all 10 types published has exactly 10 micro learning records.
|
||||||
|
|
||||||
|
If a type is rejected and regenerated, the previous artifact is superseded by
|
||||||
|
the new one. Rejected artifacts may be retained for audit but are never shown
|
||||||
|
to employees.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 3. Generation is asynchronous
|
||||||
|
|
||||||
|
Generation must not block the user interface. Each generation request is
|
||||||
|
queued and processed in the background. The requester receives a job reference
|
||||||
|
and can poll or subscribe to status updates.
|
||||||
|
|
||||||
|
Status must be reportable at two levels:
|
||||||
|
- Per Topic: how many of the 10 types are in each status
|
||||||
|
- Per job: current processing state of the batch
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 4. Content quality constraints
|
||||||
|
|
||||||
|
The AI must follow these constraints for all types:
|
||||||
|
|
||||||
|
- Content must be derived from the Topic body — the AI must not introduce
|
||||||
|
facts, claims, or examples that are not grounded in the provided Topic
|
||||||
|
- Language must match the Topic's difficulty level:
|
||||||
|
- introductory: plain language, no assumed prior knowledge
|
||||||
|
- intermediate: assumes familiarity with basic concepts in the domain
|
||||||
|
- advanced: technical precision, assumes domain competence
|
||||||
|
- Length must be appropriate to the type — see per-type requirements below
|
||||||
|
- Content must be written for the employee audience, not for a general public
|
||||||
|
- Tone must be direct and factual — no motivational filler, no excessive hedging
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 5. Structured output
|
||||||
|
|
||||||
|
Every type produces structured output — not free prose. The structure is
|
||||||
|
defined per type in the content schemas section. The AI must produce output
|
||||||
|
that strictly conforms to the schema for the requested type.
|
||||||
|
|
||||||
|
Output validation must occur before persistence. Any output that fails
|
||||||
|
schema validation must be:
|
||||||
|
1. Retried once with a stricter prompt
|
||||||
|
2. If retry also fails: marked as failed with a reason, not persisted
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 6. Admin review flow
|
||||||
|
|
||||||
|
Generated content enters a review queue before becoming available to
|
||||||
|
employees. Admins may:
|
||||||
|
|
||||||
|
- Read the generated content
|
||||||
|
- Approve it → status moves to published
|
||||||
|
- Reject it → status moves to rejected, regeneration can be requested
|
||||||
|
- Edit it inline → edits are saved, status moves to published
|
||||||
|
|
||||||
|
Bulk approval is permitted: an admin may approve all generated artifacts for
|
||||||
|
a Theme in a single action. Individual review remains available for any
|
||||||
|
artifact.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 7. Employee access
|
||||||
|
|
||||||
|
Employees see only published micro learnings. The set of available types for
|
||||||
|
a Topic is the set of types with published status. If a Topic has 7 of 10
|
||||||
|
types published, employees see 7 options.
|
||||||
|
|
||||||
|
Employees choose one type per Topic per session. Multiple types can be
|
||||||
|
completed in a single session. Completion of one type does not prevent
|
||||||
|
completing another type for the same Topic.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The ten types
|
||||||
|
|
||||||
|
Each type is described with its learning mechanism, required content
|
||||||
|
structure, constraints, and the AI prompt intent.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 1. Concept explainer
|
||||||
|
|
||||||
|
**Learning mechanism:** comprehension — understanding what something is
|
||||||
|
|
||||||
|
**Structure:**
|
||||||
|
```
|
||||||
|
paragraphs: array of 2–3 strings
|
||||||
|
example: string
|
||||||
|
```
|
||||||
|
|
||||||
|
**Constraints:**
|
||||||
|
- Paragraphs explain the concept in plain terms, building from definition
|
||||||
|
to context to implication
|
||||||
|
- Example is concrete and specific — not hypothetical ("for example, when
|
||||||
|
a circle needs to..."), not generic ("this is used in many situations")
|
||||||
|
- Total length: 150–250 words across paragraphs + example
|
||||||
|
- No bullet points — flowing prose only
|
||||||
|
|
||||||
|
**Prompt intent:** explain this concept clearly to someone encountering it
|
||||||
|
for the first time, then make it concrete with a real example from the
|
||||||
|
Topic's domain
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 2. Scenario quiz
|
||||||
|
|
||||||
|
**Learning mechanism:** application — using knowledge to reason through a
|
||||||
|
realistic situation
|
||||||
|
|
||||||
|
**Structure:**
|
||||||
|
```
|
||||||
|
scenario: string
|
||||||
|
options: array of 3–4 objects
|
||||||
|
label: string (A, B, C, D)
|
||||||
|
text: string
|
||||||
|
correct: boolean
|
||||||
|
explanation: string
|
||||||
|
```
|
||||||
|
|
||||||
|
**Constraints:**
|
||||||
|
- Scenario is a realistic, specific situation — not abstract ("imagine a
|
||||||
|
team..."), not trivial ("what is the definition of...")
|
||||||
|
- Exactly one correct option
|
||||||
|
- All incorrect options must be plausible — they should represent common
|
||||||
|
mistakes or reasonable misunderstandings, not obvious wrong answers
|
||||||
|
- Each explanation justifies why the option is correct or incorrect —
|
||||||
|
the explanation teaches, not just confirms
|
||||||
|
- Scenario length: 60–100 words
|
||||||
|
- Option text: 15–30 words each
|
||||||
|
|
||||||
|
**Prompt intent:** create a situation where someone must apply this Topic's
|
||||||
|
knowledge to make a decision, then explain the reasoning behind each choice
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 3. Common misconceptions
|
||||||
|
|
||||||
|
**Learning mechanism:** correction — replacing wrong mental models with
|
||||||
|
accurate ones
|
||||||
|
|
||||||
|
**Structure:**
|
||||||
|
```
|
||||||
|
items: array of 3–5 objects
|
||||||
|
misconception: string
|
||||||
|
correction: string
|
||||||
|
```
|
||||||
|
|
||||||
|
**Constraints:**
|
||||||
|
- Each misconception must be genuinely held by people learning this Topic —
|
||||||
|
not a strawman, not an obvious error
|
||||||
|
- Correction explains why the misconception is wrong and what is true
|
||||||
|
instead — it does not just negate the misconception
|
||||||
|
- Misconception: 1 sentence
|
||||||
|
- Correction: 2–3 sentences
|
||||||
|
- Items must be distinct — no overlapping misconceptions
|
||||||
|
|
||||||
|
**Prompt intent:** identify the most common wrong beliefs people hold about
|
||||||
|
this Topic and correct each one with an accurate explanation
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 4. Step-by-step how-to
|
||||||
|
|
||||||
|
**Learning mechanism:** procedural — learning a process by following steps
|
||||||
|
|
||||||
|
**Structure:**
|
||||||
|
```
|
||||||
|
steps: array of objects
|
||||||
|
number: integer
|
||||||
|
instruction: string
|
||||||
|
```
|
||||||
|
|
||||||
|
**Constraints:**
|
||||||
|
- Each step is a single action — not a compound instruction ("do A and
|
||||||
|
then B" should be two steps)
|
||||||
|
- Steps are ordered — the sequence matters and must be correct
|
||||||
|
- Step count: 4–8 steps
|
||||||
|
- Instruction length: 1–2 sentences per step
|
||||||
|
- Only applicable to Topics that describe a process or procedure — if the
|
||||||
|
Topic does not contain a process, this type must not be generated
|
||||||
|
|
||||||
|
**Prompt intent:** decompose the process described in this Topic into a
|
||||||
|
clear ordered sequence of actions that someone can follow
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 5. Comparison card
|
||||||
|
|
||||||
|
**Learning mechanism:** differentiation — understanding how two related
|
||||||
|
concepts differ and when each applies
|
||||||
|
|
||||||
|
**Structure:**
|
||||||
|
```
|
||||||
|
subject_a: string
|
||||||
|
subject_b: string
|
||||||
|
dimensions: array of 4–6 objects
|
||||||
|
label: string
|
||||||
|
a: string
|
||||||
|
b: string
|
||||||
|
```
|
||||||
|
|
||||||
|
**Constraints:**
|
||||||
|
- subject_a and subject_b must both be present in the Topic — do not
|
||||||
|
compare a Topic concept against an external concept not covered in
|
||||||
|
the KB
|
||||||
|
- Each dimension is a meaningful axis of comparison — not trivial
|
||||||
|
("name: A is called X, B is called Y")
|
||||||
|
- Each cell (a and b per dimension) is 1–2 sentences
|
||||||
|
- Dimensions must cover: purpose/intent, typical use, key difference,
|
||||||
|
and at least one practical implication
|
||||||
|
- Only applicable to Topics that discuss two comparable concepts
|
||||||
|
|
||||||
|
**Prompt intent:** surface the meaningful differences between the two
|
||||||
|
concepts in this Topic across dimensions that matter for applying them
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 6. Reflection prompt
|
||||||
|
|
||||||
|
**Learning mechanism:** metacognition — connecting new knowledge to
|
||||||
|
existing experience and internalising it
|
||||||
|
|
||||||
|
**Structure:**
|
||||||
|
```
|
||||||
|
prompt: string
|
||||||
|
model_answer: string
|
||||||
|
```
|
||||||
|
|
||||||
|
**Constraints:**
|
||||||
|
- Prompt is an open question that cannot be answered with a fact — it
|
||||||
|
requires the employee to connect the Topic to their own work or experience
|
||||||
|
- Prompt must be specific to the Topic's content — not generic
|
||||||
|
("how does this apply to your work?")
|
||||||
|
- Model answer demonstrates what a strong reflective response looks like —
|
||||||
|
it is not the only correct answer, but it shows the depth expected
|
||||||
|
- Prompt length: 1–2 sentences
|
||||||
|
- Model answer length: 100–150 words
|
||||||
|
|
||||||
|
**Prompt intent:** write a question that prompts the employee to connect
|
||||||
|
this Topic's content to their own professional context, then provide an
|
||||||
|
example of a thoughtful response
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 7. Spaced repetition flashcard set
|
||||||
|
|
||||||
|
**Learning mechanism:** recall — retrieving knowledge from memory through
|
||||||
|
repeated low-stakes testing
|
||||||
|
|
||||||
|
**Structure:**
|
||||||
|
```
|
||||||
|
cards: array of 5–10 objects
|
||||||
|
question: string
|
||||||
|
answer: string
|
||||||
|
```
|
||||||
|
|
||||||
|
**Constraints:**
|
||||||
|
- Each card tests one discrete fact, term, or concept from the Topic
|
||||||
|
- Questions must be answerable with the Topic's content alone — no
|
||||||
|
external knowledge required
|
||||||
|
- Answers are concise — 1–2 sentences maximum
|
||||||
|
- Questions must not overlap — each card tests something distinct
|
||||||
|
- Mix of question types across the set: definition questions, application
|
||||||
|
questions, and relationship questions (how does X relate to Y)
|
||||||
|
- No trick questions
|
||||||
|
|
||||||
|
**Prompt intent:** identify the most important facts, terms, and
|
||||||
|
relationships in this Topic and create one card per item, written for
|
||||||
|
repeated low-stakes recall practice
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 8. Case study mini-analysis
|
||||||
|
|
||||||
|
**Learning mechanism:** analytical — applying concepts to evaluate a
|
||||||
|
realistic scenario in depth
|
||||||
|
|
||||||
|
**Structure:**
|
||||||
|
```
|
||||||
|
scenario: string
|
||||||
|
questions: array of 2–4 strings
|
||||||
|
```
|
||||||
|
|
||||||
|
**Constraints:**
|
||||||
|
- Scenario is a fictional but realistic situation (150–200 words) that
|
||||||
|
directly involves the concepts in the Topic
|
||||||
|
- Scenario must be ambiguous enough to require analysis — not a situation
|
||||||
|
with an obvious single correct answer
|
||||||
|
- Questions guide analysis — they do not ask for definitions or facts,
|
||||||
|
they ask the employee to evaluate, judge, or decide
|
||||||
|
- Questions build on each other — the set forms a coherent analytical
|
||||||
|
arc, not isolated questions
|
||||||
|
|
||||||
|
**Prompt intent:** create a realistic situation that puts the Topic's
|
||||||
|
concepts in tension or under pressure, then ask questions that require
|
||||||
|
the employee to think critically rather than recall facts
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 9. Glossary anchor
|
||||||
|
|
||||||
|
**Learning mechanism:** vocabulary — precise understanding of domain terms
|
||||||
|
including how they are correctly and incorrectly applied
|
||||||
|
|
||||||
|
**Structure:**
|
||||||
|
```
|
||||||
|
term: string
|
||||||
|
definition: string
|
||||||
|
correct_use: string
|
||||||
|
misuse: string
|
||||||
|
```
|
||||||
|
|
||||||
|
**Constraints:**
|
||||||
|
- Term must be a key term from the Topic — taken from the Topic's key
|
||||||
|
terms list if available, otherwise identified from the Topic body
|
||||||
|
- Definition is precise — suitable for a reference glossary, not
|
||||||
|
conversational
|
||||||
|
- correct_use is a concrete sentence showing the term used correctly
|
||||||
|
in context
|
||||||
|
- misuse is a concrete sentence showing a common way the term is used
|
||||||
|
incorrectly, followed by a brief note on why it is wrong
|
||||||
|
- Definition: 2–3 sentences
|
||||||
|
- correct_use and misuse: 1–2 sentences each
|
||||||
|
|
||||||
|
**Prompt intent:** define this term precisely, then anchor the definition
|
||||||
|
with a correct example and a counterexample that shows a common misuse
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 10. Myth vs. evidence
|
||||||
|
|
||||||
|
**Learning mechanism:** critical thinking — evaluating a commonly held
|
||||||
|
belief against evidence
|
||||||
|
|
||||||
|
**Structure:**
|
||||||
|
```
|
||||||
|
myth: string
|
||||||
|
evidence: string
|
||||||
|
sources: array of strings (may be empty if no external sources in Topic)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Constraints:**
|
||||||
|
- Myth is a specific, commonly held false claim related to the Topic —
|
||||||
|
stated as a confident assertion, not a question
|
||||||
|
- Evidence is the factual rebuttal — it explains what is actually true
|
||||||
|
and why the myth persists
|
||||||
|
- Evidence does not simply negate the myth — it provides the accurate
|
||||||
|
alternative and ideally explains the origin of the misconception
|
||||||
|
- Myth: 1–2 sentences, stated assertively
|
||||||
|
- Evidence: 3–5 sentences
|
||||||
|
- Sources: only include sources that are explicitly referenced in the
|
||||||
|
Topic body — do not fabricate citations
|
||||||
|
|
||||||
|
**Prompt intent:** identify a specific false claim that people commonly
|
||||||
|
believe about this Topic's subject, state it directly, then dismantle it
|
||||||
|
with evidence and explain why the misconception exists
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## AI prompt requirements
|
||||||
|
|
||||||
|
The following requirements apply regardless of AI model or provider.
|
||||||
|
|
||||||
|
**Input the AI must receive for every generation call:**
|
||||||
|
- Topic title
|
||||||
|
- Topic body (full text)
|
||||||
|
- Topic difficulty level (introductory / intermediate / advanced)
|
||||||
|
- Topic key terms (list)
|
||||||
|
- The specific micro learning type being generated
|
||||||
|
- The content schema for that type (what structured output is expected)
|
||||||
|
|
||||||
|
**Output the AI must produce:**
|
||||||
|
- Structured output strictly conforming to the type's schema
|
||||||
|
- No preamble, no explanation, no prose outside the schema
|
||||||
|
- All content grounded in the provided Topic body — no external facts introduced
|
||||||
|
|
||||||
|
**Constraints on AI behaviour:**
|
||||||
|
- Output must be validated against the type's schema before persistence
|
||||||
|
- On validation failure: retry once with a stricter prompt, then surface
|
||||||
|
a failure status rather than persisting invalid output
|
||||||
|
- The AI must not add fields not present in the schema
|
||||||
|
- The AI must not omit required fields
|
||||||
|
- For types with constraints on count (flashcard_set: 5–10 cards,
|
||||||
|
scenario_quiz: 3–4 options), the count must be within the specified range
|
||||||
|
|
||||||
|
**One generation call per type:**
|
||||||
|
Each type requires a separate AI call. Do not attempt to generate multiple
|
||||||
|
types in a single call. This ensures clean validation and clean failure
|
||||||
|
isolation — a failure in one type does not affect others.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Data model (logical)
|
||||||
|
|
||||||
|
These are logical entities. Implementation may map them to any storage system.
|
||||||
|
|
||||||
|
**MicroLearning**
|
||||||
|
```
|
||||||
|
id
|
||||||
|
topic → Topic
|
||||||
|
type one of the ten type identifiers
|
||||||
|
content structured data matching the type's schema
|
||||||
|
status queued | generated | published | rejected
|
||||||
|
generation_model identifier of the model used (for audit)
|
||||||
|
generated_at datetime
|
||||||
|
published_at datetime (null until published)
|
||||||
|
updated_at datetime
|
||||||
|
```
|
||||||
|
|
||||||
|
**GenerationJob**
|
||||||
|
```
|
||||||
|
id
|
||||||
|
topic → Topic
|
||||||
|
types_requested list of type identifiers
|
||||||
|
status queued | processing | done | failed
|
||||||
|
progress
|
||||||
|
types_total integer
|
||||||
|
types_done integer
|
||||||
|
types_failed integer
|
||||||
|
error string (null unless failed)
|
||||||
|
created_at datetime
|
||||||
|
completed_at datetime (null until done or failed)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Admin capabilities summary
|
||||||
|
|
||||||
|
| Capability | When available |
|
||||||
|
|---|---|
|
||||||
|
| Trigger batch generation for a Theme | After Theme approval |
|
||||||
|
| Trigger individual type regeneration | Any time, per Topic per type |
|
||||||
|
| View generation job status | During and after generation |
|
||||||
|
| Review generated content | After generation |
|
||||||
|
| Approve individual artifact | After generation |
|
||||||
|
| Reject individual artifact | After generation |
|
||||||
|
| Edit content inline before publishing | After generation |
|
||||||
|
| Bulk approve all artifacts in a Theme | After generation |
|
||||||
|
| View per-Topic generation coverage | Any time |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Behaviours that must never occur
|
||||||
|
|
||||||
|
- Generated content is persisted without schema validation
|
||||||
|
- An artifact with status rejected is shown to employees
|
||||||
|
- A generation call produces output that introduces facts not in the Topic
|
||||||
|
- Multiple published artifacts exist for the same Topic and type simultaneously
|
||||||
|
- A how-to is generated for a Topic that contains no process or procedure
|
||||||
|
- A comparison card is generated for a Topic that contains only one concept
|
||||||
|
- Generation blocks the admin interface — it must always be asynchronous
|
||||||
|
- A failed generation is silently ignored — failure must be surfaced and
|
||||||
|
reportable
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Acceptance criteria
|
||||||
|
|
||||||
|
1. Trigger batch generation for a Theme with 4 Topics → 40 artifacts queued
|
||||||
|
(10 per Topic)
|
||||||
|
2. All 40 artifacts reach generated status without error
|
||||||
|
3. Each artifact's content validates against its type's schema
|
||||||
|
4. A generated concept_explainer contains 2–3 paragraphs and one example,
|
||||||
|
all grounded in the Topic body
|
||||||
|
5. A generated scenario_quiz contains exactly one correct option and
|
||||||
|
explanations for all options
|
||||||
|
6. A generated flashcard_set contains between 5 and 10 cards with no
|
||||||
|
overlapping questions
|
||||||
|
7. An admin rejects one artifact → status is rejected → artifact is not
|
||||||
|
visible to employees
|
||||||
|
8. Admin triggers regeneration of the rejected artifact → new artifact
|
||||||
|
generated → previous rejected artifact superseded
|
||||||
|
9. Admin edits a generated artifact inline → edits saved → status published
|
||||||
|
10. Admin bulk-approves all artifacts for a Theme → all move to published
|
||||||
|
11. A Topic with no process content does not have a how-to artifact generated
|
||||||
|
(generation is skipped or flagged, not forced)
|
||||||
|
12. A validation failure on AI output triggers one retry → if retry fails,
|
||||||
|
artifact status is set to failed with reason, nothing is persisted
|
||||||
|
13. Generation job status reflects accurate progress counts throughout
|
||||||
|
the batch process
|
||||||
|
14. Employee view of a Topic shows only published artifact types
|
||||||
Reference in New Issue
Block a user