Add comprehensive documentation for key organizational aspects
- Introduced "Pension Scheme & Benefits" detailing secondary employment benefits and pension specifics. - Created "Roles & Accountabilities" outlining the Holacracy role structure and responsibilities within Respellion. - Added "Security" section covering GDPR compliance and workplace safety protocols. - Established "Spending and Contracting" policy detailing expense categories and submission processes. - Documented "Who We Are" to define Respellion's identity, services, and operational model under Holacracy and ISO 9001.
This commit is contained in:
RaymondVerhoef
@@ -1,279 +1,179 @@
|
||||
# Architecture: employee learning platform
|
||||
# Architecture: Respellion Learning Platform
|
||||
|
||||
## Overview
|
||||
|
||||
A mobile-first progressive web application that provides employees with a structured knowledge library, a 26-week perpetual learning curriculum, and an AI-powered assistant (R42). The knowledge base is the single source of truth for all content, micro learnings, curriculum scheduling, and chat retrieval.
|
||||
A mobile-first single-page web application that gives employees a structured
|
||||
knowledge library, a 26-week per-user learning curriculum, and an AI assistant
|
||||
(R42). The **knowledge graph** stored in PocketBase is the single source of truth
|
||||
for all content, micro-learnings, curriculum scheduling, and chat retrieval.
|
||||
|
||||
Unlike the original design (a Next.js multi-service system with Qdrant), the
|
||||
shipped platform is a **React/Vite SPA talking directly to PocketBase**, with the
|
||||
Anthropic API reached through a thin reverse proxy. All application logic — AI
|
||||
orchestration, retrieval, generation, scoring — runs in the browser.
|
||||
|
||||
```
|
||||
Browser (React SPA)
|
||||
├── PocketBase SDK ───────────────► PocketBase (SQLite): all structured data + auth + files
|
||||
└── callLLM ──► /api/anthropic ──► Anthropic API (key injected by the proxy)
|
||||
(Caddy in prod / Vite proxy in dev)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## System domains
|
||||
|
||||
### Admin app
|
||||
Browser-based interface for content administrators.
|
||||
|
||||
Responsibilities:
|
||||
- Upload source documents (PDF, MD, TXT)
|
||||
- Review and approve AI-generated Theme batches
|
||||
- Edit and finetune AI-generated curriculum
|
||||
- Confirm curriculum regeneration after KB updates
|
||||
- Monitor ingestion and generation job status
|
||||
|
||||
### Employee app
|
||||
Mobile-first PWA accessible on all devices.
|
||||
|
||||
Responsibilities:
|
||||
- Weekly session delivery (Theme + Topics + micro learning type selection)
|
||||
- Knowledge library (browse all published Topics)
|
||||
- Gamification profile (heatmap, badges, streak, leaderboard)
|
||||
- R42 chatbot (available on every screen)
|
||||
|
||||
### Backend services
|
||||
Six discrete services, each with a single responsibility.
|
||||
|
||||
| Service | Responsibility |
|
||||
|---|---|
|
||||
| Ingestion service | Document upload → chunk → extract KB structure |
|
||||
| Generation service | Topics → 10 micro learning types (structured JSON) |
|
||||
| Curriculum service | KB graph → 26-week schedule, versioning, regeneration |
|
||||
| Embedding service | Chunks + topic summaries → Qdrant |
|
||||
| Chat service (R42) | Query → vector retrieval → grounded response |
|
||||
| Progress service | Completions → XP → badges → streak |
|
||||
|
||||
---
|
||||
|
||||
## Deployment topology
|
||||
## Runtime topology
|
||||
|
||||
```
|
||||
repo/
|
||||
├── .github/workflows/ ← pipeline (frozen)
|
||||
├── docker-compose.yml ← infrastructure (frozen)
|
||||
├── Dockerfile ← updated once to point at /app
|
||||
├── ansible/ ← provisioning (frozen)
|
||||
├── legacy/ ← original prototype (read-only reference)
|
||||
└── app/
|
||||
├── frontend/ ← Next.js PWA (admin + employee)
|
||||
└── services/
|
||||
├── ingestion/
|
||||
├── generation/
|
||||
├── curriculum/
|
||||
├── embedding/
|
||||
├── chat/
|
||||
└── progress/
|
||||
┌─────────────────────────────┐ ┌──────────────────────────┐
|
||||
│ Caddy container │ │ PocketBase container │
|
||||
│ - serves built SPA (/srv) │ /api/*│ - SQLite data │
|
||||
│ - /api/anthropic/* → Claude │───────►│ - auth (team_members) │
|
||||
│ - /api/*, /_/* → PocketBase │ /_/* │ - file storage │
|
||||
│ - injects ANTHROPIC_API_KEY │ │ - migrations (pb_migrations)
|
||||
└─────────────────────────────┘ └──────────────────────────┘
|
||||
```
|
||||
|
||||
In local dev, `vite.config.js` replaces Caddy: it proxies `/api/anthropic` to
|
||||
`https://api.anthropic.com` and injects `ANTHROPIC_API_KEY`. PocketBase runs
|
||||
directly (`./pocketbase.exe serve`).
|
||||
|
||||
---
|
||||
|
||||
## Tech stack
|
||||
|
||||
| Layer | Technology | Rationale |
|
||||
|---|---|---|
|
||||
| Frontend | Next.js 14, TypeScript, Tailwind CSS | PWA support, single codebase for admin + employee |
|
||||
| Backend state | PocketBase | Auth, file storage, admin UI, SQLite — no infra overhead |
|
||||
| Vector store | Qdrant (Docker) | RAG retrieval, runs as single container |
|
||||
| AI generation | Claude Sonnet 4 via Anthropic API | Structured JSON output, long-form drafting, graph reasoning |
|
||||
| AI chat (R42) | Claude Haiku 4.5 via Anthropic API | Low latency, cost-effective, grounded by retrieval layer |
|
||||
| Embeddings | OpenAI text-embedding-3-small | Cost-effective, high quality at this scale |
|
||||
| Auth | PocketBase built-in | Role-based: admin / employee |
|
||||
| Frontend | React 19 + Vite 8, React Router 7 | Fast SPA, single codebase for admin + employee |
|
||||
| Styling | CSS variables + Tailwind v4 | Premium design system; Tailwind mapped to variables |
|
||||
| Backend state | PocketBase (SQLite) | Auth, file storage, admin UI — no infra overhead |
|
||||
| Retrieval | Local TF-IDF (`src/lib/retrieval.js`) | Grounds R42 with zero external vector infra |
|
||||
| AI | Claude via Anthropic API (proxied) | Structured tool output, long-form drafting, chat |
|
||||
| Auth | PocketBase `team_members` + PIN | Lightweight internal auth, role = admin / (employee) |
|
||||
| Infra | Docker + Caddy, Ansible (`infra/`) | Containerized deploy to dev/prod |
|
||||
|
||||
There is **no Qdrant, no OpenAI/embeddings service, and no separate Node backend.**
|
||||
|
||||
---
|
||||
|
||||
## AI model responsibilities
|
||||
|
||||
| Task | Model |
|
||||
|---|---|
|
||||
| Document → KB structure extraction | Claude Sonnet 4 |
|
||||
| Topic body drafting | Claude Sonnet 4 |
|
||||
| Micro learning generation (all 10 types) | Claude Sonnet 4 |
|
||||
| Curriculum generation + versioning | Claude Sonnet 4 |
|
||||
| R42 chat responses | Claude Haiku 4.5 |
|
||||
| Embeddings | text-embedding-3-small |
|
||||
`callLLM` (`src/lib/llm.js`) selects a Claude model by **tier**:
|
||||
|
||||
| Tier | Model | Used for |
|
||||
|---|---|---|
|
||||
| `fast` | `claude-haiku-4-5-20251001` | R42 chat, weekly quiz batch, flashcard sets |
|
||||
| `standard` | `claude-sonnet-4-6` | KB extraction, article/slides/infographic, micro-learnings, curriculum generation |
|
||||
| `reasoning` | `claude-opus-4-7` | reserved for heavier reasoning tasks |
|
||||
|
||||
Admins can override the model string per tier from the Settings tab.
|
||||
|
||||
---
|
||||
|
||||
## Document ingestion pipeline
|
||||
## Knowledge ingestion pipeline
|
||||
|
||||
```
|
||||
Admin uploads file (PDF / MD / TXT)
|
||||
↓
|
||||
Format detection → text extraction
|
||||
MD: split on headings → preserve hierarchy
|
||||
PDF: pdfplumber → page + paragraph detection
|
||||
TXT: sliding window chunking with overlap
|
||||
↓
|
||||
Chunk cleaning (strip headers/footers/artefacts)
|
||||
↓
|
||||
Claude Sonnet 4 reads chunks → extracts:
|
||||
- candidate Themes
|
||||
- candidate Topics per Theme
|
||||
- Topic→Topic relationships (related, prerequisite, contrast)
|
||||
- key terms for glossary
|
||||
↓
|
||||
Draft KB written to PocketBase (status: draft)
|
||||
↓
|
||||
Embedding service: embed source chunks → write to Qdrant
|
||||
↓
|
||||
Admin reviews Theme batch → approves / edits / rejects
|
||||
↓
|
||||
On approval: Topics published, micro learning generation queued
|
||||
↓
|
||||
Curriculum regeneration notification queued for admin
|
||||
Admin uploads .txt / .md (≤5 MB) in the Sources tab
|
||||
↓
|
||||
extractionPipeline.js chunks the text (~8000 chars, 800 overlap)
|
||||
↓
|
||||
Per chunk: callLLM (standard tier) with the emit_knowledge_graph tool
|
||||
→ topics (id, label, type, description, learning_relevance)
|
||||
→ relations (source, target, type)
|
||||
↓
|
||||
Results merged into the `topics` and `relations` collections
|
||||
(topic id de-dup; relevance_locked topics keep their relevance)
|
||||
↓
|
||||
Source status tracked in `sources` (processing → completed / failed / cancelled)
|
||||
```
|
||||
|
||||
Note: embeddings are generated from **source chunks**, not only from AI-generated topic summaries. R42 retrieves from grounded source material.
|
||||
There are no embeddings. Retrieval for R42 is computed on the fly with TF-IDF
|
||||
over `topics` (`label + description`).
|
||||
|
||||
MD source files are the preferred format for admins — heading structure maps directly to Theme → Topic hierarchy and improves extraction quality.
|
||||
See `docs/ingestion-spec.md`.
|
||||
|
||||
---
|
||||
|
||||
## Curriculum lifecycle
|
||||
## Content generation
|
||||
|
||||
Two generators, both via `callLLM` with forced tool use and Zod-validated output:
|
||||
|
||||
- **Long-form content** (`learningService.js`) → `content` collection. Three types
|
||||
generated on demand and shallow-merged: `article`, `slides`, `infographic`.
|
||||
- **Micro-learnings** (`microLearningService.js`) → `micro_learnings` collection.
|
||||
Three types: `concept_explainer`, `scenario_quiz`, `flashcard_set`.
|
||||
|
||||
See `docs/generation-spec.md`.
|
||||
|
||||
---
|
||||
|
||||
## Curriculum lifecycle (per-user)
|
||||
|
||||
### Generation
|
||||
Input: all published Themes, Topics, relationship graph, complexity weights
|
||||
Process: cluster by Theme → sequence pedagogically (prerequisites first, complexity gradient) → distribute across 26 weeks → ensure full KB coverage
|
||||
Output: versioned 26-week draft schedule
|
||||
Input: published topics grouped by `theme`, ordered by `complexity_weight`.
|
||||
`curriculumService.generateCurriculumDraft()` asks Claude for a 26-week schedule
|
||||
via `emit_curriculum_schedule`, validates it, and stores a `curriculum_versions`
|
||||
row (`status='draft'`). Admin previews and confirms → `active`. Only one active
|
||||
version exists; the prior active becomes `superseded`.
|
||||
|
||||
### Perpetual cycling
|
||||
The curriculum runs continuously. After week 26, the employee begins cycle 2 on the latest curriculum version.
|
||||
### Per-user cycling
|
||||
The curriculum is **not** anchored to the calendar. Each employee enrolls when
|
||||
they choose (first-login onboarding), which sets `team_members.curriculum_started_at`.
|
||||
Their position is derived:
|
||||
|
||||
Second and subsequent cycles are not identical to cycle 1:
|
||||
- Theme sequence is varied
|
||||
- Recommended micro learning types surface types the employee has not yet used
|
||||
- Topics with low engagement in prior cycles receive increased coverage
|
||||
```
|
||||
personalWeek = floor(daysSinceStart / 7) + 1 // absolute counter, ≥1
|
||||
curriculumWeek = ((personalWeek - 1) % 26) + 1 // 1..26 slot
|
||||
cycle = floor((personalWeek - 1) / 26) + 1 // 1, 2, 3, ...
|
||||
```
|
||||
|
||||
### Versioning rules
|
||||
After week 26 the cycle restarts at week 1 with the **same** content.
|
||||
|
||||
| Event | Action |
|
||||
|---|---|
|
||||
| New source doc published to KB | Regenerate curriculum from week N+1 for all active employees |
|
||||
| Topic body edited | Micro learnings regenerated; curriculum unaffected |
|
||||
| Theme batch approved | Regeneration queued; admin confirms before it applies |
|
||||
|
||||
Completed weeks are immutable. Regeneration only affects future unstarted weeks.
|
||||
|
||||
### Admin regeneration flow
|
||||
Admin receives notification: "N new topics added. Regenerate curriculum? This will update unstarted weeks for all active employees."
|
||||
Admin can preview the proposed new schedule before confirming.
|
||||
See `docs/curriculum-spec.md`.
|
||||
|
||||
---
|
||||
|
||||
## Weekly session flow (employee)
|
||||
|
||||
```
|
||||
Week N opens
|
||||
Enroll (first login) → curriculum_started_at set
|
||||
↓
|
||||
Employee sees assigned Theme + Topics for the week
|
||||
Dashboard shows current cycle / week / assigned topic
|
||||
↓
|
||||
Per Topic: employee selects micro learning type
|
||||
(all published types for that topic are available)
|
||||
Learning Station: complete ≥1 micro-learning for the week's topic(s)
|
||||
↓
|
||||
Employee completes one or more types per topic
|
||||
Weekly Test: 5 AI-generated questions → +2 points per correct answer
|
||||
↓
|
||||
Completion recorded → XP awarded → badges evaluated
|
||||
↓
|
||||
Progress visible on public leaderboard and activity feed
|
||||
Leaderboard updates; badges evaluated at render time
|
||||
```
|
||||
|
||||
Sessions support multiple micro learning types per topic in a single session.
|
||||
|
||||
---
|
||||
|
||||
## Micro learning types
|
||||
|
||||
All 10 types are generated by Claude Sonnet 4 as structured JSON, stored in PocketBase, and rendered by the frontend. One or more types may be published per topic.
|
||||
|
||||
| # | Type | Format |
|
||||
|---|---|---|
|
||||
| 1 | Concept explainer | 2–3 paragraphs + example |
|
||||
| 2 | Scenario quiz | situation + 3–4 MCQ options + explained answers |
|
||||
| 3 | Common misconceptions | 3–5 false beliefs + corrections |
|
||||
| 4 | Step-by-step how-to | numbered procedure |
|
||||
| 5 | Comparison card | side-by-side on 4–6 dimensions |
|
||||
| 6 | Reflection prompt | open question + model answer |
|
||||
| 7 | Spaced repetition flashcards | 5–10 Q&A pairs |
|
||||
| 8 | Case study mini-analysis | 150–200 word scenario + guiding questions |
|
||||
| 9 | Glossary anchor | term + definition + correct use + misuse |
|
||||
| 10 | Myth vs. evidence | false claim + evidence-based rebuttal |
|
||||
|
||||
---
|
||||
|
||||
## R42 — chat service design
|
||||
|
||||
R42 is a functional KB-grounded assistant available on every screen in the employee app.
|
||||
R42 is a KB-grounded assistant on every screen (`src/components/chat/`).
|
||||
|
||||
Behaviour:
|
||||
- Stateless per session (no memory between conversations)
|
||||
- Retrieves relevant chunks from Qdrant using the employee's query
|
||||
- Knows the employee's current curriculum week → retrieval is context-weighted
|
||||
- Cites source topic in every response ("based on the **Holacratic roles** topic")
|
||||
- Explicitly refuses to answer outside KB scope rather than hallucinating
|
||||
- Scope: internal KB only
|
||||
- Persists the conversation per user in `localStorage` (`chat:thread:{userId}`, cap 50; ~12 turns sent to the API).
|
||||
- Builds context with the TF-IDF index (top-K topics + verbatim mentions), injects related relations and limited deep content.
|
||||
- Can propose a `propose_graph_delta` (≤3 topics, ≤5 relations). Admins apply directly; non-admins queue a suggestion for admin approval.
|
||||
- Hidden during quizzes (the `quiz:active:{userId}` integrity rule).
|
||||
|
||||
Implementation:
|
||||
- Employee query → embed → Qdrant nearest-neighbour retrieval → top-K chunks
|
||||
- Chunks + employee context injected into Haiku 4.5 prompt
|
||||
- Response streamed to frontend
|
||||
|
||||
UI: floating button bottom-right, unobtrusive on mobile.
|
||||
See `docs/r42-spec.md`.
|
||||
|
||||
---
|
||||
|
||||
## Gamification system
|
||||
## Gamification
|
||||
|
||||
Inspired by the visual language of GitHub, Stack Overflow, and Duolingo. Mechanics use developer-native terminology.
|
||||
- Points: +2 per correct quiz answer, stored in `leaderboard`.
|
||||
- Badges (render-time): First Steps (1 test), Veteran (5 tests), Perfectionist (100% score).
|
||||
- Leaderboard excludes admins.
|
||||
|
||||
### XP unit: commits
|
||||
Every completed topic earns commits. Quantity varies by micro learning type complexity.
|
||||
|
||||
### Levels
|
||||
`Intern → Junior → Medior → Senior → Staff → Principal`
|
||||
Based on cumulative commits across all cycles.
|
||||
|
||||
### Streak
|
||||
Counted in consecutive weeks, not days. Resets if a week is skipped entirely.
|
||||
|
||||
### Activity heatmap
|
||||
GitHub-style contribution graph spanning the full 26-week cycle. Cell darkness = number of types completed that week.
|
||||
|
||||
### Badges
|
||||
|
||||
| Tier | Condition |
|
||||
|---|---|
|
||||
| Bronze | Complete any session |
|
||||
| Silver | 5 sessions completed, 5 different types used |
|
||||
| Gold | 13 sessions without skipping a week |
|
||||
| Legendary | All 26 sessions, all 10 types used at least once |
|
||||
|
||||
Named content badges (examples):
|
||||
- `governance nerd` — all holacratic structure topics completed
|
||||
- `process architect` — all internal process topics completed
|
||||
- `deep reader` — case study type used 5+ times
|
||||
|
||||
### Milestone cards (public)
|
||||
At weeks 13 and 26, a public card is posted to the shared activity feed:
|
||||
|
||||
```
|
||||
🚀 [Name] shipped the full curriculum
|
||||
26 weeks · [N] commits · [badges]
|
||||
Longest streak: [N] weeks
|
||||
```
|
||||
|
||||
Language: shipping vocabulary, not school vocabulary.
|
||||
|
||||
### Leaderboard
|
||||
Not ranked 1–N by score. Displays multiple dimensions:
|
||||
|
||||
| Employee | Commits | Streak | Types used | Badges |
|
||||
|---|---|---|---|---|
|
||||
|
||||
Multiple paths to visibility. No single metric determines standing.
|
||||
See `docs/gamification-spec.md`.
|
||||
|
||||
---
|
||||
|
||||
## Security and privacy
|
||||
|
||||
- Auth: PocketBase role-based (admin / employee)
|
||||
- Gamification data (commits, badges, streak) is public to all employees
|
||||
- Session completion data (which topic, which type, when) is public
|
||||
- Source documents are admin-only
|
||||
- No PII beyond display name stored in gamification context
|
||||
- R42 is stateless — no chat history persisted
|
||||
- Auth: PocketBase `team_members` with PIN; role `admin` unlocks the Admin panel.
|
||||
- The Anthropic API key never reaches the client — it is injected by Caddy / the Vite proxy.
|
||||
- R42 conversations are stored client-side per user; no server-side chat history.
|
||||
- Source documents and the knowledge graph are managed by admins.
|
||||
|
||||
@@ -1,407 +1,86 @@
|
||||
# Curriculum service spec
|
||||
# Curriculum spec: 26-week per-user cycle
|
||||
|
||||
## Responsibility
|
||||
|
||||
Generates a versioned 26-week learning schedule from the published knowledge
|
||||
base. Manages perpetual cycling, version transitions, and employee curriculum
|
||||
state. Handles regeneration when the KB changes.
|
||||
The curriculum sequences the knowledge base into a 26-week schedule. Every
|
||||
employee runs their **own** cycle, starting when they enroll. Implemented in
|
||||
`src/lib/curriculumService.js`; admin UI in
|
||||
`src/components/admin/CurriculumManager.jsx`.
|
||||
|
||||
---
|
||||
|
||||
## Service location
|
||||
## Data
|
||||
|
||||
```
|
||||
app/services/curriculum/
|
||||
├── src/
|
||||
│ ├── index.ts entry point, Fastify server
|
||||
│ ├── routes/
|
||||
│ │ ├── curriculum.ts POST /generate, GET /current, GET /preview
|
||||
│ │ └── employee.ts GET /state/:userId, POST /advance/:userId
|
||||
│ ├── generator/
|
||||
│ │ ├── build.ts KB graph → 26-week schedule (AI call)
|
||||
│ │ ├── sequence.ts prerequisite + complexity ordering
|
||||
│ │ └── cycle.ts cycle 2+ variation logic
|
||||
│ ├── versioning/
|
||||
│ │ ├── apply.ts apply new version to active employees
|
||||
│ │ └── freeze.ts protect completed weeks
|
||||
│ └── lib/
|
||||
│ ├── pocketbase.ts
|
||||
│ └── anthropic.ts
|
||||
├── package.json
|
||||
├── tsconfig.json
|
||||
└── .env.example
|
||||
```
|
||||
- **`curriculum_versions`** — generated schedules. Lifecycle `draft → active →
|
||||
superseded`; exactly one `active`. The `schedule` JSON is an array of 26 week
|
||||
objects: `{ week_number (1–26), theme, topic_ids[], estimated_duration (15–45),
|
||||
week_rationale }`. `coverage_stats` records theme/topic coverage.
|
||||
- **`topics`** — supply `theme`, `complexity_weight` (1–5), and `difficulty` as
|
||||
generation input.
|
||||
|
||||
---
|
||||
|
||||
## API surface
|
||||
## Topic enrichment (prerequisite)
|
||||
|
||||
### POST /generate
|
||||
|
||||
Triggers curriculum generation from current published KB.
|
||||
Called by admin app after confirming regeneration.
|
||||
|
||||
Request:
|
||||
```json
|
||||
{
|
||||
"triggeredBy": "string",
|
||||
"reason": "new_topics" | "manual"
|
||||
}
|
||||
```
|
||||
|
||||
Response (202 Accepted):
|
||||
```json
|
||||
{
|
||||
"jobId": "string",
|
||||
"status": "queued"
|
||||
}
|
||||
```
|
||||
Generation needs themed, weighted topics. `enrichTopicsForCurriculum()` finds
|
||||
topics missing a `theme` (excluding `type='fact'` and `learning_relevance='exclude'`)
|
||||
and, in batches of 20, calls Claude with `emit_topic_enrichment` to assign
|
||||
`theme`, `complexity_weight`, and `difficulty`. Triggered from the Curriculum tab.
|
||||
|
||||
---
|
||||
|
||||
### GET /preview
|
||||
## Generation
|
||||
|
||||
Returns proposed new curriculum before admin confirms.
|
||||
Called by admin app to show preview before regeneration is applied.
|
||||
`generateCurriculumDraft(reason)`:
|
||||
1. Group learning topics by `theme` (`buildThemeTopicMap`), sorted by
|
||||
`complexity_weight` ascending.
|
||||
2. Build a prompt describing themes and their topic ids. If there are more than 26
|
||||
themes, the model is instructed to **merge** closely related themes.
|
||||
3. `callLLM` (standard tier, `maxTokens: 8192`, temp 0) with forced
|
||||
`emit_curriculum_schedule`. Up to 2 attempts; on validation failure the errors
|
||||
are fed back into the retry prompt.
|
||||
4. Validate (`validateSchedule`): exactly 26 weeks, correct `week_number` sequence,
|
||||
durations 15–45, every `topic_id` exists, ≥1 topic per week. Unscheduled themes
|
||||
are warnings, not hard errors.
|
||||
5. Supersede any existing draft and store the new `draft` version with coverage stats.
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"version": 3,
|
||||
"weeks": [
|
||||
{
|
||||
"weekNumber": 1,
|
||||
"theme": { "id": "string", "title": "string" },
|
||||
"topics": [
|
||||
{ "id": "string", "title": "string", "complexityWeight": 2 }
|
||||
],
|
||||
"estimatedDurationMinutes": 25
|
||||
}
|
||||
],
|
||||
"coverageStats": {
|
||||
"themesTotal": 8,
|
||||
"themesCovered": 8,
|
||||
"topicsTotal": 42,
|
||||
"topicsCovered": 42
|
||||
}
|
||||
}
|
||||
```
|
||||
`confirmVersion(versionId, adminUserId)` activates a draft (and supersedes the old
|
||||
active version); `rejectVersion` discards a draft.
|
||||
|
||||
---
|
||||
|
||||
### GET /current
|
||||
## Per-user scheduling (the cycle)
|
||||
|
||||
Returns the currently active curriculum version with all week slots.
|
||||
The cycle is **detached from the calendar**. Enrollment sets
|
||||
`team_members.curriculum_started_at` (see `docs/frontend-spec.md` for onboarding).
|
||||
`AppContext` derives the user's position:
|
||||
|
||||
```js
|
||||
getPersonalWeekNumber(startedAt) // floor(daysSinceStart / 7) + 1, ≥1 (0 if not enrolled)
|
||||
getCurriculumWeek(personalWeek) // ((n - 1) % 26) + 1 → 1..26 slot
|
||||
getCurriculumCycle(personalWeek) // floor((n - 1) / 26) + 1 → 1, 2, 3, ...
|
||||
```
|
||||
|
||||
- Week 1 begins the day the employee enrolls.
|
||||
- After week 26 the cycle restarts at week 1 with the **same** content.
|
||||
- `state.weekNumber` (the absolute counter) is `0` until the user enrolls; pages are
|
||||
gated behind onboarding so they never render with week 0.
|
||||
|
||||
---
|
||||
|
||||
### GET /state/:userId
|
||||
## Content & progress for a week
|
||||
|
||||
Returns an employee's current curriculum state.
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"userId": "string",
|
||||
"currentCycle": 1,
|
||||
"currentWeek": 7,
|
||||
"startDate": "2026-01-15T00:00:00Z",
|
||||
"activeVersionId": "string",
|
||||
"nextSessionTheme": { "id": "string", "title": "string" },
|
||||
"nextSessionTopics": []
|
||||
}
|
||||
```
|
||||
- `getCurrentWeekContent(personalWeek)` reads the active version's schedule, maps the
|
||||
26-week slot to its `topic_ids`, and returns `{ cycle, weekNumber, theme, topics,
|
||||
estimatedDuration, rationale }`.
|
||||
- `getAssignedTopic(userId, week)` returns the week's primary topic, falling back to
|
||||
a deterministic hash of `userId:week` when no curriculum is active. **Keep the
|
||||
fallback.**
|
||||
- `getYearProgress(userId, personalWeek)` computes completion for the current cycle.
|
||||
|
||||
---
|
||||
|
||||
### POST /advance/:userId
|
||||
## Notes
|
||||
|
||||
Called by progress service when an employee completes a week.
|
||||
Increments currentWeek, handles cycle transition at week 26.
|
||||
|
||||
Request:
|
||||
```json
|
||||
{
|
||||
"completedWeek": 7
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Curriculum generation
|
||||
|
||||
### Input
|
||||
|
||||
All published Themes and Topics retrieved from PocketBase:
|
||||
|
||||
```typescript
|
||||
type KBSnapshot = {
|
||||
themes: {
|
||||
id: string
|
||||
title: string
|
||||
description: string
|
||||
topics: {
|
||||
id: string
|
||||
title: string
|
||||
complexityWeight: number // 1–5
|
||||
difficulty: string
|
||||
prerequisiteTopics: string[] // topic IDs
|
||||
relatedTopics: string[]
|
||||
contrastTopics: string[]
|
||||
}[]
|
||||
}[]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Pre-processing: sequence topics within themes
|
||||
|
||||
Before the AI call, the service resolves topic ordering within each Theme
|
||||
using a topological sort on prerequisite relationships.
|
||||
|
||||
```
|
||||
For each Theme:
|
||||
Build directed graph: prerequisite_topics edges
|
||||
Topological sort → ordered topic list
|
||||
If cycle detected (should not occur but handle): log warning, fall back to
|
||||
complexity_weight ascending order
|
||||
```
|
||||
|
||||
This pre-processing means the AI does not need to reason about prerequisites —
|
||||
it receives already-ordered topic lists and focuses on Theme sequencing.
|
||||
|
||||
---
|
||||
|
||||
### AI call: Theme sequencing across 26 weeks
|
||||
|
||||
System prompt:
|
||||
```
|
||||
You are a curriculum designer. Your task is to distribute a set of learning
|
||||
Themes across 26 weekly sessions to create an effective learning journey.
|
||||
|
||||
Output ONLY valid JSON matching the schema provided. No preamble, no
|
||||
explanation, no markdown fences.
|
||||
|
||||
Rules:
|
||||
- Every Theme must appear at least once across 26 weeks
|
||||
- Themes with more Topics (higher topic count) may span multiple weeks or
|
||||
appear in multiple cycles within the 26 weeks
|
||||
- Sequence Themes so foundational concepts precede dependent ones
|
||||
- Distribute complexity progressively: introductory Themes early, advanced
|
||||
Themes in the second half
|
||||
- If total Topics across all Themes exceeds what 26 weeks can cover in depth,
|
||||
prioritise breadth in cycle 1 — every Theme covered, key Topics per Theme
|
||||
- Assign an estimated duration in minutes per week (15–45 minutes per session)
|
||||
- Return exactly 26 week slots
|
||||
```
|
||||
|
||||
User prompt:
|
||||
```
|
||||
Knowledge base snapshot:
|
||||
{KBSnapshot as JSON}
|
||||
|
||||
Generate a 26-week curriculum schedule.
|
||||
```
|
||||
|
||||
Output schema:
|
||||
```typescript
|
||||
type CurriculumDraft = {
|
||||
weeks: {
|
||||
weekNumber: number // 1–26
|
||||
themeId: string
|
||||
topicIds: string[] // ordered subset of theme's topics
|
||||
estimatedDurationMinutes: number
|
||||
rationale: string // one sentence — shown to admin in preview
|
||||
}[]
|
||||
}
|
||||
```
|
||||
|
||||
AI call configuration:
|
||||
```typescript
|
||||
{
|
||||
model: 'claude-sonnet-4-20250514',
|
||||
max_tokens: 4000,
|
||||
temperature: 0
|
||||
}
|
||||
```
|
||||
|
||||
Validation: Zod schema on output. Check all themeIds and topicIds exist in
|
||||
the KB snapshot before writing. Reject and retry once on validation failure.
|
||||
|
||||
---
|
||||
|
||||
### Write to PocketBase
|
||||
|
||||
```
|
||||
Create curriculum_versions record {
|
||||
version: latest + 1,
|
||||
status: 'draft',
|
||||
generated_at: now,
|
||||
generation_notes: reason
|
||||
}
|
||||
|
||||
For each week in CurriculumDraft:
|
||||
Create curriculum_weeks record {
|
||||
curriculum_version: versionId,
|
||||
week_number: weekNumber,
|
||||
theme: themeId,
|
||||
topics: topicIds,
|
||||
topic_order: [0, 1, 2, ...],
|
||||
estimated_duration_minutes: value,
|
||||
admin_notes: ''
|
||||
}
|
||||
|
||||
Set curriculum_versions.status → 'draft'
|
||||
Notify admin: preview available at GET /preview
|
||||
```
|
||||
|
||||
Draft version is not applied until admin confirms via POST /generate confirm.
|
||||
|
||||
---
|
||||
|
||||
## Versioning and regeneration
|
||||
|
||||
### Applying a new version
|
||||
|
||||
When admin confirms, `apply.ts` runs:
|
||||
|
||||
```
|
||||
Get all employees from employee_curriculum_state
|
||||
|
||||
For each employee:
|
||||
frozenWeek = employee.current_week
|
||||
|
||||
Update employee_curriculum_state:
|
||||
active_version = new version ID
|
||||
|
||||
Note: completed weeks are protected by current_week value
|
||||
The frontend only renders weeks >= current_week from active_version
|
||||
Weeks < current_week are rendered from session_completions history
|
||||
(immutable records — not from curriculum_weeks)
|
||||
|
||||
Set old curriculum_versions.status → 'superseded'
|
||||
Set new curriculum_versions.status → 'active'
|
||||
```
|
||||
|
||||
Completed weeks are never stored against a curriculum version — they live
|
||||
in session_completions. The version only determines future week content.
|
||||
|
||||
---
|
||||
|
||||
## Perpetual cycling
|
||||
|
||||
### Week 26 completion → cycle transition
|
||||
|
||||
When progress service calls POST /advance/:userId with completedWeek: 26:
|
||||
|
||||
```
|
||||
employee.currentCycle += 1
|
||||
employee.currentWeek = 1
|
||||
employee.startDate = now
|
||||
employee.activeVersion = current active version
|
||||
|
||||
Generate cycle variant (see below)
|
||||
```
|
||||
|
||||
### Cycle variant generation
|
||||
|
||||
Cycle 2+ is not identical to cycle 1. The AI call receives additional context:
|
||||
|
||||
Additional fields in user prompt for cycle 2+:
|
||||
```json
|
||||
{
|
||||
"cycleNumber": 2,
|
||||
"employeeHistory": {
|
||||
"typesUsed": ["concept_explainer", "scenario_quiz", "how_to"],
|
||||
"typesNotUsed": ["case_study", "myth_vs_evidence", "comparison_card"],
|
||||
"lowEngagementTopics": ["topic-id-1", "topic-id-2"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Additional rules added to system prompt for cycle 2+:
|
||||
```
|
||||
- Vary the Theme sequence from the previous cycle
|
||||
- Topics identified as low engagement should appear earlier in this cycle
|
||||
- The rationale field should note what is different from cycle 1
|
||||
```
|
||||
|
||||
Low engagement is determined by: topics where the employee completed only
|
||||
one micro learning type (minimum engagement). Retrieved from session_completions
|
||||
by progress service and passed to curriculum service on cycle transition.
|
||||
|
||||
---
|
||||
|
||||
## Admin curriculum editor
|
||||
|
||||
The curriculum editor in the admin app (built in frontend phase) calls:
|
||||
- GET /preview to display the proposed schedule
|
||||
- PATCH /weeks/:weekId to update theme or topic assignment
|
||||
- POST /confirm to apply the version
|
||||
|
||||
The PATCH route allows admin to:
|
||||
- Reassign a Theme to a different week (swap two weeks)
|
||||
- Add or remove Topics from a week's topic list
|
||||
- Edit admin_notes per week
|
||||
|
||||
Changes made via PATCH update the draft curriculum_weeks records before
|
||||
the version is confirmed and applied.
|
||||
|
||||
---
|
||||
|
||||
## Environment variables
|
||||
|
||||
```
|
||||
ANTHROPIC_API_KEY=
|
||||
POCKETBASE_URL=
|
||||
POCKETBASE_ADMIN_EMAIL=
|
||||
POCKETBASE_ADMIN_PASSWORD=
|
||||
CURRICULUM_PORT=3003
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
```json
|
||||
{
|
||||
"dependencies": {
|
||||
"fastify": "^4",
|
||||
"@anthropic-ai/sdk": "^0.24",
|
||||
"pocketbase": "^0.21",
|
||||
"zod": "^3",
|
||||
"uuid": "^9"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## TypeScript strict mode requirements
|
||||
|
||||
- No `any` types
|
||||
- KBSnapshot typed explicitly — validated against PocketBase response
|
||||
- CurriculumDraft validated through Zod before any PocketBase writes
|
||||
- Topological sort implemented with explicit typed graph structure
|
||||
|
||||
---
|
||||
|
||||
## What this service does NOT do
|
||||
|
||||
- Does not generate micro learnings → generation service
|
||||
- Does not record completions → progress service
|
||||
- Does not serve KB content → frontend reads PocketBase directly
|
||||
- Does not handle auth → PocketBase + frontend
|
||||
|
||||
---
|
||||
|
||||
## Testing checkpoints
|
||||
|
||||
1. Generate curriculum from a KB with 5+ themes → confirm 26 weeks produced
|
||||
2. Confirm all themes appear at least once
|
||||
3. Confirm topic order within a week respects prerequisites
|
||||
4. Add a new theme to KB → trigger regeneration → confirm employee at week 5
|
||||
sees weeks 1–5 unchanged, weeks 6–26 updated
|
||||
5. Advance employee through week 26 → confirm cycle 2 starts with varied sequence
|
||||
6. Admin edits week 3 theme → confirm patch updates draft before confirmation
|
||||
- There is no shared "current week" and no `admin:current_week` setting.
|
||||
- Regeneration produces a new version; activating it changes future weeks for all
|
||||
users. Completion history (`micro_learning_completions`) is append-only and never
|
||||
rewritten.
|
||||
|
||||
@@ -1,397 +1,252 @@
|
||||
# Data model: employee learning platform
|
||||
# Data model: Respellion Learning Platform
|
||||
|
||||
## Overview
|
||||
|
||||
Two storage systems:
|
||||
- **PocketBase** — all structured relational data (SQLite under the hood)
|
||||
- **Qdrant** — all vector embeddings for RAG retrieval
|
||||
All structured data lives in **PocketBase** (SQLite). There is **no vector store**
|
||||
— retrieval is computed at runtime with a local TF-IDF index over `topics`
|
||||
(`src/lib/retrieval.js`).
|
||||
|
||||
Schema is defined by JS migrations in `pb_migrations/` (applied automatically by
|
||||
the PocketBase binary) and mirrored for local bootstrap in
|
||||
`scripts/setup-pb-collections.mjs`. The data-access layer is `src/lib/db.js`.
|
||||
|
||||
All collections use PocketBase's auto `id`, plus `created` / `updated` autodate
|
||||
fields unless noted otherwise.
|
||||
|
||||
---
|
||||
|
||||
## PocketBase collections
|
||||
|
||||
### `source_documents`
|
||||
Uploaded source files. Parent of all generated KB content.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| filename | string | original filename |
|
||||
| file | file | PocketBase file storage |
|
||||
| format | select | `pdf` `md` `txt` |
|
||||
| status | select | `processing` `processed` `failed` |
|
||||
| ingested_at | datetime | |
|
||||
| chunk_count | number | total chunks extracted |
|
||||
| created_by | relation → `users` | admin who uploaded |
|
||||
|
||||
---
|
||||
|
||||
### `themes`
|
||||
Top-level content groupings. One Theme = one weekly session.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| title | string | |
|
||||
| description | text | AI drafted, admin editable |
|
||||
| status | select | `draft` `published` |
|
||||
| source_documents | relation[] → `source_documents` | which docs contributed |
|
||||
| approved_by | relation → `users` | admin who approved batch |
|
||||
| approved_at | datetime | |
|
||||
| created_at | datetime | |
|
||||
| updated_at | datetime | |
|
||||
|
||||
---
|
||||
|
||||
### `topics`
|
||||
Atomic knowledge units. Always belong to a Theme.
|
||||
Knowledge graph nodes. Created during ingestion, enriched for curriculum.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| theme | relation → `themes` | parent theme |
|
||||
| title | string | |
|
||||
| body | text (rich) | AI drafted, admin editable |
|
||||
| difficulty | select | `introductory` `intermediate` `advanced` |
|
||||
| complexity_weight | number | 1–5, used by curriculum generator |
|
||||
| status | select | `draft` `published` |
|
||||
| related_topics | relation[] → `topics` | lateral relationships |
|
||||
| prerequisite_topics | relation[] → `topics` | must-complete-first |
|
||||
| contrast_topics | relation[] → `topics` | deliberate opposites |
|
||||
| key_terms | json | string[] — feeds glossary |
|
||||
| qdrant_chunk_ids | json | string[] — references to embedded chunks |
|
||||
| created_at | datetime | |
|
||||
| updated_at | datetime | |
|
||||
| id | text | kebab-case slug (e.g. `holacratic-roles`) |
|
||||
| label | text | display name |
|
||||
| type | text | `concept` · `role` · `process` (ingestion); `fact` is excluded from learning |
|
||||
| description | text | 1–2 sentence summary |
|
||||
| learning_relevance | text | `core` · `standard` · `peripheral` · `exclude` |
|
||||
| relevance_locked | bool | if true, re-ingestion will not overwrite `learning_relevance` |
|
||||
| theme | text | subject grouping (used by curriculum generation) |
|
||||
| complexity_weight | number | 1–5 (curriculum ordering) |
|
||||
| difficulty | text | `introductory` · `intermediate` · `advanced` |
|
||||
|
||||
Relationship types (related / prerequisite / contrast) are stored via the three explicit relation fields rather than a generic relationship table. This keeps queries simple at this scale.
|
||||
Topics with `type='fact'` or `learning_relevance='exclude'` are filtered out of
|
||||
learning, micro-learning, curriculum, and test selection.
|
||||
|
||||
---
|
||||
|
||||
### `relations`
|
||||
Knowledge graph edges between topics.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| source | text | topic id |
|
||||
| target | text | topic id |
|
||||
| type | text | `related_to` · `depends_on` · `part_of` · `executed_by` |
|
||||
|
||||
Edges are de-duplicated on the `(source, target, type)` tuple.
|
||||
|
||||
---
|
||||
|
||||
### `content`
|
||||
On-demand long-form learning content, one record per topic.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| topic_id | text | topic this content belongs to |
|
||||
| data | json | merged object — only generated types are present |
|
||||
|
||||
`data` shape (each key generated independently and shallow-merged):
|
||||
|
||||
```json
|
||||
{
|
||||
"article": { "title", "intro", "sections": [{ "heading", "body" }], "keyTakeaways": [] },
|
||||
"slides": [ { "title", "bullets": [], "speakerNote" } ],
|
||||
"infographic": { "headline", "tagline", "stats": [{ "value", "label", "icon" }],
|
||||
"steps": [{ "number", "title", "description", "icon" }], "quote", "colorTheme" }
|
||||
}
|
||||
```
|
||||
|
||||
> There is no `podcast` key. The podcast type was removed.
|
||||
|
||||
---
|
||||
|
||||
### `micro_learnings`
|
||||
Generated content artifacts. One record per topic per type.
|
||||
Generated micro-learning artifacts. One record per topic per type.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| topic | relation → `topics` | |
|
||||
| type | select | see type enum below |
|
||||
| content | json | structured output — schema varies per type |
|
||||
| status | select | `queued` `generated` `published` `rejected` |
|
||||
| generation_model | string | model version used |
|
||||
| generated_at | datetime | |
|
||||
| published_at | datetime | |
|
||||
| updated_at | datetime | |
|
||||
| topic_id | relation → `topics` | cascade delete |
|
||||
| type | select | `concept_explainer` · `scenario_quiz` · `flashcard_set` |
|
||||
| content | json | structured output, schema varies per type |
|
||||
| status | select | `draft` · `published` (only `published` is visible to employees) |
|
||||
|
||||
**Type enum:**
|
||||
`concept_explainer` `scenario_quiz` `misconceptions` `how_to` `comparison_card` `reflection_prompt` `flashcard_set` `case_study` `glossary_anchor` `myth_vs_evidence`
|
||||
|
||||
**Content JSON schemas per type:**
|
||||
**Content JSON per type:**
|
||||
|
||||
```json
|
||||
// concept_explainer
|
||||
{
|
||||
"paragraphs": ["string", "string"],
|
||||
"example": "string"
|
||||
}
|
||||
{ "sections": [ { "title": "string", "content": "string (HTML: <p>, <ul>, <li>, <strong>)" } ] } // ≥3 sections
|
||||
|
||||
// scenario_quiz
|
||||
{
|
||||
"scenario": "string",
|
||||
"options": [
|
||||
{ "label": "A", "text": "string", "correct": true, "explanation": "string" }
|
||||
]
|
||||
}
|
||||
|
||||
// misconceptions
|
||||
{
|
||||
"items": [
|
||||
{ "misconception": "string", "correction": "string" }
|
||||
]
|
||||
}
|
||||
|
||||
// how_to
|
||||
{
|
||||
"steps": [
|
||||
{ "number": 1, "instruction": "string" }
|
||||
]
|
||||
}
|
||||
|
||||
// comparison_card
|
||||
{
|
||||
"subject_a": "string",
|
||||
"subject_b": "string",
|
||||
"dimensions": [
|
||||
{ "label": "string", "a": "string", "b": "string" }
|
||||
]
|
||||
}
|
||||
|
||||
// reflection_prompt
|
||||
{
|
||||
"prompt": "string",
|
||||
"model_answer": "string"
|
||||
}
|
||||
{ "scenario": "string",
|
||||
"options": [ { "text": "string", "isCorrect": true, "explanation": "string" } ] } // 3–4 options, exactly 1 correct
|
||||
|
||||
// flashcard_set
|
||||
{
|
||||
"cards": [
|
||||
{ "question": "string", "answer": "string" }
|
||||
]
|
||||
}
|
||||
|
||||
// case_study
|
||||
{
|
||||
"scenario": "string",
|
||||
"questions": ["string"]
|
||||
}
|
||||
|
||||
// glossary_anchor
|
||||
{
|
||||
"term": "string",
|
||||
"definition": "string",
|
||||
"correct_use": "string",
|
||||
"misuse": "string"
|
||||
}
|
||||
|
||||
// myth_vs_evidence
|
||||
{
|
||||
"myth": "string",
|
||||
"evidence": "string",
|
||||
"sources": ["string"]
|
||||
}
|
||||
{ "cards": [ { "front": "string", "back": "string" } ] } // 5–10 cards
|
||||
```
|
||||
|
||||
> A former `reflection_prompt` type was dropped and is no longer generated.
|
||||
|
||||
---
|
||||
|
||||
### `micro_learning_completions`
|
||||
Append-only completion events. Never updated or deleted.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| team_member_id | relation → `team_members` | the employee |
|
||||
| micro_learning_id | relation → `micro_learnings` | the artifact completed |
|
||||
| topic_id | relation → `topics` | denormalized topic |
|
||||
| type | text | type at time of completion |
|
||||
| session_week | number | the user's **absolute** curriculum week (week 1 = day they enrolled) |
|
||||
|
||||
The 26-week slot and cycle are derived from `session_week`; there is no stored
|
||||
`cycle` field.
|
||||
|
||||
---
|
||||
|
||||
### `curriculum_versions`
|
||||
Versioned 26-week schedule. New version created on each regeneration.
|
||||
Versioned 26-week schedules. New version on each (re)generation.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| version | number | increments on each regeneration |
|
||||
| status | select | `draft` `active` `superseded` |
|
||||
| generated_at | datetime | |
|
||||
| approved_by | relation → `users` | admin who confirmed regeneration |
|
||||
| approved_at | datetime | |
|
||||
| generation_notes | text | why this version was created |
|
||||
| version_number | number | increments per generation |
|
||||
| status | text | `draft` · `active` · `superseded` (exactly one `active`) |
|
||||
| generation_reason | text | why this version was created |
|
||||
| confirmed_by | text | admin id who activated it |
|
||||
| confirmed_at | text | ISO datetime |
|
||||
| schedule | json | array of 26 week objects (below) |
|
||||
| coverage_stats | json | `{ themes_kb, themes_scheduled, topics_kb, topics_scheduled }` |
|
||||
|
||||
Only one version has status `active` at any time.
|
||||
**`schedule[]` week object:**
|
||||
|
||||
---
|
||||
|
||||
### `curriculum_weeks`
|
||||
Individual week slots. Child of a curriculum version.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| curriculum_version | relation → `curriculum_versions` | |
|
||||
| week_number | number | 1–26 |
|
||||
| theme | relation → `themes` | |
|
||||
| topics | relation[] → `topics` | ordered subset of theme topics |
|
||||
| topic_order | json | number[] — explicit ordering |
|
||||
| estimated_duration_minutes | number | AI estimate |
|
||||
| admin_notes | text | freeform admin annotation |
|
||||
|
||||
---
|
||||
|
||||
### `employee_curriculum_state`
|
||||
Tracks each employee's position in the curriculum. One record per employee.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| user | relation → `users` | |
|
||||
| current_cycle | number | starts at 1, increments on loop |
|
||||
| current_week | number | 1–26 |
|
||||
| start_date | datetime | rolling start |
|
||||
| active_version | relation → `curriculum_versions` | version employee is on |
|
||||
| updated_at | datetime | |
|
||||
|
||||
When curriculum regenerates: `active_version` updates for all employees whose `current_week` is less than the first regenerated week.
|
||||
|
||||
---
|
||||
|
||||
### `session_completions`
|
||||
Immutable completion records. One record per employee per topic per type.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| user | relation → `users` | |
|
||||
| topic | relation → `topics` | |
|
||||
| micro_learning | relation → `micro_learnings` | specific type completed |
|
||||
| week_number | number | curriculum week at time of completion |
|
||||
| cycle | number | which cycle |
|
||||
| completed_at | datetime | |
|
||||
|
||||
Records are never updated or deleted. This is the canonical history.
|
||||
|
||||
---
|
||||
|
||||
### `gamification_profiles`
|
||||
One record per employee. Updated by progress service on each completion.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| user | relation → `users` | |
|
||||
| total_commits | number | cumulative XP |
|
||||
| current_level | select | `intern` `junior` `medior` `senior` `staff` `principal` |
|
||||
| current_streak_weeks | number | consecutive weeks with ≥1 completion |
|
||||
| longest_streak_weeks | number | all-time high |
|
||||
| types_used | json | string[] — which of 10 types used at least once |
|
||||
| last_active_week | number | used to detect streak breaks |
|
||||
| updated_at | datetime | |
|
||||
|
||||
---
|
||||
|
||||
### `badges`
|
||||
Badge definitions. Seeded at startup, not user-generated.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| key | string | unique slug e.g. `governance_nerd` |
|
||||
| tier | select | `bronze` `silver` `gold` `legendary` `content` |
|
||||
| label | string | display name |
|
||||
| description | string | award condition description |
|
||||
| icon | string | emoji or icon key |
|
||||
|
||||
---
|
||||
|
||||
### `employee_badges`
|
||||
Junction: which employees have earned which badges.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| user | relation → `users` | |
|
||||
| badge | relation → `badges` | |
|
||||
| earned_at | datetime | |
|
||||
| cycle | number | which cycle it was earned in |
|
||||
|
||||
---
|
||||
|
||||
### `milestone_cards`
|
||||
Public milestone events at weeks 13 and 26.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| user | relation → `users` | |
|
||||
| cycle | number | |
|
||||
| week | number | 13 or 26 |
|
||||
| total_commits | number | snapshot at time of milestone |
|
||||
| streak_weeks | number | snapshot |
|
||||
| badge_keys | json | string[] — badges held at milestone |
|
||||
| created_at | datetime | public feed ordered by this |
|
||||
|
||||
---
|
||||
|
||||
## PocketBase users collection (extended)
|
||||
|
||||
Standard PocketBase `users` collection with additional fields:
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| role | select | `admin` `employee` |
|
||||
| display_name | string | used in gamification feed |
|
||||
| avatar | file | optional |
|
||||
|
||||
---
|
||||
|
||||
## Qdrant collections
|
||||
|
||||
### `source_chunks`
|
||||
Embeddings of raw source document chunks. Primary retrieval target for R42.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | UUID |
|
||||
| vector | float[] | 1536 dimensions (text-embedding-3-small) |
|
||||
| source_document_id | string | reference to PocketBase |
|
||||
| chunk_index | number | position within document |
|
||||
| text | string | raw chunk text |
|
||||
| theme_id | string | assigned theme (post-extraction) |
|
||||
| topic_id | string | assigned topic (post-extraction, nullable) |
|
||||
| format | string | pdf / md / txt |
|
||||
|
||||
### `topic_summaries`
|
||||
Embeddings of AI-generated topic body text. Secondary retrieval target.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | UUID |
|
||||
| vector | float[] | 1536 dimensions |
|
||||
| topic_id | string | reference to PocketBase |
|
||||
| theme_id | string | |
|
||||
| title | string | for display in R42 citation |
|
||||
| text | string | full topic body |
|
||||
|
||||
---
|
||||
|
||||
## Retrieval strategy for R42
|
||||
|
||||
R42 queries both Qdrant collections and merges results:
|
||||
|
||||
```
|
||||
Employee query
|
||||
↓
|
||||
Embed query → text-embedding-3-small
|
||||
↓
|
||||
Qdrant search: source_chunks (top 5) + topic_summaries (top 3)
|
||||
↓
|
||||
Filter: boost chunks from employee's current week theme
|
||||
↓
|
||||
Merge + deduplicate by topic_id
|
||||
↓
|
||||
Top-K context injected into Haiku 4.5 prompt
|
||||
↓
|
||||
Response includes: answer + cited topic title(s)
|
||||
```json
|
||||
{ "week_number": 1, // 1..26
|
||||
"theme": "string",
|
||||
"topic_ids": ["topic-id"], // 1+ topic ids
|
||||
"estimated_duration": 30, // 15..45 minutes
|
||||
"week_rationale": "string" }
|
||||
```
|
||||
|
||||
Source chunks are weighted higher than topic summaries to keep R42 grounded in original source material rather than AI-generated abstractions.
|
||||
|
||||
---
|
||||
|
||||
## Indexes and query patterns
|
||||
### `team_members`
|
||||
Registered users with PIN auth. This is the auth + employee record.
|
||||
|
||||
Critical query patterns the data model must support efficiently:
|
||||
|
||||
| Query | Collection | Index |
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| All published topics for a theme | topics | theme + status |
|
||||
| All micro learnings for a topic | micro_learnings | topic + status |
|
||||
| Employee's current week | employee_curriculum_state | user |
|
||||
| Weeks for a curriculum version | curriculum_weeks | curriculum_version + week_number |
|
||||
| Employee completion history | session_completions | user + cycle |
|
||||
| Public leaderboard | gamification_profiles | total_commits + streak |
|
||||
| Milestone feed | milestone_cards | created_at DESC |
|
||||
| Badges earned by employee | employee_badges | user |
|
||||
| name | text | display name |
|
||||
| pin | text | login PIN |
|
||||
| role | text | `admin` or empty/`employee` |
|
||||
| curriculum_started_at | date | timestamp the user enrolled (week 1 anchor); empty until enrolled |
|
||||
| enrollment_status | text | `not_started` · `active` |
|
||||
|
||||
PocketBase creates indexes on relation fields by default. Composite indexes on `status` fields should be added manually where query frequency warrants it.
|
||||
A user is gated through the `/onboarding` screen until `enrollment_status='active'`
|
||||
(admins are exempt when heading to the admin panel).
|
||||
|
||||
---
|
||||
|
||||
## Data flow summary
|
||||
### `sources`
|
||||
Uploaded source documents and their extraction status.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| name | text | original filename |
|
||||
| status | text | `processing` · `completed` · `failed` · `cancelled` |
|
||||
| error | text | failure message, if any |
|
||||
| progress | json | `{ current, total, message }` during chunked extraction |
|
||||
|
||||
---
|
||||
|
||||
### `leaderboard`
|
||||
Points ledger, one row per user.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| user_id | text | team member id |
|
||||
| name | text | display name |
|
||||
| points | number | cumulative (+2 per correct quiz answer) |
|
||||
| tests_completed | number | count of completed tests |
|
||||
| learnings_completed | number | reserved counter |
|
||||
|
||||
Admins are filtered out of the public leaderboard at render time.
|
||||
|
||||
---
|
||||
|
||||
### `settings`
|
||||
App-wide key/value store.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| key | text | setting key |
|
||||
| value | text | stringified value |
|
||||
|
||||
---
|
||||
|
||||
### `llm_calls`
|
||||
Best-effort telemetry for every Anthropic call (written by `callLLM`).
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| task | text | logging label (e.g. `learning.article`, `chat.r42`) |
|
||||
| model | text | resolved model string |
|
||||
| tier | text | `fast` · `standard` · `reasoning` |
|
||||
| duration_ms | number | wall-clock |
|
||||
| input_tokens / output_tokens | number | usage |
|
||||
| cache_read_tokens / cache_create_tokens | number | prompt-cache usage |
|
||||
| stop_reason | text | `end_turn` · `tool_use` · `max_tokens` |
|
||||
| ok | bool | success flag |
|
||||
| error_msg | text | error, if any |
|
||||
|
||||
---
|
||||
|
||||
## Dropped / legacy collections
|
||||
|
||||
These existed in earlier iterations and have been removed. Their `db.js` helpers
|
||||
remain as deprecated no-op stubs — do not build on them:
|
||||
|
||||
`quiz_banks`, `quiz_results`, `quiz_cache`, `learn_progress`, and the v1
|
||||
`curriculum` collection.
|
||||
|
||||
---
|
||||
|
||||
## Client-side storage (not PocketBase)
|
||||
|
||||
`localStorage` is used only for admin/browser-local state:
|
||||
|
||||
| Key | Purpose |
|
||||
|---|---|
|
||||
| `admin:model:{fast,standard,reasoning}` | per-tier model overrides (legacy `admin:model`) |
|
||||
| `admin:use_simulation` | stub LLM responses instead of calling Anthropic |
|
||||
| `kb:suggestions` | R42 graph-delta suggestion queue (managed via `kbStore`) |
|
||||
| `quiz:active:{userId}` | mid-quiz flag (hides R42) |
|
||||
| `chat:thread:{userId}` | R42 conversation, capped at 50 messages |
|
||||
|
||||
`sessionStorage.respellion_session` holds the logged-in team member id.
|
||||
|
||||
---
|
||||
|
||||
## Retrieval (no vector DB)
|
||||
|
||||
R42 context is built by `src/lib/retrieval.js`:
|
||||
|
||||
```
|
||||
source_documents
|
||||
└── (ingestion service)
|
||||
└── qdrant: source_chunks
|
||||
└── themes (draft)
|
||||
└── topics (draft)
|
||||
└── (approval)
|
||||
└── topics (published)
|
||||
└── qdrant: topic_summaries
|
||||
└── micro_learnings (queued → published)
|
||||
└── (curriculum service)
|
||||
└── curriculum_versions
|
||||
└── curriculum_weeks
|
||||
└── (employee progress)
|
||||
└── session_completions
|
||||
└── gamification_profiles
|
||||
└── employee_badges
|
||||
└── milestone_cards
|
||||
buildIndex(topics) → TF-IDF index over (label + description), cached by array ref
|
||||
retrieveTopK(index, q, k) → top-K topics, score = Σ (1 + log(tf)) · log((N+1)/(df+1))
|
||||
```
|
||||
|
||||
`src/components/chat/rag.js` combines top-K results with verbatim topic mentions,
|
||||
filters relations to the retrieved set, and injects limited deep content for
|
||||
explicitly named topics.
|
||||
|
||||
@@ -1,701 +1,93 @@
|
||||
# Frontend spec
|
||||
|
||||
## Responsibility
|
||||
|
||||
Single Next.js 14 codebase serving two distinct role-based experiences:
|
||||
- `/admin/*` — content administration (document upload, KB review, curriculum)
|
||||
- `/app/*` — employee learning experience (sessions, library, R42, gamification)
|
||||
|
||||
Mobile-first. Designed for 375px width, scales up. Installable as a PWA.
|
||||
A React 19 SPA built with Vite, routed by React Router 7. Entry: `src/main.jsx` →
|
||||
`src/App.jsx`. Global state lives in `src/store/AppContext.jsx`.
|
||||
|
||||
---
|
||||
|
||||
## Location
|
||||
## Routing & access control (`src/App.jsx`)
|
||||
|
||||
```
|
||||
app/frontend/
|
||||
├── src/
|
||||
│ ├── app/ Next.js app router
|
||||
│ │ ├── layout.tsx root layout — global stylesheet import
|
||||
│ │ ├── page.tsx redirect → role-based landing
|
||||
│ │ ├── admin/
|
||||
│ │ │ ├── layout.tsx admin shell (sidebar nav)
|
||||
│ │ │ ├── page.tsx admin dashboard
|
||||
│ │ │ ├── documents/
|
||||
│ │ │ │ └── page.tsx document upload + ingestion status
|
||||
│ │ │ ├── knowledge/
|
||||
│ │ │ │ ├── page.tsx theme batch review list
|
||||
│ │ │ │ └── [themeId]/page.tsx theme detail + topic edit
|
||||
│ │ │ └── curriculum/
|
||||
│ │ │ └── page.tsx curriculum editor + regeneration
|
||||
│ │ ├── app/
|
||||
│ │ │ ├── layout.tsx employee shell (bottom nav + R42)
|
||||
│ │ │ ├── page.tsx redirect → /app/session
|
||||
│ │ │ ├── session/
|
||||
│ │ │ │ └── page.tsx current week session
|
||||
│ │ │ ├── library/
|
||||
│ │ │ │ ├── page.tsx knowledge library browse
|
||||
│ │ │ │ └── [topicId]/page.tsx topic detail
|
||||
│ │ │ └── profile/
|
||||
│ │ │ └── page.tsx gamification profile + heatmap + badges
|
||||
│ │ ├── auth/
|
||||
│ │ │ └── page.tsx login (PocketBase auth)
|
||||
│ │ └── api/ Next.js API routes (thin proxies only)
|
||||
│ ├── components/
|
||||
│ │ ├── admin/ admin-specific components
|
||||
│ │ ├── employee/ employee-specific components
|
||||
│ │ ├── micro-learnings/ one component per micro learning type
|
||||
│ │ ├── r42/ R42 chatbot components
|
||||
│ │ ├── gamification/ heatmap, badges, leaderboard
|
||||
│ │ └── ui/ shared primitives
|
||||
│ ├── lib/
|
||||
│ │ ├── pocketbase.ts PocketBase client (browser)
|
||||
│ │ ├── services.ts typed API calls to backend services
|
||||
│ │ ├── auth.ts auth helpers + role guards
|
||||
│ │ └── hooks/ custom React hooks
|
||||
│ └── types/
|
||||
│ └── index.ts shared TypeScript types
|
||||
├── public/
|
||||
│ ├── manifest.json PWA manifest
|
||||
│ ├── sw.js service worker (generated)
|
||||
│ └── icons/ PWA icons (192, 512)
|
||||
├── next.config.js
|
||||
├── tailwind.config.ts
|
||||
├── tsconfig.json
|
||||
└── .env.example
|
||||
```
|
||||
| Route | Screen | Access |
|
||||
|---|---|---|
|
||||
| `/login` | `Login.jsx` | public |
|
||||
| `/onboarding` | `Onboarding.jsx` | logged-in, not yet enrolled |
|
||||
| `/` | `Dashboard.jsx` | enrolled user |
|
||||
| `/learn` | `Leren.jsx` | enrolled user |
|
||||
| `/test` | `Testen.jsx` | enrolled user |
|
||||
| `/leaderboard` | `Leaderboard.jsx` | enrolled user |
|
||||
| `/admin/*` | `Admin/index.jsx` | `role === 'admin'` |
|
||||
|
||||
`ProtectedRoute`:
|
||||
- redirects to `/login` if not authenticated;
|
||||
- redirects to `/onboarding` if `enrollment_status !== 'active'` — **except** admins
|
||||
heading to the admin panel, who are exempt;
|
||||
- enforces `requireAdmin` for `/admin`.
|
||||
|
||||
Navigation chrome (top bar + mobile bottom nav) is rendered by `ProtectedRoute`.
|
||||
`ChatLauncher` (R42) is mounted globally.
|
||||
|
||||
---
|
||||
|
||||
## Stylesheet integration
|
||||
## Auth & global state (`AppContext.jsx`)
|
||||
|
||||
`/stylesheet.css` lives at the repo root — not inside `app/frontend/`.
|
||||
|
||||
Import it as the first global stylesheet in `src/app/layout.tsx`:
|
||||
|
||||
```tsx
|
||||
import '../../../stylesheet.css' // path from app/frontend/src/app/
|
||||
import './globals.css' // Tailwind directives second
|
||||
```
|
||||
|
||||
Rules:
|
||||
- stylesheet.css is the authoritative visual style — never override it
|
||||
- Where Tailwind utility classes conflict with stylesheet.css rules,
|
||||
stylesheet.css wins
|
||||
- Tailwind is used for layout, spacing, and elements not covered by the
|
||||
stylesheet — match the visual language (spacing scale, colour, type) of
|
||||
the existing stylesheet when doing so
|
||||
- Inspect stylesheet.css before implementing any component — use its CSS
|
||||
custom properties (if any) rather than hardcoding values
|
||||
- Loads `team_members` on mount; auto-creates an `Admin` (PIN `0000`) if the table is empty.
|
||||
- PIN login resolves a member and stores the id in `sessionStorage.respellion_session`.
|
||||
- `state.currentUser` holds the member; `state.weekNumber` is the user's **absolute
|
||||
curriculum week**, derived from `curriculum_started_at` via `getPersonalWeekNumber`
|
||||
(0 until enrolled).
|
||||
- `enrollCurrentUser()` stamps `curriculum_started_at = now`, sets
|
||||
`enrollment_status = 'active'`, and updates state.
|
||||
|
||||
---
|
||||
|
||||
## PWA configuration
|
||||
## Onboarding (`Onboarding.jsx`)
|
||||
|
||||
### next.config.js
|
||||
|
||||
Use `next-pwa` package to generate service worker and manifest wiring:
|
||||
|
||||
```javascript
|
||||
const withPWA = require('next-pwa')({
|
||||
dest: 'public',
|
||||
register: true,
|
||||
skipWaiting: true,
|
||||
disable: process.env.NODE_ENV === 'development'
|
||||
})
|
||||
|
||||
module.exports = withPWA({
|
||||
reactStrictMode: true,
|
||||
})
|
||||
```
|
||||
|
||||
### public/manifest.json
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "Learning Platform",
|
||||
"short_name": "Learn",
|
||||
"description": "Employee knowledge and learning",
|
||||
"start_url": "/app",
|
||||
"display": "standalone",
|
||||
"background_color": "#ffffff",
|
||||
"theme_color": "#ffffff",
|
||||
"orientation": "portrait",
|
||||
"icons": [
|
||||
{ "src": "/icons/192.png", "sizes": "192x192", "type": "image/png" },
|
||||
{ "src": "/icons/512.png", "sizes": "512x512", "type": "image/png" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Note: set `theme_color` and `background_color` to match stylesheet.css
|
||||
primary background after inspecting the file.
|
||||
|
||||
### Service worker caching strategy
|
||||
|
||||
- Static assets: cache-first
|
||||
- PocketBase API calls: network-first, fall back to cache
|
||||
- Backend service calls: network-only (no caching for dynamic content)
|
||||
A blocking first-login screen. One CTA — "Start my journey" — calls
|
||||
`enrollCurrentUser()` and routes to `/`. Week 1 begins immediately. Users already
|
||||
enrolled are redirected to `/`. See `docs/curriculum-spec.md`.
|
||||
|
||||
---
|
||||
|
||||
## Auth
|
||||
## Employee screens
|
||||
|
||||
PocketBase handles auth. Two roles: `admin` and `employee`.
|
||||
- **Dashboard** — current cycle/week, assigned topic, cycle progress ring, quick
|
||||
links to Learn and Test, mini leaderboard, recent activity.
|
||||
- **Learning Station (`/learn`)** — the week's required topic + the rest of the
|
||||
knowledge library; opening a topic shows the micro-learning selector
|
||||
(`src/components/micro_learning/`). Completing ≥1 micro-learning marks the week done.
|
||||
- **Test (`/test`)** — 5-question quiz with a 5-minute timer, per-question feedback,
|
||||
and a results/review screen. Sets the `quiz:active` flag to hide R42.
|
||||
- **Leaderboard (`/leaderboard`)** — podium + ranked list + badges (see
|
||||
`docs/gamification-spec.md`).
|
||||
|
||||
### Login flow
|
||||
|
||||
```
|
||||
/auth page → email + password form
|
||||
↓
|
||||
PocketBase authWithPassword()
|
||||
↓
|
||||
Store token in PocketBase SDK (persists in localStorage)
|
||||
↓
|
||||
Read user.role from auth record
|
||||
↓
|
||||
role === 'admin' → redirect to /admin
|
||||
role === 'employee' → redirect to /app
|
||||
```
|
||||
|
||||
### Route guards
|
||||
|
||||
Implement as Next.js middleware (`middleware.ts` at app root):
|
||||
|
||||
```typescript
|
||||
// Admin routes: require role === 'admin'
|
||||
// Employee routes: require role === 'employee'
|
||||
// Unauthenticated: redirect to /auth
|
||||
// Wrong role: redirect to correct landing
|
||||
```
|
||||
|
||||
### PocketBase client (browser)
|
||||
|
||||
```typescript
|
||||
// lib/pocketbase.ts
|
||||
import PocketBase from 'pocketbase'
|
||||
export const pb = new PocketBase(process.env.NEXT_PUBLIC_POCKETBASE_URL)
|
||||
```
|
||||
|
||||
Use `pb.authStore` for auth state. Use `pb.collection().getFullList()` etc.
|
||||
for direct PocketBase reads. The frontend reads KB content (topics, micro
|
||||
learnings) directly from PocketBase — it does not proxy through backend services.
|
||||
|
||||
### Service calls
|
||||
|
||||
Backend services (ingestion, generation, curriculum, chat, progress) are called
|
||||
via typed fetch wrappers in `lib/services.ts`:
|
||||
|
||||
```typescript
|
||||
// Example
|
||||
export async function postComplete(payload: CompletePayload) {
|
||||
const res = await fetch(`${PROGRESS_URL}/complete`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify(payload)
|
||||
})
|
||||
if (!res.ok) throw new Error(`Complete failed: ${res.status}`)
|
||||
return res.json() as Promise<CompleteResponse>
|
||||
}
|
||||
```
|
||||
|
||||
All service response types imported from `types/index.ts`.
|
||||
Labels show `Cycle X · Week Y of 26`, where Y/X come from `getCurriculumWeek` /
|
||||
`getCurriculumCycle` applied to `state.weekNumber`.
|
||||
|
||||
---
|
||||
|
||||
## Admin app
|
||||
## Admin panel (`Admin/index.jsx`)
|
||||
|
||||
### Shell layout (`admin/layout.tsx`)
|
||||
|
||||
Sidebar navigation on desktop, top navigation on mobile.
|
||||
|
||||
Nav items:
|
||||
- Documents
|
||||
- Knowledge base
|
||||
- Curriculum
|
||||
- (link back to employee app)
|
||||
|
||||
### Documents page (`admin/documents/page.tsx`)
|
||||
|
||||
**Upload section**
|
||||
- Drag-and-drop file input: accepts .pdf, .md, .txt
|
||||
- On upload: POST file to PocketBase storage →
|
||||
then POST to ingestion service `/ingest` with document metadata
|
||||
- Show upload confirmation with filename
|
||||
|
||||
**Job status list**
|
||||
- Poll GET /status/:jobId every 3 seconds while status is not done/failed
|
||||
- Show per-job progress:
|
||||
- Status badge: queued / extracting / chunking / structuring / embedding / done / failed
|
||||
- Progress bar derived from chunksEmbedded / chunksTotal
|
||||
- On done: "N themes, N topics ready for review" → link to knowledge base
|
||||
- On failed: error reason in red, no retry (admin re-uploads)
|
||||
- Stop polling when status === 'done' or 'failed'
|
||||
|
||||
**Document history**
|
||||
- List of all source_documents from PocketBase
|
||||
- Columns: filename, format, status, ingested_at, chunk_count
|
||||
Tabbed: **Sources** (upload + extraction), **Content** (review/refine generated
|
||||
content), **Quizzes**, **Curriculum** (generate/preview/activate a schedule),
|
||||
**Graph** (D3 knowledge graph + R42 suggestions queue), **Team** (manage members),
|
||||
**Settings** (per-tier model overrides, simulation toggle, smoke-test reset).
|
||||
|
||||
---
|
||||
|
||||
### Knowledge base page (`admin/knowledge/page.tsx`)
|
||||
## Design system
|
||||
|
||||
Lists all Themes with status indicator.
|
||||
|
||||
**Theme card**
|
||||
```
|
||||
[Theme title] [status badge: draft / published]
|
||||
N topics · from: filename.pdf
|
||||
[Approve batch] [Edit] [Reject]
|
||||
```
|
||||
|
||||
Approve batch:
|
||||
- Calls PocketBase to set theme.status → 'published', all child topics → 'published'
|
||||
- Triggers generation service: POST /generate-all with themeId
|
||||
- Shows toast: "Generation queued for N topics"
|
||||
|
||||
Reject:
|
||||
- Sets theme.status → 'rejected'
|
||||
- Removes from list
|
||||
|
||||
Edit → navigates to `/admin/knowledge/[themeId]`
|
||||
|
||||
**Theme detail page (`admin/knowledge/[themeId]/page.tsx`)**
|
||||
|
||||
Displays all Topics in the Theme as editable cards.
|
||||
|
||||
Topic card fields (all editable inline):
|
||||
- title (text input)
|
||||
- body (textarea — rich enough for paragraphs, no full rich text editor needed)
|
||||
- difficulty (select: introductory / intermediate / advanced)
|
||||
- key_terms (tag input — comma-separated)
|
||||
- related_topics (multi-select from published topics)
|
||||
- prerequisite_topics (multi-select)
|
||||
|
||||
Save button per card — calls PocketBase PATCH on the topic record.
|
||||
|
||||
Below topic list: [Approve batch] button — approves all topics in the theme.
|
||||
|
||||
**Micro learning generation status**
|
||||
After batch approval, show generation status per topic:
|
||||
|
||||
```
|
||||
Concept explainer ✓ published
|
||||
Scenario quiz ⏳ generating
|
||||
Comparison card ✓ published
|
||||
...
|
||||
```
|
||||
|
||||
Poll micro_learnings collection filtered by topic until all 10 are published.
|
||||
- CSS variables in `src/index.css`: colors (`--color-bg`, `--color-paper`,
|
||||
`--color-teal`, `--color-accent`), radii (`--r-sm`, `--r-lg`, `--r-org`).
|
||||
- Tailwind v4 utilities map to those variables (`bg-teal`, `text-fg-muted`,
|
||||
`border-bg-warm`, …). Avoid raw hex.
|
||||
- `stylesheet.css` (repo root) is the authoritative visual reference — frozen.
|
||||
- Reuse `src/components/ui/` primitives: `Card`, `Button`, `Tag`, `Input`.
|
||||
- Framer Motion for entry animations; mobile-first (target 375px).
|
||||
|
||||
---
|
||||
|
||||
### Curriculum page (`admin/curriculum/page.tsx`)
|
||||
## Build (`vite.config.js`)
|
||||
|
||||
**Current curriculum view**
|
||||
26 weeks displayed as a list. Each week shows:
|
||||
```
|
||||
Week 7
|
||||
[Theme: Holacratic roles]
|
||||
Topics: Role definitions · Circle structure · Lead link responsibilities
|
||||
Estimated: 25 min
|
||||
[Edit week] [Admin notes]
|
||||
```
|
||||
|
||||
**Regeneration banner**
|
||||
When a pending regeneration is queued:
|
||||
```
|
||||
⚠ 8 new topics added. A new curriculum version is ready to preview.
|
||||
[Preview changes] [Confirm regeneration] [Dismiss]
|
||||
```
|
||||
|
||||
Preview: shows proposed schedule with diff highlighting — weeks that changed
|
||||
are highlighted, weeks that stay the same are dimmed.
|
||||
|
||||
Confirm: calls POST /generate confirm on curriculum service →
|
||||
applies new version to all active employees.
|
||||
|
||||
**Drag-to-reorder**
|
||||
Each week row is draggable. Reordering calls PATCH /weeks/:weekId on the
|
||||
curriculum service to swap theme assignments.
|
||||
|
||||
**Admin notes**
|
||||
Inline text input per week — saved to curriculum_weeks.admin_notes.
|
||||
|
||||
---
|
||||
|
||||
## Employee app
|
||||
|
||||
### Shell layout (`app/layout.tsx`)
|
||||
|
||||
Bottom navigation bar (mobile-first):
|
||||
|
||||
```
|
||||
[Session] [Library] [Profile]
|
||||
```
|
||||
|
||||
R42 floating button: fixed position, bottom-right, above the nav bar.
|
||||
Z-index above all content.
|
||||
|
||||
### Session page (`app/session/page.tsx`)
|
||||
|
||||
**Week header**
|
||||
```
|
||||
Week 7 of 26 · Cycle 1
|
||||
[Theme title: Holacratic roles]
|
||||
[Progress bar: N of 26 weeks complete]
|
||||
```
|
||||
|
||||
**Topic list**
|
||||
Each topic in the week's theme rendered as a card:
|
||||
|
||||
```
|
||||
[Topic title]
|
||||
[difficulty badge] [estimated: 10 min]
|
||||
|
||||
Choose how to learn this topic:
|
||||
[Concept explainer] [Scenario quiz] [How-to] ...
|
||||
(only published types shown as buttons)
|
||||
|
||||
[Completed types: ✓ Concept explainer]
|
||||
```
|
||||
|
||||
Selecting a type opens the micro learning inline (no navigation — expands in
|
||||
place on mobile). Employee reads/completes it, then taps [Mark complete].
|
||||
|
||||
On mark complete:
|
||||
- POST to progress service `/complete`
|
||||
- Response displays: commits earned + any new badges as a toast notification
|
||||
- Topic card updates to show type as completed (✓)
|
||||
- All types in topic completable in one session
|
||||
|
||||
**Week complete state**
|
||||
When all topics in the week have at least one completed type:
|
||||
```
|
||||
🚀 Week 7 complete
|
||||
You earned N commits
|
||||
[Continue to Week 8]
|
||||
```
|
||||
|
||||
Continue button calls POST /advance/:userId on curriculum service.
|
||||
|
||||
---
|
||||
|
||||
### Micro learning components
|
||||
|
||||
One component per type in `components/micro-learnings/`.
|
||||
Each receives the `content` JSON field from the micro_learnings record.
|
||||
|
||||
| Component | Key interactions |
|
||||
|---|---|
|
||||
| ConceptExplainer | Render paragraphs + example — read only |
|
||||
| ScenarioQuiz | Select option → reveal explanation — stateful |
|
||||
| Misconceptions | Accordion: tap misconception to reveal correction |
|
||||
| HowTo | Numbered steps — tap step to check it off |
|
||||
| ComparisonCard | Two-column table — swipeable on mobile |
|
||||
| ReflectionPrompt | Open text area → reveal model answer on submit |
|
||||
| FlashcardSet | Flip card interaction — swipe through deck |
|
||||
| CaseStudy | Scenario text + open questions — read only |
|
||||
| GlossaryAnchor | Term card with definition + examples |
|
||||
| MythVsEvidence | Myth card → tap to reveal evidence |
|
||||
|
||||
All components are self-contained. They receive content JSON and emit an
|
||||
`onComplete` callback. They do not call any services directly.
|
||||
|
||||
```typescript
|
||||
type MicroLearningProps = {
|
||||
content: unknown // typed per component
|
||||
onComplete: () => void
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Knowledge library (`app/library/page.tsx`)
|
||||
|
||||
**Browse view**
|
||||
All published topics, grouped by Theme.
|
||||
Search input: filters by title and key_terms in real time (client-side).
|
||||
Filter chips: by difficulty (introductory / intermediate / advanced).
|
||||
|
||||
Each topic shown as a card:
|
||||
```
|
||||
[Topic title]
|
||||
[Theme] · [difficulty badge]
|
||||
[key terms as chips]
|
||||
```
|
||||
|
||||
Tap → navigate to topic detail.
|
||||
|
||||
**Topic detail (`app/library/[topicId]/page.tsx`)**
|
||||
|
||||
```
|
||||
[Topic title]
|
||||
[Theme] · [difficulty]
|
||||
|
||||
[Topic body — rendered as paragraphs]
|
||||
|
||||
Key terms: [chip] [chip] [chip]
|
||||
|
||||
Related topics: [card] [card]
|
||||
Prerequisite for: [card] [card]
|
||||
|
||||
How to learn this topic:
|
||||
[micro learning type buttons — same as session view]
|
||||
```
|
||||
|
||||
Completing a micro learning from the library records the completion via
|
||||
progress service. Week_number is set to the employee's current week.
|
||||
|
||||
---
|
||||
|
||||
### Profile page (`app/profile/page.tsx`)
|
||||
|
||||
**Header**
|
||||
```
|
||||
[Display name]
|
||||
[Level badge: Junior] [N commits]
|
||||
[Current streak: 5 weeks] [Longest: 8 weeks]
|
||||
```
|
||||
|
||||
**Heatmap**
|
||||
GitHub-style contribution graph.
|
||||
26 columns (weeks) × rows implied by completions per week.
|
||||
Cell colour: 0 completions = lightest, 5+ completions = darkest.
|
||||
Tap a cell → tooltip: "Week N · N completions".
|
||||
Scrollable horizontally on mobile if needed.
|
||||
|
||||
Implementation: render as SVG or CSS grid — no charting library required.
|
||||
|
||||
```typescript
|
||||
// Data from GET /profile/:userId → heatmap[]
|
||||
// Colour scale: 4 levels based on completions count
|
||||
// 0: var(--heatmap-0)
|
||||
// 1: var(--heatmap-1)
|
||||
// 2-3: var(--heatmap-2)
|
||||
// 4+: var(--heatmap-3)
|
||||
// Use CSS custom properties — values derived from stylesheet.css palette
|
||||
```
|
||||
|
||||
**Badges**
|
||||
Grid of earned badges. Unearned badges shown as locked (greyed out).
|
||||
Tap badge → tooltip with award condition.
|
||||
|
||||
```
|
||||
🥉 First commit ✓
|
||||
🥈 Five sessions ✓
|
||||
🥇 On a streak 🔒 (13 week streak needed)
|
||||
⭐ Shipped 🔒
|
||||
---
|
||||
🏷 Governance nerd ✓
|
||||
🏷 Deep reader 🔒 (3/5 case studies)
|
||||
```
|
||||
|
||||
**Leaderboard tab**
|
||||
Toggle between "My profile" and "Leaderboard".
|
||||
|
||||
Leaderboard: table of all employees from GET /leaderboard.
|
||||
Columns: Name · Commits · Streak · Types used · Badges · Level.
|
||||
Not ranked 1–N. No sorting by the user — display order is commits descending.
|
||||
Current employee row is highlighted.
|
||||
|
||||
**Activity feed tab**
|
||||
Third tab: "Feed".
|
||||
Milestone cards from GET /feed.
|
||||
Most recent first.
|
||||
|
||||
```
|
||||
🚀 Alex shipped the full curriculum
|
||||
26 weeks · 847 commits · 3 badges
|
||||
Longest streak: 18 weeks
|
||||
[timestamp]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## R42 chatbot components (`components/r42/`)
|
||||
|
||||
### R42Button
|
||||
|
||||
Fixed position, bottom-right, above bottom nav bar.
|
||||
Circle button with R42 label or icon.
|
||||
Tap → opens R42Drawer.
|
||||
|
||||
```tsx
|
||||
// Position: fixed, bottom: calc(nav-height + 16px), right: 16px
|
||||
// Z-index: above all content, below modals
|
||||
```
|
||||
|
||||
### R42Drawer
|
||||
|
||||
Slides up from bottom on mobile (sheet pattern).
|
||||
On desktop: expands to a side panel.
|
||||
|
||||
```
|
||||
[R42 header bar] [close ×]
|
||||
─────────────────────────────────────────────
|
||||
[Response area — scrollable]
|
||||
|
||||
Based on: [Holacratic roles ×] [Circle structure ×]
|
||||
─────────────────────────────────────────────
|
||||
[Type a question...] [Send →]
|
||||
```
|
||||
|
||||
**State machine:**
|
||||
```
|
||||
idle → loading (query sent) → streaming → done
|
||||
↘ out_of_scope
|
||||
```
|
||||
|
||||
**Streaming implementation:**
|
||||
|
||||
```typescript
|
||||
// POST /chat with fetch, read SSE stream
|
||||
const response = await fetch(`${CHAT_URL}/chat`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ query, userId })
|
||||
})
|
||||
|
||||
const reader = response.body!.getReader()
|
||||
const decoder = new TextDecoder()
|
||||
|
||||
while (true) {
|
||||
const { done, value } = await reader.read()
|
||||
if (done) break
|
||||
const lines = decoder.decode(value).split('\n')
|
||||
for (const line of lines) {
|
||||
if (!line.startsWith('data: ')) continue
|
||||
const event = JSON.parse(line.slice(6))
|
||||
if (event.type === 'chunk') appendText(event.text)
|
||||
if (event.type === 'citations') setCitations(event.topics)
|
||||
if (event.type === 'out_of_scope') setOutOfScope(event.text)
|
||||
if (event.type === 'done') setDone()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Citations**
|
||||
Rendered as tappable pills below the response.
|
||||
Tap → closes R42Drawer, navigates to `/app/library/[topicId]`.
|
||||
|
||||
**Out of scope response**
|
||||
Render as a muted message (not an error state):
|
||||
"This doesn't appear to be covered in the knowledge base.
|
||||
You can browse the full library in the Knowledge section."
|
||||
|
||||
**Stateless by design**
|
||||
Conversation cleared on drawer close. No history persisted.
|
||||
Input cleared on send.
|
||||
|
||||
---
|
||||
|
||||
## Mobile-first layout rules
|
||||
|
||||
All layout decisions start at 375px and scale up.
|
||||
|
||||
- Bottom navigation: fixed, height 56px, icons + labels
|
||||
- R42 button: 48px circle, positioned above nav bar
|
||||
- Session topic cards: full width, stack vertically
|
||||
- Micro learning components: full width, no horizontal scroll except
|
||||
ComparisonCard (swipeable)
|
||||
- Heatmap: horizontal scroll container on narrow screens
|
||||
- Leaderboard table: horizontally scrollable on mobile, sticky name column
|
||||
- Drawer/sheet pattern for R42 on mobile, side panel on desktop (breakpoint: 768px)
|
||||
- Tap targets: minimum 44×44px on all interactive elements
|
||||
- No hover-only interactions — all hover states have tap equivalents
|
||||
|
||||
---
|
||||
|
||||
## Environment variables
|
||||
|
||||
```
|
||||
NEXT_PUBLIC_POCKETBASE_URL=http://localhost:8090
|
||||
NEXT_PUBLIC_INGESTION_URL=http://localhost:3001
|
||||
NEXT_PUBLIC_GENERATION_URL=http://localhost:3002
|
||||
NEXT_PUBLIC_CURRICULUM_URL=http://localhost:3003
|
||||
NEXT_PUBLIC_CHAT_URL=http://localhost:3004
|
||||
NEXT_PUBLIC_PROGRESS_URL=http://localhost:3005
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
```json
|
||||
{
|
||||
"dependencies": {
|
||||
"next": "14",
|
||||
"react": "^18",
|
||||
"react-dom": "^18",
|
||||
"pocketbase": "^0.21",
|
||||
"next-pwa": "^5",
|
||||
"zod": "^3"
|
||||
},
|
||||
"devDependencies": {
|
||||
"typescript": "^5",
|
||||
"tailwindcss": "^3",
|
||||
"autoprefixer": "^10",
|
||||
"postcss": "^8",
|
||||
"@types/react": "^18",
|
||||
"@types/node": "^20"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
No component library. No charting library. No drag-and-drop library —
|
||||
implement curriculum drag-to-reorder with native HTML5 drag API.
|
||||
The heatmap is SVG or CSS grid — no D3.
|
||||
|
||||
---
|
||||
|
||||
## TypeScript strict mode requirements
|
||||
|
||||
- No `any` types
|
||||
- All PocketBase collection responses typed against data-model.md schemas
|
||||
- All service API responses typed against response types from each service spec
|
||||
- Micro learning content JSON typed per type using discriminated union:
|
||||
|
||||
```typescript
|
||||
type MicroLearningContent =
|
||||
| { type: 'concept_explainer'; paragraphs: string[]; example: string }
|
||||
| { type: 'scenario_quiz'; scenario: string; options: QuizOption[] }
|
||||
| { type: 'misconceptions'; items: MisconceptionItem[] }
|
||||
// ... all 10 types
|
||||
```
|
||||
|
||||
- SSE event types as discriminated union
|
||||
- No implicit any on event handlers
|
||||
|
||||
---
|
||||
|
||||
## What the frontend does NOT do
|
||||
|
||||
- Does not run AI calls directly — all AI goes through backend services
|
||||
- Does not write to Qdrant — embedding is the ingestion service's responsibility
|
||||
- Does not implement auth logic — delegates entirely to PocketBase SDK
|
||||
- Does not implement curriculum generation — calls curriculum service
|
||||
|
||||
---
|
||||
|
||||
## Testing checkpoints
|
||||
|
||||
### Admin app
|
||||
1. Upload a PDF → ingestion job created → status polls and updates → done state shows link
|
||||
2. Theme batch appears after ingestion → approve → generation queued
|
||||
3. Edit a topic title and body → save → changes persisted in PocketBase
|
||||
4. Curriculum renders 26 weeks → drag week 3 and week 5 → order persists
|
||||
5. Regeneration banner appears → preview shows → confirm applies new version
|
||||
|
||||
### Employee app
|
||||
6. Login as employee → redirected to /app/session → correct week shown
|
||||
7. Select micro learning type → content renders → mark complete → commits toast shown
|
||||
8. Complete all topics in week → week complete state shown → advance to next week
|
||||
9. Library browse → search filters results → topic detail renders body + related topics
|
||||
10. Profile page → heatmap renders for current cycle → badges show locked/unlocked state
|
||||
11. Leaderboard tab → all employees shown → current employee row highlighted
|
||||
12. R42 button visible on every screen → opens drawer → question answered with citations
|
||||
13. R42 citation tap → navigates to correct topic in library
|
||||
14. Out-of-scope question → muted message shown, no citations
|
||||
15. All screens render correctly at 375px width — no horizontal overflow except
|
||||
intentional scroll containers
|
||||
- Injects `__BUILD_SHA__` / `__BUILD_TIME__` (shown by `BuildStamp`).
|
||||
- Dev server proxies `/api/anthropic` to Anthropic and injects `ANTHROPIC_API_KEY`.
|
||||
- Source maps on; production build outputs to `dist/`.
|
||||
|
||||
@@ -1,469 +1,54 @@
|
||||
# Gamification and progress service spec
|
||||
# Gamification spec
|
||||
|
||||
## Responsibility
|
||||
A lightweight points-and-badges layer that motivates weekly completion. There is
|
||||
no separate progress service — points are written by the test flow and badges are
|
||||
computed at render time.
|
||||
|
||||
Records session completions, calculates XP (commits), manages levels and
|
||||
streaks, evaluates badge conditions, generates milestone cards, and serves
|
||||
leaderboard data. All gamification data is public to all employees.
|
||||
- **Ledger:** the `leaderboard` collection
|
||||
- **Award path:** `src/lib/testService.js → saveTestResult`
|
||||
- **Display:** `src/pages/Leaderboard.jsx` (+ the Dashboard mini-leaderboard)
|
||||
|
||||
---
|
||||
|
||||
## Service location
|
||||
## Points
|
||||
|
||||
```
|
||||
app/services/progress/
|
||||
├── src/
|
||||
│ ├── index.ts entry point, Fastify server
|
||||
│ ├── routes/
|
||||
│ │ ├── completions.ts POST /complete
|
||||
│ │ ├── profile.ts GET /profile/:userId
|
||||
│ │ ├── leaderboard.ts GET /leaderboard
|
||||
│ │ └── feed.ts GET /feed
|
||||
│ ├── engine/
|
||||
│ │ ├── xp.ts commit calculation per type
|
||||
│ │ ├── level.ts level thresholds + promotion
|
||||
│ │ ├── streak.ts streak evaluation
|
||||
│ │ ├── badges.ts badge condition evaluation
|
||||
│ │ └── milestone.ts milestone card generation
|
||||
│ └── lib/
|
||||
│ └── pocketbase.ts
|
||||
├── package.json
|
||||
├── tsconfig.json
|
||||
└── .env.example
|
||||
```
|
||||
- **+2 points per correct quiz answer** (`pointsEarned = score * 2`).
|
||||
- Written via `db.upsertLeaderboardEntry(userId, name, pointsDelta, testsCompletedDelta)`,
|
||||
which increments `points` and `tests_completed`.
|
||||
|
||||
`leaderboard` fields: `user_id`, `name`, `points`, `tests_completed`,
|
||||
`learnings_completed` (reserved).
|
||||
|
||||
---
|
||||
|
||||
## API surface
|
||||
## Badges (computed at render time)
|
||||
|
||||
### POST /complete
|
||||
Defined in `Leaderboard.jsx`; not stored in the database:
|
||||
|
||||
Called by the frontend when an employee completes a micro learning.
|
||||
|
||||
Request:
|
||||
```json
|
||||
{
|
||||
"userId": "string",
|
||||
"topicId": "string",
|
||||
"microLearningId": "string",
|
||||
"microLearningType": "string",
|
||||
"weekNumber": 7,
|
||||
"cycle": 1
|
||||
}
|
||||
```
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"commitsEarned": 15,
|
||||
"totalCommits": 342,
|
||||
"levelBefore": "junior",
|
||||
"levelAfter": "junior",
|
||||
"streakWeeks": 5,
|
||||
"newBadges": [
|
||||
{ "key": "deep_reader", "label": "Deep reader", "tier": "content" }
|
||||
],
|
||||
"milestoneCard": null
|
||||
}
|
||||
```
|
||||
|
||||
`newBadges` is empty array if no badges earned this completion.
|
||||
`milestoneCard` is populated only at weeks 13 and 26.
|
||||
| Badge | Condition |
|
||||
|---|---|
|
||||
| First Steps | `tests_completed > 0` |
|
||||
| Veteran | `tests_completed >= 5` |
|
||||
| Perfectionist | any test scored 100% (`perfectScores > 0`) |
|
||||
|
||||
---
|
||||
|
||||
### GET /profile/:userId
|
||||
## Leaderboard
|
||||
|
||||
Returns full gamification profile for an employee.
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"userId": "string",
|
||||
"displayName": "string",
|
||||
"totalCommits": 342,
|
||||
"level": "junior",
|
||||
"currentStreakWeeks": 5,
|
||||
"longestStreakWeeks": 8,
|
||||
"typesUsed": ["concept_explainer", "scenario_quiz", "how_to"],
|
||||
"typesNotUsed": ["case_study", "myth_vs_evidence"],
|
||||
"badges": [
|
||||
{ "key": "bronze_1", "label": "First commit", "tier": "bronze", "earnedAt": "..." }
|
||||
],
|
||||
"heatmap": [
|
||||
{ "week": 1, "completions": 3 },
|
||||
{ "week": 2, "completions": 0 }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Heatmap returns 26 entries per cycle. `completions` = number of micro
|
||||
learning types completed that week.
|
||||
- Admins are filtered out of the public board.
|
||||
- All non-admin members appear, even with 0 points.
|
||||
- Sorted by `points` descending; top 3 shown as a podium, the rest as a ranked list
|
||||
with their earned badges. The current user is highlighted.
|
||||
- The Dashboard shows a compact top-3 + the viewer's rank.
|
||||
|
||||
---
|
||||
|
||||
### GET /leaderboard
|
||||
|
||||
Returns all employee gamification profiles for leaderboard rendering.
|
||||
Not paginated at 150 employees.
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"employees": [
|
||||
{
|
||||
"userId": "string",
|
||||
"displayName": "string",
|
||||
"totalCommits": 847,
|
||||
"currentStreakWeeks": 18,
|
||||
"typesUsedCount": 9,
|
||||
"badgeCount": 5,
|
||||
"level": "senior"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Sorted by totalCommits descending. Frontend renders as multi-dimension
|
||||
table — not a ranked list.
|
||||
|
||||
---
|
||||
|
||||
### GET /feed
|
||||
|
||||
Returns recent milestone cards for the public activity feed.
|
||||
Most recent first.
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"milestones": [
|
||||
{
|
||||
"userId": "string",
|
||||
"displayName": "string",
|
||||
"cycle": 1,
|
||||
"week": 26,
|
||||
"totalCommits": 847,
|
||||
"streakWeeks": 18,
|
||||
"badges": ["on_streak", "shipped"],
|
||||
"createdAt": "string"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## XP (commits) calculation
|
||||
|
||||
Each micro learning type earns a different number of commits based on
|
||||
cognitive effort required.
|
||||
|
||||
```typescript
|
||||
const COMMITS_PER_TYPE: Record<MicroLearningType, number> = {
|
||||
concept_explainer: 10,
|
||||
glossary_anchor: 10,
|
||||
misconceptions: 15,
|
||||
how_to: 15,
|
||||
flashcard_set: 15,
|
||||
comparison_card: 20,
|
||||
reflection_prompt: 20,
|
||||
scenario_quiz: 25,
|
||||
myth_vs_evidence: 25,
|
||||
case_study: 30,
|
||||
}
|
||||
```
|
||||
|
||||
First completion of a type the employee has never used before: +5 bonus
|
||||
commits (rewards type diversity).
|
||||
|
||||
Duplicate completion (same topic + same type in same cycle): 0 commits.
|
||||
Check session_completions before awarding — idempotent.
|
||||
|
||||
---
|
||||
|
||||
## Levels
|
||||
|
||||
Thresholds based on cumulative commits across all cycles.
|
||||
|
||||
```typescript
|
||||
const LEVEL_THRESHOLDS = {
|
||||
intern: 0,
|
||||
junior: 100,
|
||||
medior: 300,
|
||||
senior: 600,
|
||||
staff: 1000,
|
||||
principal: 1500,
|
||||
}
|
||||
```
|
||||
|
||||
Level evaluated after every commit update. Level can only increase —
|
||||
never decreases between cycles.
|
||||
|
||||
---
|
||||
|
||||
## Streak
|
||||
|
||||
Counts consecutive weeks with at least one completion.
|
||||
|
||||
Evaluation logic on every POST /complete:
|
||||
|
||||
```typescript
|
||||
const lastActiveWeek = profile.last_active_week
|
||||
const currentWeek = completionWeekNumber
|
||||
|
||||
if (currentWeek === lastActiveWeek) {
|
||||
// Same week — streak unchanged
|
||||
} else if (currentWeek === lastActiveWeek + 1) {
|
||||
// Consecutive — increment
|
||||
streak += 1
|
||||
} else {
|
||||
// Gap — reset to 1
|
||||
streak = 1
|
||||
}
|
||||
|
||||
profile.last_active_week = currentWeek
|
||||
profile.longest_streak_weeks = Math.max(streak, profile.longest_streak_weeks)
|
||||
```
|
||||
|
||||
Week number resets to 1 on cycle start. Streak does not reset on cycle
|
||||
transition — a streak spanning the cycle boundary is maintained.
|
||||
|
||||
---
|
||||
|
||||
## Badges
|
||||
|
||||
### Badge seed data
|
||||
|
||||
Seeded into PocketBase badges collection at startup.
|
||||
|
||||
```typescript
|
||||
const BADGE_DEFINITIONS = [
|
||||
{ key: 'first_commit', tier: 'bronze',
|
||||
label: 'First commit',
|
||||
description: 'Complete any session' },
|
||||
|
||||
{ key: 'five_sessions', tier: 'silver',
|
||||
label: 'Five sessions',
|
||||
description: 'Complete 5 sessions using 5 different micro learning types' },
|
||||
|
||||
{ key: 'on_streak', tier: 'gold',
|
||||
label: 'On a streak',
|
||||
description: 'Complete 13 sessions without skipping a week' },
|
||||
|
||||
{ key: 'shipped', tier: 'legendary',
|
||||
label: 'Shipped',
|
||||
description: 'Complete all 26 sessions using all 10 micro learning types' },
|
||||
|
||||
{ key: 'governance_nerd', tier: 'content',
|
||||
label: 'Governance nerd',
|
||||
description: 'Complete all topics in the holacratic structure theme' },
|
||||
|
||||
{ key: 'process_architect', tier: 'content',
|
||||
label: 'Process architect',
|
||||
description: 'Complete all topics in the internal processes theme' },
|
||||
|
||||
{ key: 'deep_reader', tier: 'content',
|
||||
label: 'Deep reader',
|
||||
description: 'Use the case study micro learning type 5 or more times' },
|
||||
|
||||
{ key: 'handbook_expert', tier: 'content',
|
||||
label: 'Handbook expert',
|
||||
description: 'Complete all topics in the employee handbook theme' },
|
||||
|
||||
{ key: 'type_collector', tier: 'content',
|
||||
label: 'Type collector',
|
||||
description: 'Use all 10 micro learning types at least once' },
|
||||
]
|
||||
```
|
||||
|
||||
Content badges are theme-specific. Theme association resolved at runtime
|
||||
by matching badge key to theme title pattern — not hardcoded to theme IDs.
|
||||
|
||||
```typescript
|
||||
const THEME_BADGE_PATTERNS: Record<string, string> = {
|
||||
'governance_nerd': 'holacrat',
|
||||
'process_architect': 'process',
|
||||
'handbook_expert': 'handbook',
|
||||
}
|
||||
```
|
||||
|
||||
Case-insensitive substring match on theme title.
|
||||
|
||||
---
|
||||
|
||||
### Badge evaluation
|
||||
|
||||
Run after every POST /complete. Check all conditions, award unearned badges.
|
||||
|
||||
```typescript
|
||||
async function evaluateBadges(userId: string, profile: GamificationProfile):
|
||||
Promise<BadgeDefinition[]> {
|
||||
|
||||
const earnedKeys = await getEarnedBadgeKeys(userId)
|
||||
const newBadges: string[] = []
|
||||
|
||||
if (!earnedKeys.includes('first_commit')) {
|
||||
const count = await countCompletions(userId)
|
||||
if (count >= 1) newBadges.push('first_commit')
|
||||
}
|
||||
|
||||
if (!earnedKeys.includes('five_sessions')) {
|
||||
const sessions = await countUniqueSessions(userId)
|
||||
if (sessions >= 5 && profile.types_used.length >= 5)
|
||||
newBadges.push('five_sessions')
|
||||
}
|
||||
|
||||
if (!earnedKeys.includes('on_streak')) {
|
||||
if (profile.longest_streak_weeks >= 13) newBadges.push('on_streak')
|
||||
}
|
||||
|
||||
if (!earnedKeys.includes('shipped')) {
|
||||
const cycleComplete = await isCycleComplete(userId, profile.current_cycle)
|
||||
if (cycleComplete && profile.types_used.length === 10)
|
||||
newBadges.push('shipped')
|
||||
}
|
||||
|
||||
if (!earnedKeys.includes('deep_reader')) {
|
||||
const count = await countTypeCompletions(userId, 'case_study')
|
||||
if (count >= 5) newBadges.push('deep_reader')
|
||||
}
|
||||
|
||||
if (!earnedKeys.includes('type_collector')) {
|
||||
if (profile.types_used.length === 10) newBadges.push('type_collector')
|
||||
}
|
||||
|
||||
for (const [badgeKey, pattern] of Object.entries(THEME_BADGE_PATTERNS)) {
|
||||
if (!earnedKeys.includes(badgeKey)) {
|
||||
const complete = await isThemeComplete(userId, pattern)
|
||||
if (complete) newBadges.push(badgeKey)
|
||||
}
|
||||
}
|
||||
|
||||
for (const key of newBadges) {
|
||||
await awardBadge(userId, key, profile.current_cycle)
|
||||
}
|
||||
|
||||
return newBadges.map(k => BADGE_DEFINITIONS.find(b => b.key === k)!)
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Milestone cards
|
||||
|
||||
Generated at weeks 13 and 26 of each cycle.
|
||||
|
||||
```typescript
|
||||
async function generateMilestoneCard(
|
||||
userId: string,
|
||||
weekNumber: number,
|
||||
cycle: number,
|
||||
profile: GamificationProfile
|
||||
): Promise<MilestoneCard | null> {
|
||||
|
||||
if (weekNumber !== 13 && weekNumber !== 26) return null
|
||||
|
||||
const exists = await milestoneExists(userId, cycle, weekNumber)
|
||||
if (exists) return null
|
||||
|
||||
const badges = await getEarnedBadges(userId)
|
||||
|
||||
return await pocketbase.collection('milestone_cards').create({
|
||||
user: userId,
|
||||
cycle,
|
||||
week: weekNumber,
|
||||
total_commits: profile.total_commits,
|
||||
streak_weeks: profile.current_streak_weeks,
|
||||
badge_keys: badges.map(b => b.key),
|
||||
created_at: new Date().toISOString()
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Heatmap data
|
||||
|
||||
Built from session_completions filtered by userId + cycle.
|
||||
Returns 26 entries — one per week.
|
||||
|
||||
```typescript
|
||||
async function buildHeatmap(userId: string, cycle: number):
|
||||
Promise<HeatmapEntry[]> {
|
||||
|
||||
const completions = await pocketbase
|
||||
.collection('session_completions')
|
||||
.getFullList({ filter: `user="${userId}" && cycle=${cycle}` })
|
||||
|
||||
const byWeek = groupBy(completions, c => c.week_number)
|
||||
|
||||
return Array.from({ length: 26 }, (_, i) => ({
|
||||
week: i + 1,
|
||||
completions: byWeek[i + 1]?.length ?? 0
|
||||
}))
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Environment variables
|
||||
|
||||
```
|
||||
POCKETBASE_URL=
|
||||
POCKETBASE_ADMIN_EMAIL=
|
||||
POCKETBASE_ADMIN_PASSWORD=
|
||||
PROGRESS_PORT=3005
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
```json
|
||||
{
|
||||
"dependencies": {
|
||||
"fastify": "^4",
|
||||
"pocketbase": "^0.21",
|
||||
"zod": "^3"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
No AI calls. No Qdrant. No OpenAI. Pure business logic over PocketBase.
|
||||
|
||||
---
|
||||
|
||||
## TypeScript strict mode requirements
|
||||
|
||||
- No `any` types
|
||||
- MicroLearningType as explicit string union
|
||||
- Badge keys as explicit string union matching BADGE_DEFINITIONS
|
||||
- COMMITS_PER_TYPE keyed by MicroLearningType — compile-time exhaustiveness
|
||||
- HeatmapEntry, MilestoneCard, BadgeDefinition typed explicitly
|
||||
|
||||
---
|
||||
|
||||
## What this service does NOT do
|
||||
|
||||
- Does not generate content
|
||||
- Does not handle curriculum scheduling → curriculum service
|
||||
- Does not serve KB or micro learning data → frontend reads PocketBase
|
||||
- Does not handle auth → PocketBase + frontend
|
||||
|
||||
---
|
||||
|
||||
## Testing checkpoints
|
||||
|
||||
1. POST /complete for new user → first_commit badge awarded, commits added
|
||||
2. POST /complete same topic + type twice → 0 commits second call (idempotent)
|
||||
3. Complete 5 sessions with 5 types → five_sessions badge awarded
|
||||
4. Simulate 13 consecutive weekly completions → on_streak badge awarded
|
||||
5. Skip a week → streak resets to 1
|
||||
6. Complete all topics in a theme → content badge awarded
|
||||
7. Use all 10 types → type_collector badge awarded
|
||||
8. Complete week 13 → milestone_card created and returned in response
|
||||
9. GET /leaderboard → all employees returned with correct fields
|
||||
10. GET /feed → milestone cards ordered most recent first
|
||||
11. Cycle transition: week 26 complete → streak spans boundary, level preserved,
|
||||
heatmap resets for new cycle
|
||||
## Notes & possible extensions
|
||||
|
||||
- The earlier design described developer-themed XP ("commits"), levels
|
||||
(Intern→Principal), weekly streaks, a GitHub-style heatmap, and public milestone
|
||||
cards. **None of those shipped** — the live system is the points + 3-badge model
|
||||
above.
|
||||
- If extending: add fields to `leaderboard` (or a new collection + migration in
|
||||
`pb_migrations/`, mirrored in `scripts/setup-pb-collections.mjs`), increment them
|
||||
on the relevant completion events, and surface them in `Leaderboard.jsx`.
|
||||
|
||||
@@ -1,583 +1,87 @@
|
||||
# Generation service spec
|
||||
# Generation spec: learning content & micro-learnings
|
||||
|
||||
## Responsibility
|
||||
|
||||
Accepts a Theme ID from the admin app (on batch approval) and generates all 10
|
||||
micro learning types for every published Topic in that Theme. One Claude Sonnet 4
|
||||
call per type per topic. All outputs validated through Zod schemas before write.
|
||||
|
||||
This service runs entirely server-side. The admin app calls it via REST. All AI
|
||||
calls go through the Anthropic API. No generation logic lives in the frontend.
|
||||
Two generators turn a topic into learner-facing material. Both go through
|
||||
`callLLM` with forced tool use and Zod-validated output. All content is cached in
|
||||
PocketBase so it is generated once per topic/type.
|
||||
|
||||
---
|
||||
|
||||
## Service location
|
||||
## A. Long-form content — `src/lib/learningService.js`
|
||||
|
||||
```
|
||||
app/services/generation/
|
||||
├── src/
|
||||
│ ├── index.ts entry point, Fastify server
|
||||
│ ├── routes/
|
||||
│ │ ├── generate.ts POST /generate, GET /status/:jobId
|
||||
│ │ └── publish.ts PATCH /micro-learnings/:id
|
||||
│ ├── pipeline/
|
||||
│ │ └── generate.ts per-type generation logic
|
||||
│ ├── jobs/
|
||||
│ │ └── queue.ts async job queue (in-memory)
|
||||
│ ├── lib/
|
||||
│ │ ├── pocketbase.ts PocketBase client
|
||||
│ │ └── anthropic.ts Anthropic client
|
||||
│ └── types.ts shared TypeScript types + Zod schemas
|
||||
├── package.json
|
||||
├── tsconfig.json
|
||||
├── .env.example
|
||||
└── .gitignore
|
||||
```
|
||||
Stored in the `content` collection (one record per topic, `data` is a merged
|
||||
object). Three types, generated **on demand**:
|
||||
|
||||
| Type | Tool | Min requirements |
|
||||
|---|---|---|
|
||||
| `article` | `emit_learning_article` | ≥3 sections, ≥2 takeaways |
|
||||
| `slides` | `emit_learning_slides` | ≥4 slides |
|
||||
| `infographic` | `emit_learning_infographic` | ≥3 stats, ≥3 steps |
|
||||
|
||||
`generateLearningContent(topic, force, selectedType)`:
|
||||
- tier `standard`, `maxTokens: 8192`
|
||||
- `selectedType` is one of the three, or `'all'` (`emit_learning_all`) for admin regeneration
|
||||
- cache check looks at `content[selectedType]`; on generation the new payload is
|
||||
**shallow-merged** into the cached object so other types survive
|
||||
- there is **no podcast type**
|
||||
|
||||
**Article refinement** (`refineLearningContent`): the admin describes a change and
|
||||
the model edits via targeted patch tools — `set_intro`, `set_section`,
|
||||
`add_section`, `remove_section`, `replace_takeaways` — so only the affected parts
|
||||
change. Patches are applied and re-validated in `src/lib/articlePatches.js`.
|
||||
|
||||
---
|
||||
|
||||
## API surface
|
||||
## B. Micro-learnings — `src/lib/microLearningService.js`
|
||||
|
||||
### POST /generate
|
||||
Stored in the `micro_learnings` collection (one record per topic per type,
|
||||
`status='published'`). Three types:
|
||||
|
||||
Triggered by admin app when a Theme batch is approved.
|
||||
| Type | Tool | Tier | Shape |
|
||||
|---|---|---|---|
|
||||
| `concept_explainer` | `emit_concept_explainer` | standard | `{ sections: [{ title, content (HTML) }] }`, ≥3 sections |
|
||||
| `scenario_quiz` | `emit_scenario_quiz` | standard | `{ scenario, options: [{ text, isCorrect, explanation }] }`, 3–4 options, exactly 1 correct |
|
||||
| `flashcard_set` | `emit_flashcard_set` | fast (Haiku) | `{ cards: [{ front, back }] }`, 5–10 cards |
|
||||
|
||||
Request:
|
||||
```json
|
||||
{
|
||||
"themeId": "string"
|
||||
}
|
||||
```
|
||||
`getOrGenerateMicroLearning(topicId, type)`:
|
||||
- returns the cached published record if one exists (`findExisting`)
|
||||
- otherwise loads the topic, calls `callLLM` with forced tool choice, and creates a
|
||||
`micro_learnings` record with the validated `content`
|
||||
|
||||
Response (202 Accepted):
|
||||
```json
|
||||
{
|
||||
"jobId": "string",
|
||||
"status": "queued",
|
||||
"topicsFound": 5,
|
||||
"totalItems": 50
|
||||
}
|
||||
```
|
||||
> A former `reflection_prompt` type was dropped. Do not re-add it.
|
||||
|
||||
Processing is async. The admin app polls job status.
|
||||
|
||||
Behaviour:
|
||||
- Fetches all published Topics for the given themeId
|
||||
- Creates one micro_learnings record per topic per type with status `queued`
|
||||
- Generates each item sequentially; updates status to `generated` on success
|
||||
- On failure: sets individual item status to `failed`, continues remaining items
|
||||
- Job completes when all items are either `generated` or `failed`
|
||||
Completion is recorded (append-only) by `useMicroLearningCompletions` into
|
||||
`micro_learning_completions` with `{ team_member_id, micro_learning_id, topic_id,
|
||||
type, session_week }`.
|
||||
|
||||
---
|
||||
|
||||
### GET /status/:jobId
|
||||
## C. Weekly quiz — `src/lib/testService.js`
|
||||
|
||||
Returns current job progress.
|
||||
Generates a 5-question multiple-choice test for the user's current week.
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"jobId": "string",
|
||||
"status": "queued" | "running" | "done" | "failed",
|
||||
"progress": {
|
||||
"topicsTotal": 5,
|
||||
"topicsProcessed": 3,
|
||||
"itemsTotal": 50,
|
||||
"itemsGenerated": 28,
|
||||
"itemsFailed": 2
|
||||
},
|
||||
"error": "string | null"
|
||||
}
|
||||
```
|
||||
- **Topic selection** (`selectTestTopics`): primary topic from the active
|
||||
curriculum week (else hash fallback) + a few review topics for breadth.
|
||||
- **Batch generation** (`callQuizBatchModel`): a single `fast`-tier call
|
||||
(`emit_quiz_questions`, `maxTokens: 4096`, 25s timeout) returns all 5 questions.
|
||||
- **Quality gates** (`validateBatchQuality`): no duplicate options; no banned
|
||||
fillers ("all/none of the above", "both A and B"); explanations ≥20 chars; reject
|
||||
if `correctIndex` is dominated by one position (>80%) and re-roll.
|
||||
- **Scoring** (`saveTestResult`): `pointsEarned = score * 2`, written to
|
||||
`leaderboard` via `db.upsertLeaderboardEntry`.
|
||||
|
||||
Question shape: `{ id, question, topicLabel, options[4], correctIndex (0–3),
|
||||
explanation, difficulty }`.
|
||||
|
||||
---
|
||||
|
||||
### PATCH /micro-learnings/:id
|
||||
|
||||
Admin publishes or rejects an individual micro learning.
|
||||
|
||||
Request:
|
||||
```json
|
||||
{
|
||||
"status": "published" | "rejected"
|
||||
}
|
||||
```
|
||||
|
||||
Response (200 OK):
|
||||
```json
|
||||
{
|
||||
"id": "string",
|
||||
"status": "published" | "rejected",
|
||||
"published_at": "datetime | null"
|
||||
}
|
||||
```
|
||||
|
||||
Rules:
|
||||
- Only `generated` records can be published or rejected
|
||||
- `published_at` set on publish, left null on reject
|
||||
- Returns 400 if record is not in `generated` status
|
||||
- Returns 404 if record not found
|
||||
|
||||
---
|
||||
|
||||
## Generation pipeline
|
||||
|
||||
### Input
|
||||
|
||||
For each Topic in the approved Theme:
|
||||
```
|
||||
topic.title: string
|
||||
topic.body: string
|
||||
topic.key_terms: string[]
|
||||
topic.difficulty: 'introductory' | 'intermediate' | 'advanced'
|
||||
```
|
||||
|
||||
### Output
|
||||
|
||||
10 micro_learnings records per topic, one per type.
|
||||
|
||||
---
|
||||
|
||||
## AI call configuration
|
||||
|
||||
```typescript
|
||||
{
|
||||
model: 'claude-sonnet-4-20250514',
|
||||
max_tokens: 2000,
|
||||
temperature: 0 // deterministic structured output
|
||||
}
|
||||
```
|
||||
|
||||
One call per type per topic. Do not batch multiple types into one call — isolated
|
||||
calls are easier to retry and validate independently.
|
||||
|
||||
---
|
||||
|
||||
## Prompt strategy
|
||||
|
||||
### System prompt (all types)
|
||||
|
||||
```
|
||||
You are a learning content designer. Your task is to generate structured learning
|
||||
content for a specific topic in an employee learning platform.
|
||||
|
||||
Output ONLY valid JSON matching the schema provided. No preamble, no explanation,
|
||||
no markdown fences.
|
||||
|
||||
The content should be accurate, practical, and appropriate for the stated
|
||||
difficulty level. Tone: professional but accessible.
|
||||
```
|
||||
|
||||
### User prompt template (all types)
|
||||
|
||||
```
|
||||
Topic: {topic.title}
|
||||
Difficulty: {topic.difficulty}
|
||||
Body:
|
||||
{topic.body}
|
||||
|
||||
Key terms: {topic.key_terms.join(', ')}
|
||||
|
||||
Generate a {type_label} for this topic.
|
||||
|
||||
Output schema:
|
||||
{JSON.stringify(schemaDescription)}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Type-specific prompts and schemas
|
||||
|
||||
### concept_explainer
|
||||
|
||||
Type label: `Concept Explainer`
|
||||
|
||||
Schema description:
|
||||
```json
|
||||
{
|
||||
"paragraphs": ["2 to 3 paragraphs explaining the concept in plain language"],
|
||||
"example": "one concrete real-world example"
|
||||
}
|
||||
```
|
||||
|
||||
Zod schema:
|
||||
```typescript
|
||||
z.object({
|
||||
paragraphs: z.array(z.string()).min(2).max(3),
|
||||
example: z.string().min(20)
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### scenario_quiz
|
||||
|
||||
Type label: `Scenario Quiz`
|
||||
|
||||
Schema description:
|
||||
```json
|
||||
{
|
||||
"scenario": "a realistic workplace scenario",
|
||||
"options": [
|
||||
{ "label": "A", "text": "answer text", "correct": false, "explanation": "why" },
|
||||
{ "label": "B", "text": "answer text", "correct": true, "explanation": "why" },
|
||||
{ "label": "C", "text": "answer text", "correct": false, "explanation": "why" },
|
||||
{ "label": "D", "text": "answer text", "correct": false, "explanation": "why" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Rules: exactly 4 options, exactly 1 correct.
|
||||
|
||||
Zod schema:
|
||||
```typescript
|
||||
z.object({
|
||||
scenario: z.string().min(30),
|
||||
options: z.array(z.object({
|
||||
label: z.enum(['A', 'B', 'C', 'D']),
|
||||
text: z.string().min(5),
|
||||
correct: z.boolean(),
|
||||
explanation: z.string().min(10)
|
||||
})).length(4).refine(
|
||||
opts => opts.filter(o => o.correct).length === 1,
|
||||
{ message: 'exactly one correct option required' }
|
||||
)
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### misconceptions
|
||||
|
||||
Type label: `Misconceptions`
|
||||
|
||||
Schema description:
|
||||
```json
|
||||
{
|
||||
"items": [
|
||||
{ "misconception": "common wrong belief", "correction": "accurate explanation" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Rules: 3 to 5 items.
|
||||
|
||||
Zod schema:
|
||||
```typescript
|
||||
z.object({
|
||||
items: z.array(z.object({
|
||||
misconception: z.string().min(10),
|
||||
correction: z.string().min(10)
|
||||
})).min(3).max(5)
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### how_to
|
||||
|
||||
Type label: `How-To Guide`
|
||||
|
||||
Schema description:
|
||||
```json
|
||||
{
|
||||
"steps": [
|
||||
{ "number": 1, "instruction": "what to do" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Rules: 3 to 8 steps.
|
||||
|
||||
Zod schema:
|
||||
```typescript
|
||||
z.object({
|
||||
steps: z.array(z.object({
|
||||
number: z.number().int().positive(),
|
||||
instruction: z.string().min(10)
|
||||
})).min(3).max(8)
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### comparison_card
|
||||
|
||||
Type label: `Comparison Card`
|
||||
|
||||
Schema description:
|
||||
```json
|
||||
{
|
||||
"subject_a": "first concept or approach",
|
||||
"subject_b": "second concept or approach",
|
||||
"dimensions": [
|
||||
{ "label": "dimension name", "a": "how A differs", "b": "how B differs" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Rules: 3 to 6 dimensions.
|
||||
|
||||
Zod schema:
|
||||
```typescript
|
||||
z.object({
|
||||
subject_a: z.string().min(2),
|
||||
subject_b: z.string().min(2),
|
||||
dimensions: z.array(z.object({
|
||||
label: z.string().min(2),
|
||||
a: z.string().min(5),
|
||||
b: z.string().min(5)
|
||||
})).min(3).max(6)
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### reflection_prompt
|
||||
|
||||
Type label: `Reflection Prompt`
|
||||
|
||||
Schema description:
|
||||
```json
|
||||
{
|
||||
"prompt": "open-ended question for the employee to reflect on",
|
||||
"model_answer": "a thoughtful example answer the employee can compare against"
|
||||
}
|
||||
```
|
||||
|
||||
Zod schema:
|
||||
```typescript
|
||||
z.object({
|
||||
prompt: z.string().min(20),
|
||||
model_answer: z.string().min(50)
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### flashcard_set
|
||||
|
||||
Type label: `Flashcard Set`
|
||||
|
||||
Schema description:
|
||||
```json
|
||||
{
|
||||
"cards": [
|
||||
{ "question": "question text", "answer": "answer text" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Rules: 5 to 10 cards.
|
||||
|
||||
Zod schema:
|
||||
```typescript
|
||||
z.object({
|
||||
cards: z.array(z.object({
|
||||
question: z.string().min(5),
|
||||
answer: z.string().min(5)
|
||||
})).min(5).max(10)
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### case_study
|
||||
|
||||
Type label: `Case Study`
|
||||
|
||||
Schema description:
|
||||
```json
|
||||
{
|
||||
"scenario": "a detailed real-world scenario (150+ words)",
|
||||
"questions": ["discussion or reflection question 1", "discussion or reflection question 2"]
|
||||
}
|
||||
```
|
||||
|
||||
Rules: 2 to 4 questions.
|
||||
|
||||
Zod schema:
|
||||
```typescript
|
||||
z.object({
|
||||
scenario: z.string().min(150),
|
||||
questions: z.array(z.string().min(10)).min(2).max(4)
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### glossary_anchor
|
||||
|
||||
Type label: `Glossary Anchor`
|
||||
|
||||
Schema description:
|
||||
```json
|
||||
{
|
||||
"term": "the key term",
|
||||
"definition": "precise definition",
|
||||
"correct_use": "example sentence showing correct use",
|
||||
"misuse": "common incorrect usage to avoid"
|
||||
}
|
||||
```
|
||||
|
||||
Prompt addition: use the first key term from `topic.key_terms` as the anchor term.
|
||||
|
||||
Zod schema:
|
||||
```typescript
|
||||
z.object({
|
||||
term: z.string().min(2),
|
||||
definition: z.string().min(20),
|
||||
correct_use: z.string().min(20),
|
||||
misuse: z.string().min(20)
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### myth_vs_evidence
|
||||
|
||||
Type label: `Myth vs Evidence`
|
||||
|
||||
Schema description:
|
||||
```json
|
||||
{
|
||||
"myth": "a commonly held misconception about this topic",
|
||||
"evidence": "the evidence-based counterpoint",
|
||||
"sources": ["source or reference if applicable — leave empty array if none"]
|
||||
}
|
||||
```
|
||||
|
||||
Zod schema:
|
||||
```typescript
|
||||
z.object({
|
||||
myth: z.string().min(20),
|
||||
evidence: z.string().min(30),
|
||||
sources: z.array(z.string())
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Error handling
|
||||
|
||||
**Per item:**
|
||||
- JSON parse failure → retry once with stricter prompt ("respond with valid JSON only, no other text")
|
||||
- Second failure → set micro_learning status to `failed`, log raw response, continue to next item
|
||||
- Zod validation failure → same as parse failure: retry once, then `failed`
|
||||
- Anthropic API error (rate limit / timeout) → exponential backoff, 3 retries, then `failed`
|
||||
|
||||
**Per job:**
|
||||
- If all items for a topic fail → log, continue to next topic
|
||||
- Job status becomes `done` when all items processed, regardless of individual failures
|
||||
- Job status becomes `failed` only if the initial topic fetch fails (PocketBase error before generation starts)
|
||||
|
||||
---
|
||||
|
||||
## PocketBase write
|
||||
|
||||
For each generated item:
|
||||
```typescript
|
||||
{
|
||||
topic: topicId,
|
||||
type: type, // one of the 10 type enum values
|
||||
content: validatedContent, // JSON, validated by Zod
|
||||
status: 'generated',
|
||||
generation_model: 'claude-sonnet-4-20250514',
|
||||
generated_at: new Date().toISOString()
|
||||
}
|
||||
```
|
||||
|
||||
Create record with status `queued` before generation starts.
|
||||
Update to `generated` (with content) or `failed` after attempt.
|
||||
|
||||
---
|
||||
|
||||
## Job lifecycle
|
||||
|
||||
```
|
||||
POST /generate received
|
||||
↓
|
||||
Fetch published Topics for Theme
|
||||
↓
|
||||
Create micro_learning records: status = queued
|
||||
↓
|
||||
Job created → status: running
|
||||
↓
|
||||
For each topic:
|
||||
For each of 10 types:
|
||||
Claude call → validate → write content → status = generated
|
||||
↓
|
||||
All items processed
|
||||
↓
|
||||
Job status: done
|
||||
```
|
||||
|
||||
On topic fetch failure:
|
||||
```
|
||||
status: failed
|
||||
error: { reason: 'topic_fetch_failed', detail: ... }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Environment variables required
|
||||
|
||||
```
|
||||
ANTHROPIC_API_KEY=
|
||||
POCKETBASE_URL=
|
||||
POCKETBASE_ADMIN_EMAIL=
|
||||
POCKETBASE_ADMIN_PASSWORD=
|
||||
GENERATION_PORT=3002
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
```json
|
||||
{
|
||||
"dependencies": {
|
||||
"fastify": "^4",
|
||||
"@anthropic-ai/sdk": "^0.24",
|
||||
"pocketbase": "^0.21",
|
||||
"uuid": "^9",
|
||||
"zod": "^3"
|
||||
},
|
||||
"devDependencies": {
|
||||
"typescript": "^5",
|
||||
"@types/node": "^20",
|
||||
"tsx": "^4"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## TypeScript strict mode requirements
|
||||
|
||||
- No `any` types
|
||||
- All Claude response parsing through Zod schema validation before PocketBase write
|
||||
- All PocketBase writes typed against micro_learnings schema from data-model.md
|
||||
- Content type is `unknown` after JSON.parse — always validate through Zod before use
|
||||
|
||||
---
|
||||
|
||||
## What this service does NOT do
|
||||
|
||||
- Does not extract or chunk source documents → ingestion service
|
||||
- Does not build or schedule the curriculum → curriculum service
|
||||
- Does not handle admin auth → PocketBase + admin app
|
||||
- Does not embed content into Qdrant → ingestion service handles all embeddings
|
||||
- Does not serve R42 queries → chat service
|
||||
|
||||
---
|
||||
|
||||
## Testing checkpoints
|
||||
|
||||
1. Call POST /generate with a themeId that has 2 published topics → verify 20 micro_learning records created
|
||||
2. All 10 types generated for each topic → verify content JSON parses correctly
|
||||
3. All Zod schemas pass for each of the 10 types
|
||||
4. PATCH /micro-learnings/:id with `published` → verify status + published_at updated
|
||||
5. PATCH /micro-learnings/:id with `rejected` → verify status updated, published_at null
|
||||
6. Force a JSON parse error (mock) → verify retry logic fires once, then sets status to `failed`
|
||||
7. GET /status/:jobId during processing → verify progress counters increment correctly
|
||||
## Shared infrastructure (`src/lib/llm.js`)
|
||||
|
||||
- **Tiers:** `fast` (Haiku 4.5), `standard` (Sonnet 4.6), `reasoning` (Opus 4.7);
|
||||
per-tier admin overrides via `admin:model:{tier}`.
|
||||
- **Structured output:** prefer tool use with forced `toolChoice`; inputs validated
|
||||
by `toolSchemaRegistry`. Text responses go through `parseStructuredText`.
|
||||
- **Caching:** wrap stable system text with `cachedSystem(...)`.
|
||||
- **Retry/limits:** `src/lib/llmRetry.js` — backoff + jitter on 408/425/429/5xx/529,
|
||||
honors `Retry-After`, rate limiters for bulk work.
|
||||
- **Telemetry:** every call logged to `llm_calls`.
|
||||
- **Simulation:** with `admin:use_simulation`, calls return stub output (no API hit).
|
||||
|
||||
291
docs/handover.md
291
docs/handover.md
@@ -1,231 +1,114 @@
|
||||
# Handover: employee learning platform
|
||||
# Handover: Respellion Learning Platform
|
||||
|
||||
## Purpose of this document
|
||||
|
||||
This document captures every design decision made before implementation started.
|
||||
It is the authoritative source for rationale. When a spec file is ambiguous,
|
||||
resolve it against this document. Do not ask the human — the answers are here.
|
||||
This document captures the **design decisions as actually built**. The platform
|
||||
diverged substantially from its original design vision (a Next.js multi-service
|
||||
system with Qdrant and OpenAI embeddings). This handover reflects what shipped:
|
||||
a React/Vite SPA on PocketBase with local TF-IDF retrieval.
|
||||
|
||||
When sources conflict, trust the code in `src/` first, then this document, then
|
||||
`docs/data-model.md` (schema), then `docs/architecture.md`.
|
||||
|
||||
---
|
||||
|
||||
## What this application does
|
||||
|
||||
Employees of a tech company use this platform to build and maintain knowledge of
|
||||
the employee handbook, holacratic structures, and internal processes.
|
||||
Employees use the platform to build and maintain knowledge of the company's
|
||||
internal handbook, roles, and processes.
|
||||
|
||||
Core mechanics:
|
||||
- Admins upload source documents → AI extracts a structured knowledge base
|
||||
- The KB is organised into Themes (broad) and Topics (specific)
|
||||
- An AI generates 10 types of micro learning content per Topic
|
||||
- Employees follow a 26-week curriculum of weekly sessions
|
||||
- Each session covers one Theme (multiple related Topics)
|
||||
- Employees choose which micro learning type to use per Topic
|
||||
- After 26 weeks the curriculum restarts, varied to reinforce rather than repeat
|
||||
- A chatbot called R42 answers KB-grounded questions on every screen
|
||||
- A gamification system using developer-native language motivates completion
|
||||
- Admins upload source documents → Claude extracts a structured knowledge graph (topics + relations)
|
||||
- AI generates learning content and micro-learnings per topic
|
||||
- Each employee follows a 26-week curriculum, **starting whenever they enroll**
|
||||
- Each week presents an assigned topic; the employee completes micro-learnings and a test
|
||||
- After week 26 the cycle restarts at week 1 with the same content
|
||||
- R42, an AI assistant, answers KB-grounded questions on every screen
|
||||
- A gamification layer (points, badges, leaderboard) motivates completion
|
||||
|
||||
---
|
||||
|
||||
## All confirmed design decisions
|
||||
## Key decisions as built
|
||||
|
||||
### Architecture
|
||||
- **Single-page React app, not microservices.** All logic runs in the browser
|
||||
(`src/`). PocketBase is the only backend; the Anthropic API is reached through a
|
||||
reverse proxy (Caddy in prod, Vite in dev). The original `app/` Next.js scaffold
|
||||
was abandoned and is not deployed.
|
||||
- **PocketBase for everything stateful** — auth, structured data, file storage.
|
||||
SQLite is sufficient at this scale.
|
||||
- **No vector database.** Retrieval is a dependency-free TF-IDF index over the
|
||||
knowledge graph (`src/lib/retrieval.js`). Qdrant and the embedding service from
|
||||
the original design were never built.
|
||||
|
||||
### Knowledge base
|
||||
- **Extracted, not hand-authored.** Admins upload `.txt` / `.md` (≤5 MB). Claude
|
||||
(standard tier) extracts topics and relations chunk by chunk.
|
||||
- **Flat graph, not a Theme→Topic tree.** The KB is `topics` + `relations`. A
|
||||
topic's `theme` is a string used for curriculum grouping, not a separate entity.
|
||||
- **Relation types:** `related_to`, `depends_on`, `part_of`, `executed_by`.
|
||||
- **Topic relevance** (`core` / `standard` / `peripheral` / `exclude`) controls
|
||||
what enters learning/curriculum; `relevance_locked` protects admin overrides on
|
||||
re-ingestion.
|
||||
|
||||
**Decision: KB is extracted from source documents, not manually authored**
|
||||
Admins upload raw source material. Claude Sonnet 4 extracts Themes, Topics, and
|
||||
relationships. Admins review and approve in batches (one Theme at a time, not
|
||||
one Topic at a time). Topic bodies are AI-drafted and admin-editable after approval.
|
||||
|
||||
**Decision: two-level hierarchy — Theme → Topic**
|
||||
A Theme is a broad subject area. A Topic is one specific concept within a Theme.
|
||||
One weekly session = one Theme. Multiple Topics within that Theme per session.
|
||||
|
||||
**Decision: three relationship types between Topics**
|
||||
- related: Topics that complement each other
|
||||
- prerequisite: Topic A should be understood before Topic B
|
||||
- contrast: Topics representing opposing approaches
|
||||
|
||||
These relationships are stored as explicit PocketBase relation fields, not a
|
||||
generic junction table.
|
||||
|
||||
**Decision: source material format priority**
|
||||
Accepted formats: PDF, MD, TXT only. MD is the highest quality input —
|
||||
heading structure maps directly to Theme → Topic hierarchy. Admins should be
|
||||
recommended to provide MD where possible.
|
||||
|
||||
**Decision: embeddings from source chunks, not topic summaries only**
|
||||
R42 retrieves from original source material chunks as primary source, with
|
||||
topic summaries as secondary. This keeps R42 grounded and reduces hallucination.
|
||||
|
||||
---
|
||||
|
||||
### Micro learnings
|
||||
|
||||
**Decision: 10 types, all generated by AI as structured JSON**
|
||||
Types:
|
||||
1. concept_explainer
|
||||
2. scenario_quiz
|
||||
3. misconceptions
|
||||
4. how_to
|
||||
5. comparison_card
|
||||
6. reflection_prompt
|
||||
7. flashcard_set
|
||||
8. case_study
|
||||
9. glossary_anchor
|
||||
10. myth_vs_evidence
|
||||
|
||||
Each type has a defined JSON schema in data-model.md. Generation uses
|
||||
Claude Sonnet 4. Output is validated against Zod schemas before storage.
|
||||
|
||||
**Decision: employees choose type per topic per session**
|
||||
Employees are not locked to one type globally. Each session, per Topic, the
|
||||
employee selects from all published types for that topic. Multiple types can
|
||||
be completed in one session.
|
||||
|
||||
**Decision: pre-generate, don't generate on demand**
|
||||
All 10 types are generated when a Topic is approved, not when an employee
|
||||
requests them. This controls quality and cost. R42 is the only on-demand
|
||||
AI interaction.
|
||||
|
||||
---
|
||||
### Learning content
|
||||
- **Long-form content is generated on demand**, three types: `article`, `slides`,
|
||||
`infographic` (the `content` collection). New types shallow-merge into the cached
|
||||
object. **No podcast type.**
|
||||
- **Micro-learnings**, three types: `concept_explainer`, `scenario_quiz`,
|
||||
`flashcard_set` (the `micro_learnings` collection). A former `reflection_prompt`
|
||||
type was dropped.
|
||||
- **Employee chooses the format** per topic per session. Completion is not
|
||||
quality-gated; engaging with the full micro-learning counts.
|
||||
|
||||
### Curriculum
|
||||
|
||||
**Decision: AI generates curriculum, admin edits**
|
||||
Claude Sonnet 4 reads the full KB graph (Themes, Topics, relationships,
|
||||
complexity weights) and produces a 26-week schedule. Admin reviews, reorders,
|
||||
and finetunes. Admin does not build from scratch.
|
||||
|
||||
**Decision: one Theme per week session**
|
||||
A session covers all Topics under one Theme. Topics within the session are
|
||||
ordered by the curriculum generator based on prerequisites and complexity.
|
||||
|
||||
**Decision: perpetual curriculum with versioning**
|
||||
The curriculum runs indefinitely. After week 26, cycle 2 begins on the latest
|
||||
curriculum version. Cycle 2+ varies sequence, surfaces unused micro learning
|
||||
types, and increases coverage of low-engagement topics.
|
||||
|
||||
**Decision: completed weeks are immutable**
|
||||
Regeneration only affects future unstarted weeks. An employee's completion
|
||||
history is never altered regardless of curriculum version changes.
|
||||
|
||||
**Decision: regeneration requires admin confirmation**
|
||||
When new Topics are approved, the system queues a regeneration but does not
|
||||
apply it until the admin explicitly confirms. Admin sees a preview of the
|
||||
proposed new schedule before confirming.
|
||||
|
||||
**Decision: rolling starts**
|
||||
Each employee has their own start date. There are no cohorts or shared
|
||||
start dates.
|
||||
|
||||
---
|
||||
|
||||
### Gamification
|
||||
|
||||
**Decision: developer-native visual language**
|
||||
Inspired by GitHub (heatmap), Stack Overflow (badges, reputation), and
|
||||
Duolingo (streak, XP, levels). Language uses developer vocabulary throughout.
|
||||
|
||||
**Decision: XP unit is called commits**
|
||||
Every completed Topic earns commits. Quantity varies by micro learning type.
|
||||
|
||||
**Decision: levels use developer rank names**
|
||||
Intern → Junior → Medior → Senior → Staff → Principal
|
||||
Based on cumulative commits across all cycles.
|
||||
|
||||
**Decision: streak is weekly, not daily**
|
||||
Consecutive weeks with at least one completion. Resets on a skipped week.
|
||||
|
||||
**Decision: activity heatmap covers 26-week cycle**
|
||||
GitHub-style contribution graph. Cell darkness = number of types completed
|
||||
that week.
|
||||
|
||||
**Decision: no social layer**
|
||||
No comments, reactions, or direct messaging. Gamification is visible but
|
||||
not interactive between employees.
|
||||
|
||||
**Decision: public milestone cards, not ranked leaderboard**
|
||||
At weeks 13 and 26, a public card is posted to the shared activity feed.
|
||||
Language: "shipped", not "graduated". The leaderboard shows multiple
|
||||
dimensions (commits, streak, types used, badges) — not a single ranking.
|
||||
|
||||
**Decision: named content badges**
|
||||
Examples: governance_nerd, process_architect, deep_reader. These are seeded
|
||||
at startup, not user-generated. See data-model.md for badge schema.
|
||||
|
||||
---
|
||||
- **AI generates, admin confirms.** Claude proposes a 26-week schedule from the
|
||||
themed/weighted topic set; the admin previews and activates it. Versions move
|
||||
`draft → active → superseded`; exactly one is active.
|
||||
- **Per-user, self-paced start (current behavior).** Each employee enrolls on first
|
||||
login; their week/cycle is derived from `curriculum_started_at`. There is **no
|
||||
shared calendar week**. Week 1 is the first 7 days after they enroll.
|
||||
- **Perpetual, repeating cycles.** After week 26, the cycle restarts at week 1 with
|
||||
the same content. Completion history (`micro_learning_completions`) is append-only.
|
||||
- **Hash fallback.** If no curriculum version is active, topic assignment falls back
|
||||
to a deterministic hash of user id + week. Keep this fallback.
|
||||
|
||||
### R42 chatbot
|
||||
- **KB-grounded via TF-IDF**, not vector search. Context = top-K topics + verbatim
|
||||
mentions + filtered relations + limited deep content.
|
||||
- **Conversation persists per user** in `localStorage` (cap 50 messages; ~12 turns
|
||||
sent to the API). It is not stored server-side.
|
||||
- **Can propose graph edits** (`propose_graph_delta`, ≤3 topics / ≤5 relations).
|
||||
Admins apply immediately; non-admins queue a suggestion for admin approval.
|
||||
- **Hidden during quizzes** to protect test integrity.
|
||||
|
||||
**Decision: functional only, no personality**
|
||||
R42 answers questions grounded in the KB. It does not have a defined persona,
|
||||
tone, or name story beyond the label R42.
|
||||
### Gamification
|
||||
- **Points:** +2 per correct quiz answer, in the `leaderboard` collection.
|
||||
- **Badges** computed at render time: First Steps (1 test), Veteran (5 tests),
|
||||
Perfectionist (a 100% score).
|
||||
- Admins are excluded from the public leaderboard.
|
||||
|
||||
**Decision: stateless per session**
|
||||
No chat history is persisted between sessions. This avoids privacy complexity
|
||||
and keeps the implementation simple.
|
||||
|
||||
**Decision: internal KB scope only**
|
||||
R42 cannot search external sources. If a question cannot be answered from the
|
||||
KB, R42 says so explicitly.
|
||||
|
||||
**Decision: context-weighted retrieval**
|
||||
R42 knows the employee's current curriculum week. Retrieval boosts chunks
|
||||
from the current week's Theme. General KB questions are not restricted.
|
||||
|
||||
**Decision: always cites source Topic**
|
||||
Every R42 response includes the Topic title(s) its answer draws from.
|
||||
|
||||
**Decision: Haiku 4.5 for R42, Sonnet 4 for generation**
|
||||
Low latency matters for chat. The retrieval layer compensates for Haiku's
|
||||
smaller knowledge base. Sonnet 4 is reserved for generation tasks where
|
||||
structure and quality matter more than speed.
|
||||
### Auth & infrastructure
|
||||
- **PIN auth** against `team_members`; the session id lives in `sessionStorage`.
|
||||
Role `admin` unlocks the Admin panel.
|
||||
- **Claude model tiers:** `fast` = Haiku 4.5, `standard` = Sonnet 4.6,
|
||||
`reasoning` = Opus 4.7. Admins can override per tier from Settings.
|
||||
- **Simulation mode** (`admin:use_simulation`) returns stub LLM output for UI work.
|
||||
- **Deploy:** Docker image (Caddy serving the built SPA) + PocketBase container;
|
||||
Ansible playbooks under `infra/` for dev and prod.
|
||||
|
||||
---
|
||||
|
||||
### Infrastructure
|
||||
## Notable divergences from the original vision
|
||||
|
||||
**Decision: PocketBase as primary backend**
|
||||
Auth, file storage, structured data, and admin UI in one binary. SQLite is
|
||||
sufficient for ~150 users. No PostgreSQL needed at this scale.
|
||||
| Original design (not built) | What shipped |
|
||||
|---|---|
|
||||
| Next.js 14 PWA + 6 Fastify services | Single React/Vite SPA, no backend services |
|
||||
| Qdrant + OpenAI embeddings | Local TF-IDF, no embeddings |
|
||||
| Theme/Topic entity hierarchy, batch approval | Flat `topics` + `relations` graph |
|
||||
| 10 micro-learning types | 3 micro-learning types |
|
||||
| `employee_curriculum_state`, `badges`, `milestone_cards`, etc. | `team_members` fields + `leaderboard` + render-time badges |
|
||||
| Shared calendar-week curriculum | Per-user start, self-paced |
|
||||
|
||||
**Decision: Qdrant for vector storage**
|
||||
Separate Docker container. Keeps vector operations out of SQLite.
|
||||
pgvector was rejected — adding Postgres just for vectors is unnecessary overhead.
|
||||
|
||||
**Decision: Next.js 14 PWA for frontend**
|
||||
Single codebase for admin and employee app. PWA covers mobile without a native
|
||||
app. Learning platforms do not require native device APIs.
|
||||
|
||||
**Decision: five discrete backend services**
|
||||
Ingestion, generation, curriculum, chat, progress. Each is a separate Fastify
|
||||
service with its own port and responsibility. They do not call each other
|
||||
directly — they read/write shared PocketBase collections.
|
||||
|
||||
**Decision: PDF parsing starts with pdf-parse (Node.js)**
|
||||
Switch to pdfplumber Python sidecar only if pdf-parse quality is insufficient
|
||||
for actual source documents. Do not over-engineer the extraction layer upfront.
|
||||
|
||||
---
|
||||
|
||||
## What is not yet specced
|
||||
|
||||
The following spec files still need to be written before their phases begin:
|
||||
- /docs/generation-spec.md — micro learning generation service
|
||||
- /docs/curriculum-spec.md — curriculum generator + versioning
|
||||
- /docs/r42-spec.md — chat service
|
||||
- /docs/gamification-spec.md — progress service + gamification mechanics
|
||||
- /docs/frontend-spec.md — employee app, admin app, PWA config
|
||||
|
||||
Do not begin a phase without its spec file. Flag the gap if you reach it.
|
||||
|
||||
---
|
||||
|
||||
## Source of truth hierarchy
|
||||
|
||||
When sources conflict, resolve in this order:
|
||||
1. This handover document (rationale and decisions)
|
||||
2. The relevant spec file (implementation detail)
|
||||
3. data-model.md (schema is authoritative)
|
||||
4. architecture.md (system structure)
|
||||
|
||||
Do not use legacy/ code as a source of truth for anything.
|
||||
The abandoned scaffolding for the original design still exists under `/app` — it is
|
||||
not part of the running system.
|
||||
|
||||
@@ -1,394 +1,62 @@
|
||||
# Implementation plan
|
||||
# Implementation status & maintenance guide
|
||||
|
||||
## How to use this document
|
||||
|
||||
Work through phases in order. Do not start phase N+1 before phase N passes
|
||||
all acceptance criteria. Each phase lists the spec file to read, the steps
|
||||
to execute, and the criteria that define done.
|
||||
|
||||
At the start of each session: state the phase and step.
|
||||
At the end of each session: state completed steps and next starting point.
|
||||
The platform is **fully implemented and deployed**. This document replaces the
|
||||
original phased build plan (which targeted a Next.js + Qdrant architecture that was
|
||||
never shipped). It describes what exists today and where to work.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Infrastructure + ingestion service
|
||||
## Build status
|
||||
|
||||
**Spec to read:** /docs/ingestion-spec.md, /docs/data-model.md
|
||||
| Area | Status | Where it lives |
|
||||
|---|---|---|
|
||||
| Auth & onboarding | ✅ shipped | `src/store/AppContext.jsx`, `src/pages/Login.jsx`, `src/pages/Onboarding.jsx` |
|
||||
| Knowledge ingestion | ✅ shipped | `src/lib/extractionPipeline.js`, `src/components/admin/UploadZone.jsx` |
|
||||
| Knowledge graph (view/edit) | ✅ shipped | `src/components/admin/KnowledgeGraph.jsx` (D3) |
|
||||
| Learning content generation | ✅ shipped | `src/lib/learningService.js` (article/slides/infographic) |
|
||||
| Micro-learnings | ✅ shipped | `src/lib/microLearningService.js`, `src/components/micro_learning/` |
|
||||
| Weekly test | ✅ shipped | `src/lib/testService.js`, `src/pages/Testen.jsx` |
|
||||
| Curriculum (26-week, per-user) | ✅ shipped | `src/lib/curriculumService.js`, `src/components/admin/CurriculumManager.jsx` |
|
||||
| R42 chatbot | ✅ shipped | `src/components/chat/`, `src/lib/kbStore.js`, `src/lib/retrieval.js` |
|
||||
| Gamification | ✅ shipped | `src/pages/Leaderboard.jsx`, `leaderboard` collection |
|
||||
| Admin panel | ✅ shipped | `src/pages/Admin/index.jsx` (+ `src/components/admin/`) |
|
||||
| AI wrapper (tiers/retry/telemetry) | ✅ shipped | `src/lib/llm.js`, `src/lib/llmRetry.js`, `src/lib/llmSchemas.js`, `src/lib/llmTools.js` |
|
||||
| Deployment (Docker/Caddy/Ansible) | ✅ shipped | `Dockerfile`, `Caddyfile`, `docker-compose.yml`, `infra/` |
|
||||
|
||||
### Steps
|
||||
---
|
||||
|
||||
**1.1 — Repo scaffold**
|
||||
```
|
||||
app/
|
||||
frontend/ (empty, Next.js init comes in phase 4)
|
||||
services/
|
||||
ingestion/
|
||||
generation/ (empty placeholder)
|
||||
curriculum/ (empty placeholder)
|
||||
chat/ (empty placeholder)
|
||||
progress/ (empty placeholder)
|
||||
## Where to make changes
|
||||
|
||||
- **New PocketBase field/collection:** add a migration in `pb_migrations/`
|
||||
(follow the existing JS migration style) and mirror it in
|
||||
`scripts/setup-pb-collections.mjs`. Then add async helpers in `src/lib/db.js`.
|
||||
- **New AI capability:** add a tool in `src/lib/llmTools.js`, a Zod schema in
|
||||
`src/lib/llmSchemas.js` (register it in `toolSchemaRegistry`), and call it through
|
||||
`callLLM`. Never hit `/api/anthropic` directly.
|
||||
- **New screen:** add a page under `src/pages/`, route it in `src/App.jsx`, and gate
|
||||
it with `ProtectedRoute` (which also enforces curriculum enrollment).
|
||||
- **New admin tool:** add a tab in `src/pages/Admin/index.jsx` and a panel under
|
||||
`src/components/admin/`.
|
||||
|
||||
---
|
||||
|
||||
## Verification before shipping
|
||||
|
||||
```bash
|
||||
npm test # Vitest unit tests (lib services)
|
||||
npm run lint # ESLint
|
||||
npm run build # production build to dist/
|
||||
```
|
||||
|
||||
Create `app/services/ingestion/` with:
|
||||
- package.json (dependencies from ingestion-spec.md)
|
||||
- tsconfig.json (strict mode)
|
||||
- .env.example (all env vars from ingestion-spec.md)
|
||||
- .gitignore
|
||||
|
||||
**1.2 — PocketBase collections**
|
||||
PocketBase runs as a binary. Create a migration script at
|
||||
`app/services/ingestion/migrations/001_initial_schema.ts` that uses the
|
||||
PocketBase JS SDK to create all collections defined in data-model.md:
|
||||
|
||||
Collections to create:
|
||||
- source_documents
|
||||
- themes
|
||||
- topics
|
||||
- micro_learnings (schema only — no data yet)
|
||||
- curriculum_versions (schema only)
|
||||
- curriculum_weeks (schema only)
|
||||
- employee_curriculum_state (schema only)
|
||||
- session_completions (schema only)
|
||||
- gamification_profiles (schema only)
|
||||
- badges (schema only)
|
||||
- employee_badges (schema only)
|
||||
- milestone_cards (schema only)
|
||||
|
||||
Seed the badges collection with all badge definitions from data-model.md.
|
||||
|
||||
**1.3 — Qdrant collections**
|
||||
Create `app/services/ingestion/migrations/002_qdrant_setup.ts` that
|
||||
initialises both Qdrant collections:
|
||||
- source_chunks (1536 dimensions, cosine distance)
|
||||
- topic_summaries (1536 dimensions, cosine distance)
|
||||
|
||||
**1.4 — Ingestion service scaffold**
|
||||
Build the Fastify server with two routes:
|
||||
- POST /ingest
|
||||
- GET /status/:jobId
|
||||
|
||||
Use the file structure from ingestion-spec.md exactly.
|
||||
|
||||
**1.5 — Stage 1: text extraction**
|
||||
Implement extract.ts per ingestion-spec.md:
|
||||
- TXT: direct UTF-8 read
|
||||
- MD: direct UTF-8 read, preserve heading markers
|
||||
- PDF: pdf-parse, page break markers
|
||||
|
||||
**1.6 — Stage 2–3: chunking + cleaning**
|
||||
Implement chunk.ts and clean.ts per ingestion-spec.md:
|
||||
- MD: heading-based splitting
|
||||
- TXT: sliding window (800 chars, 150 overlap)
|
||||
- PDF: page + paragraph splitting
|
||||
- Cleaning: whitespace, artefacts, minimum length filter
|
||||
|
||||
**1.7 — Stage 4: structure extraction**
|
||||
Implement structure.ts per ingestion-spec.md:
|
||||
- Claude Sonnet 4 call with system + user prompt from spec
|
||||
- Zod validation of DraftKB output
|
||||
- Batch handling for documents > 60 chunks
|
||||
- Retry logic on parse failure
|
||||
- Error handling: failed job status + reason
|
||||
|
||||
**1.8 — Stage 5: PocketBase write**
|
||||
Implement the PocketBase write logic:
|
||||
- Create Theme records (status: draft)
|
||||
- Create Topic records under each Theme (status: draft)
|
||||
- Resolve relationships between Topics after all records created
|
||||
|
||||
**1.9 — Stage 6: embeddings + Qdrant write**
|
||||
Implement embed.ts:
|
||||
- OpenAI text-embedding-3-small, batches of 100
|
||||
- Write to Qdrant source_chunks collection
|
||||
- Write to Qdrant topic_summaries collection
|
||||
- Update Topic.qdrant_chunk_ids in PocketBase
|
||||
|
||||
**1.10 — Job status tracking**
|
||||
Wire all stages into the job queue (jobs/queue.ts):
|
||||
- Status transitions: queued → extracting → chunking → structuring →
|
||||
writing → embedding → done / failed
|
||||
- Progress counters (chunksTotal, chunksEmbedded, themesFound, topicsFound)
|
||||
- GET /status/:jobId returns current state
|
||||
|
||||
### Acceptance criteria
|
||||
|
||||
- [ ] POST /ingest with a small MD file completes without error
|
||||
- [ ] GET /status/:jobId returns `done` after processing
|
||||
- [ ] PocketBase contains draft Theme + Topic records with correct hierarchy
|
||||
- [ ] Topic.body contains AI-drafted content (not empty)
|
||||
- [ ] Topic relationships are resolved (related_topics populated where applicable)
|
||||
- [ ] Qdrant source_chunks contains vectors with correct payload fields
|
||||
- [ ] Qdrant topic_summaries contains vectors for each published topic
|
||||
- [ ] Topic.qdrant_chunk_ids is populated
|
||||
- [ ] POST /ingest with a PDF file completes without error
|
||||
- [ ] POST /ingest with a TXT file completes without error
|
||||
- [ ] A document > 60 chunks triggers batch processing without error
|
||||
- [ ] A malformed PDF returns status `failed` with reason, not an uncaught exception
|
||||
- [ ] All Zod validations pass — no `any` types in codebase
|
||||
For UI/feature correctness, run the app against a local PocketBase
|
||||
(`npm run dev` + `./pocketbase.exe serve`) and exercise the flow in the browser.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Generation service
|
||||
## Constraints (see CLAUDE.md / PROTECTED.md)
|
||||
|
||||
**Spec to read:** /docs/generation-spec.md (write this spec before starting)
|
||||
|
||||
### Steps
|
||||
|
||||
**2.1 — Generation service scaffold**
|
||||
Fastify service at app/services/generation/
|
||||
Routes: POST /generate, GET /status/:jobId
|
||||
|
||||
**2.2 — Generate all 10 types per topic**
|
||||
One Claude Sonnet 4 call per type per topic.
|
||||
Structured JSON output validated against Zod schemas from data-model.md.
|
||||
Write to micro_learnings collection (status: generated).
|
||||
|
||||
**2.3 — Batch generation on theme approval**
|
||||
When admin approves a Theme batch, queue generation for all Topics in that Theme.
|
||||
All 10 types per Topic.
|
||||
|
||||
**2.4 — Admin publish flow**
|
||||
Route to update micro_learning status from generated → published or rejected.
|
||||
This is called by the admin app (built in phase 4).
|
||||
|
||||
### Acceptance criteria (to be detailed in generation-spec.md)
|
||||
|
||||
- [ ] All 10 micro learning types generated for a test topic
|
||||
- [ ] All 10 JSON outputs validate against their Zod schemas
|
||||
- [ ] Generated content written to PocketBase with status: generated
|
||||
- [ ] Admin can publish or reject individual micro learnings
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Curriculum service
|
||||
|
||||
**Spec to read:** /docs/curriculum-spec.md (write this spec before starting)
|
||||
|
||||
### Steps
|
||||
|
||||
**3.1 — Curriculum service scaffold**
|
||||
Fastify service at app/services/curriculum/
|
||||
|
||||
**3.2 — Curriculum generator**
|
||||
Claude Sonnet 4 reads full KB graph → produces 26-week schedule.
|
||||
Written to curriculum_versions + curriculum_weeks.
|
||||
|
||||
**3.3 — Versioning logic**
|
||||
- New version created on regeneration
|
||||
- Completed weeks frozen (employee_curriculum_state.current_week used as boundary)
|
||||
- Admin confirmation required before applying new version
|
||||
|
||||
**3.4 — Perpetual cycling**
|
||||
On week 26 completion, cycle increments, new cycle starts on latest version.
|
||||
Second cycle: varied sequence, surfaces unused micro learning types.
|
||||
|
||||
### Acceptance criteria (to be detailed in curriculum-spec.md)
|
||||
|
||||
- [ ] Curriculum generated from a populated KB
|
||||
- [ ] 26 weeks produced, all Themes covered
|
||||
- [ ] Prerequisites respected in ordering
|
||||
- [ ] Regeneration does not alter completed weeks
|
||||
- [ ] Admin confirmation flow works correctly
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — Frontend: admin app
|
||||
|
||||
**Spec to read:** /docs/frontend-spec.md (write this spec before starting)
|
||||
|
||||
### Steps
|
||||
|
||||
**4.1 — Next.js 14 scaffold**
|
||||
Mobile-first, TypeScript strict, Tailwind CSS, PWA config.
|
||||
Role-based routing: /admin/* and /app/* from single Next.js codebase.
|
||||
|
||||
**4.2 — Auth**
|
||||
PocketBase auth integration. Admin role routes to /admin/*.
|
||||
|
||||
**4.3 — Document upload + ingestion status**
|
||||
Upload UI → calls ingestion service → polls job status → shows progress.
|
||||
|
||||
**4.4 — Theme batch review**
|
||||
Display draft Themes with their Topic list.
|
||||
Approve batch / edit individual topics / reject batch.
|
||||
Triggers generation service on approval.
|
||||
|
||||
**4.5 — Curriculum editor**
|
||||
Display AI-generated curriculum (26 weeks).
|
||||
Drag-to-reorder weeks. Edit Theme assignment per week.
|
||||
Confirm regeneration with preview.
|
||||
|
||||
### Acceptance criteria (to be detailed in frontend-spec.md)
|
||||
|
||||
- [ ] Admin can upload a document and see ingestion progress
|
||||
- [ ] Admin can approve a Theme batch
|
||||
- [ ] Admin can edit a Topic before approval
|
||||
- [ ] Admin can view and reorder the curriculum
|
||||
- [ ] Admin can confirm a curriculum regeneration with preview
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — Frontend: employee app
|
||||
|
||||
**Spec to read:** /docs/frontend-spec.md (same file, employee section)
|
||||
|
||||
### Steps
|
||||
|
||||
**5.1 — Employee auth + onboarding**
|
||||
PocketBase auth. Employee role routes to /app/*.
|
||||
Set start date on first login → creates employee_curriculum_state record.
|
||||
|
||||
**5.2 — Weekly session flow**
|
||||
Current week's Theme displayed.
|
||||
Topics listed with available micro learning types per topic.
|
||||
Employee selects type → content rendered → mark complete.
|
||||
|
||||
**5.3 — Knowledge library**
|
||||
Browse all published Topics.
|
||||
Filter by Theme, difficulty, key terms.
|
||||
|
||||
**5.4 — R42 chatbot**
|
||||
Floating button, every screen.
|
||||
Calls chat service → streams response.
|
||||
Cites source topic in response.
|
||||
|
||||
**5.5 — Gamification profile**
|
||||
GitHub-style heatmap (26-week view).
|
||||
Badge display.
|
||||
Streak + level + commit count.
|
||||
Public leaderboard (multi-dimension).
|
||||
Milestone cards in activity feed.
|
||||
|
||||
### Acceptance criteria (to be detailed in frontend-spec.md)
|
||||
|
||||
- [ ] Employee sees correct week based on start date
|
||||
- [ ] Employee can complete a topic with a chosen micro learning type
|
||||
- [ ] Completion is recorded and XP awarded
|
||||
- [ ] Knowledge library shows all published topics with filters
|
||||
- [ ] R42 responds with grounded answer and source citation
|
||||
- [ ] R42 is accessible from every screen
|
||||
- [ ] Heatmap renders correctly on mobile (375px)
|
||||
- [ ] Leaderboard shows all employees with multi-dimension data
|
||||
|
||||
---
|
||||
|
||||
## Phase 6 — Chat service (R42)
|
||||
|
||||
**Spec to read:** /docs/r42-spec.md (write this spec before starting)
|
||||
|
||||
### Steps
|
||||
|
||||
**6.1 — Chat service scaffold**
|
||||
Fastify service at app/services/chat/
|
||||
|
||||
**6.2 — Query → embed → retrieve**
|
||||
Employee query embedded → Qdrant nearest-neighbour on both collections.
|
||||
Boost chunks from employee's current Theme.
|
||||
|
||||
**6.3 — Response generation**
|
||||
Top-K chunks injected into Haiku 4.5 prompt.
|
||||
Response streamed to frontend.
|
||||
Source Topic titles included in response.
|
||||
|
||||
**6.4 — Out-of-scope handling**
|
||||
If retrieval confidence is below threshold, R42 responds:
|
||||
"I can only answer questions based on the internal knowledge base.
|
||||
This topic doesn't appear to be covered."
|
||||
|
||||
### Acceptance criteria (to be detailed in r42-spec.md)
|
||||
|
||||
- [ ] R42 answers a question about a published topic correctly
|
||||
- [ ] R42 cites the source topic in its response
|
||||
- [ ] R42 refuses to answer out-of-scope questions explicitly
|
||||
- [ ] Response streams to frontend (not batch)
|
||||
- [ ] Response latency < 3 seconds for typical queries
|
||||
|
||||
---
|
||||
|
||||
## Phase 7 — Progress service
|
||||
|
||||
**Spec to read:** /docs/gamification-spec.md (write this spec before starting)
|
||||
|
||||
### Steps
|
||||
|
||||
**7.1 — Progress service scaffold**
|
||||
Fastify service at app/services/progress/
|
||||
|
||||
**7.2 — Completion recording**
|
||||
Write session_completions record on topic completion.
|
||||
Calculate XP (commits) per type.
|
||||
|
||||
**7.3 — Gamification updates**
|
||||
Update gamification_profiles: commits, level, streak, types_used.
|
||||
Evaluate badge conditions → write employee_badges on award.
|
||||
|
||||
**7.4 — Milestone cards**
|
||||
Generate milestone_cards record at weeks 13 and 26.
|
||||
|
||||
**7.5 — Leaderboard query**
|
||||
Endpoint returning all gamification_profiles for leaderboard rendering.
|
||||
|
||||
### Acceptance criteria (to be detailed in gamification-spec.md)
|
||||
|
||||
- [ ] Completion writes to session_completions
|
||||
- [ ] Commits calculated and added to gamification_profile
|
||||
- [ ] Level updates correctly at commit thresholds
|
||||
- [ ] Streak increments on weekly completion, resets on skip
|
||||
- [ ] Badge awarded when condition is met
|
||||
- [ ] Milestone card created at weeks 13 and 26
|
||||
- [ ] Leaderboard endpoint returns all employees with correct data
|
||||
|
||||
---
|
||||
|
||||
## Phase 8 — Integration + hardening
|
||||
|
||||
No new spec required.
|
||||
|
||||
### Steps
|
||||
|
||||
**8.1 — Service wiring**
|
||||
Verify all services communicate through PocketBase correctly.
|
||||
No direct service-to-service calls — all state through PocketBase.
|
||||
|
||||
**8.2 — Error handling audit**
|
||||
Review all services for unhandled promise rejections, missing error states,
|
||||
and uncaught exceptions. Every external call (AI API, PocketBase, Qdrant,
|
||||
OpenAI) wrapped in try/catch with meaningful error logging.
|
||||
|
||||
**8.3 — Mobile QA**
|
||||
Test all employee app flows at 375px width.
|
||||
R42 floating button must not obscure content.
|
||||
Heatmap must render without horizontal scroll.
|
||||
|
||||
**8.4 — Environment variable audit**
|
||||
Verify no hardcoded values. All .env.example files complete.
|
||||
|
||||
**8.5 — Dockerfile update**
|
||||
Update COPY path from legacy app root to /app.
|
||||
This is the one manual change that connects the rebuild to the existing pipeline.
|
||||
|
||||
### Acceptance criteria
|
||||
|
||||
- [ ] Full flow works end-to-end: upload doc → approve → curriculum → employee completes session → R42 answers question → gamification updates
|
||||
- [ ] No uncaught exceptions in any service under normal operating conditions
|
||||
- [ ] All screens render correctly on 375px mobile
|
||||
- [ ] Dockerfile builds successfully pointing at /app
|
||||
- [ ] Existing pipeline deploys the rebuilt app without modification
|
||||
|
||||
---
|
||||
|
||||
## Spec files still to be written
|
||||
|
||||
Before starting each phase, write the corresponding spec file.
|
||||
Use ingestion-spec.md as the template for structure and detail level.
|
||||
|
||||
| Phase | Spec file needed |
|
||||
|---|---|
|
||||
| 2 | /docs/generation-spec.md |
|
||||
| 3 | /docs/curriculum-spec.md |
|
||||
| 4–5 | /docs/frontend-spec.md |
|
||||
| 6 | /docs/r42-spec.md |
|
||||
| 7 | /docs/gamification-spec.md |
|
||||
|
||||
When you reach a phase without a spec: stop, draft the spec, then proceed.
|
||||
Do not implement without a spec.
|
||||
- Do not edit `stylesheet.css` or the deployment files (`Dockerfile`, `Caddyfile`,
|
||||
`docker-compose.yml`, `infra/`, `.github/workflows/`) without a request.
|
||||
- Do not re-enable PocketBase auto-cancellation.
|
||||
- Do not re-add the podcast content type or the `reflection_prompt` micro-learning.
|
||||
- Ignore the abandoned `/app` Next.js scaffolding.
|
||||
|
||||
@@ -1,483 +1,72 @@
|
||||
# Ingestion service spec
|
||||
# Ingestion spec: source documents → knowledge graph
|
||||
|
||||
## Responsibility
|
||||
Turns admin-uploaded text into `topics` and `relations` using Claude. Runs
|
||||
entirely client-side; there is no ingestion service.
|
||||
|
||||
Accepts uploaded source documents (PDF, MD, TXT), extracts clean text, chunks it,
|
||||
generates embeddings, and produces a structured draft KB (Themes + Topics +
|
||||
relationships) ready for admin review.
|
||||
|
||||
This service runs entirely server-side. The admin app calls it via REST. All AI
|
||||
calls go through the Anthropic API. No ingestion logic lives in the frontend.
|
||||
- **UI:** `src/components/admin/UploadZone.jsx` (Admin → Sources tab)
|
||||
- **Pipeline:** `src/lib/extractionPipeline.js`
|
||||
- **Tool/schema:** `emit_knowledge_graph` (`src/lib/llmTools.js`) validated by
|
||||
`extractionResultSchema` (`src/lib/llmSchemas.js`)
|
||||
|
||||
---
|
||||
|
||||
## Service location
|
||||
## Upload
|
||||
|
||||
```
|
||||
app/services/ingestion/
|
||||
├── index.ts entry point, Fastify server
|
||||
├── routes/
|
||||
│ └── documents.ts POST /ingest, GET /status/:jobId
|
||||
├── pipeline/
|
||||
│ ├── extract.ts format detection + text extraction
|
||||
│ ├── chunk.ts chunking strategies per format
|
||||
│ ├── clean.ts chunk cleaning
|
||||
│ ├── structure.ts Claude call → Theme/Topic extraction
|
||||
│ └── embed.ts embedding generation + Qdrant write
|
||||
├── jobs/
|
||||
│ └── queue.ts async job queue (in-memory, BullMQ later if needed)
|
||||
├── lib/
|
||||
│ ├── pocketbase.ts PocketBase client
|
||||
│ ├── qdrant.ts Qdrant client
|
||||
│ ├── anthropic.ts Anthropic client
|
||||
│ └── openai.ts OpenAI embeddings client
|
||||
└── types.ts shared TypeScript types
|
||||
```
|
||||
- Accepted formats: **`.txt` and `.md`**, max **5 MB** per file.
|
||||
- Drag-and-drop or click-to-browse. Unsupported files are skipped with a toast.
|
||||
- The queue tracks each file: `pending → processing → done / failed / cancelled`.
|
||||
- Progress is polled every ~2s from the `sources.progress` field
|
||||
(`{ current, total, message }`), shown as "Chunk N/total".
|
||||
- **Orphan detection:** sources stuck in `processing` for >5 minutes (e.g. a closed
|
||||
tab) can be marked failed or deleted.
|
||||
|
||||
---
|
||||
|
||||
## API surface
|
||||
## Pipeline: `processSourceText(textContent, sourceName, { signal })`
|
||||
|
||||
### POST /ingest
|
||||
1. **Create source record** with `status='processing'`.
|
||||
2. **Chunk** the text (`chunkText`): target ~8000 chars per chunk with ~800 chars
|
||||
overlap, splitting on sentence/paragraph boundaries (hard-splitting oversized
|
||||
sentences). Overlap preserves cross-boundary context.
|
||||
3. **Known-topics hint** (`buildKnownIdsHint`): up to the 200 most recent existing
|
||||
topics are listed so the model reuses existing ids instead of duplicating them.
|
||||
4. **Per-chunk extraction** via `callLLM`:
|
||||
- tier `standard`, `maxTokens: 8192`, `timeoutMs: 180_000`
|
||||
- forced `toolChoice` on `emit_knowledge_graph`
|
||||
- rate-limited (~20 req/min, burst 2) to protect quota across many chunks
|
||||
- system prompt instructs: ≤15 topics per chunk; topic `type` ∈
|
||||
{`concept`, `role`, `process`}; `learning_relevance` ∈
|
||||
{`core`, `standard`, `peripheral`, `exclude`}; relation `type` ∈
|
||||
{`related_to`, `depends_on`, `part_of`, `executed_by`}
|
||||
5. **Update progress** before each chunk.
|
||||
6. **Merge** (`mergeKnowledgeGraph`): topics keyed by `id` (new data updates
|
||||
existing, but `learning_relevance` is preserved when `relevance_locked` is true);
|
||||
relations de-duplicated on `(source, target, type)`. Persisted via
|
||||
`db.saveTopics` / `db.saveRelations`.
|
||||
7. **Finalize:** `status='completed'`, or `failed` (error) / `cancelled` (abort).
|
||||
|
||||
Triggered by admin app on document upload.
|
||||
|
||||
Request:
|
||||
```json
|
||||
{
|
||||
"documentId": "string",
|
||||
"filename": "string",
|
||||
"format": "pdf" | "md" | "txt",
|
||||
"filePath": "string"
|
||||
}
|
||||
```
|
||||
|
||||
Response (202 Accepted):
|
||||
```json
|
||||
{
|
||||
"jobId": "string",
|
||||
"status": "queued"
|
||||
}
|
||||
```
|
||||
|
||||
Processing is async. The admin app polls job status.
|
||||
Aborting via the `signal` stops the run and marks the source `cancelled`.
|
||||
|
||||
---
|
||||
|
||||
### GET /status/:jobId
|
||||
|
||||
Returns current job progress.
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"jobId": "string",
|
||||
"status": "queued" | "extracting" | "chunking" | "structuring" | "embedding" | "done" | "failed",
|
||||
"progress": {
|
||||
"chunksTotal": 42,
|
||||
"chunksEmbedded": 18,
|
||||
"themesFound": 3,
|
||||
"topicsFound": 14
|
||||
},
|
||||
"error": "string | null"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Pipeline stages
|
||||
|
||||
### Stage 1 — Text extraction
|
||||
|
||||
Input: file path + format
|
||||
Output: raw text string
|
||||
|
||||
```
|
||||
format === 'txt'
|
||||
→ read file directly as UTF-8
|
||||
|
||||
format === 'md'
|
||||
→ read file directly as UTF-8
|
||||
→ preserve heading markers (# ## ###) — used in chunking
|
||||
|
||||
format === 'pdf'
|
||||
→ pdfplumber: extract text page by page
|
||||
→ concatenate with page break markers: \n\n---PAGE---\n\n
|
||||
→ strip known PDF artefacts: headers/footers repeating on every page,
|
||||
page numbers, watermarks
|
||||
```
|
||||
|
||||
Failure handling:
|
||||
- PDF extraction returns empty string → mark job `failed`, reason: `pdf_extraction_empty`
|
||||
- File not found → mark job `failed`, reason: `file_not_found`
|
||||
|
||||
---
|
||||
|
||||
### Stage 2 — Chunking
|
||||
|
||||
Input: raw text + format
|
||||
Output: Chunk[]
|
||||
|
||||
Chunking strategy differs per format.
|
||||
|
||||
**MD chunking — heading-based (preferred)**
|
||||
```
|
||||
Split on heading markers: #, ##, ###
|
||||
Each heading + its following content = one chunk
|
||||
Minimum chunk size: 100 characters
|
||||
→ if heading section is < 100 chars, merge with next sibling
|
||||
Maximum chunk size: 1500 characters
|
||||
→ if section exceeds limit, split on paragraph breaks within section
|
||||
Metadata preserved per chunk:
|
||||
heading_level: 1 | 2 | 3
|
||||
heading_text: string
|
||||
parent_heading: string | null
|
||||
```
|
||||
|
||||
MD chunking produces the highest quality structural signal for Theme/Topic extraction.
|
||||
Admins should be advised to provide source material as MD where possible.
|
||||
|
||||
**TXT chunking — sliding window**
|
||||
```
|
||||
Window size: 800 characters
|
||||
Overlap: 150 characters
|
||||
Split on: paragraph breaks (\n\n) first, then sentence boundaries, then hard cut
|
||||
Metadata per chunk:
|
||||
chunk_index: number
|
||||
approximate_position: 'start' | 'middle' | 'end'
|
||||
```
|
||||
|
||||
**PDF chunking — page + paragraph**
|
||||
```
|
||||
Split on ---PAGE--- markers from extraction stage
|
||||
Within each page: split on paragraph breaks (\n\n)
|
||||
Minimum chunk size: 100 characters
|
||||
→ merge sub-threshold paragraphs with adjacent chunk
|
||||
Maximum chunk size: 1200 characters
|
||||
→ hard split at sentence boundary
|
||||
Metadata per chunk:
|
||||
page_number: number
|
||||
chunk_index_on_page: number
|
||||
```
|
||||
|
||||
**Chunk type:**
|
||||
```typescript
|
||||
type Chunk = {
|
||||
id: string // UUID generated at chunking
|
||||
documentId: string
|
||||
text: string
|
||||
format: 'pdf' | 'md' | 'txt'
|
||||
index: number // global position in document
|
||||
metadata: {
|
||||
// MD-specific
|
||||
headingLevel?: number
|
||||
headingText?: string
|
||||
parentHeading?: string
|
||||
// TXT-specific
|
||||
approximatePosition?: 'start' | 'middle' | 'end'
|
||||
// PDF-specific
|
||||
pageNumber?: number
|
||||
chunkIndexOnPage?: number
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Stage 3 — Chunk cleaning
|
||||
|
||||
Input: Chunk[]
|
||||
Output: Chunk[] (cleaned)
|
||||
|
||||
Applied to all formats:
|
||||
```
|
||||
- trim leading/trailing whitespace
|
||||
- collapse 3+ consecutive newlines to 2
|
||||
- remove null bytes and non-printable characters
|
||||
- remove chunks where text.length < 80 after cleaning
|
||||
→ these are likely artefacts (page numbers, standalone headers)
|
||||
- normalise unicode: NFC normalisation
|
||||
- do not strip punctuation or alter sentence structure
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Stage 4 — Structure extraction (AI)
|
||||
|
||||
Input: Chunk[]
|
||||
Output: DraftKB
|
||||
|
||||
This is the core AI call. Claude Sonnet 4 reads all chunks and returns a structured
|
||||
KB draft as JSON.
|
||||
|
||||
**Prompt strategy:**
|
||||
|
||||
System prompt:
|
||||
```
|
||||
You are a knowledge architect. Your task is to analyse a set of text chunks from
|
||||
a source document and extract a structured knowledge base.
|
||||
|
||||
Output ONLY valid JSON matching the schema provided. No preamble, no explanation,
|
||||
no markdown fences.
|
||||
|
||||
Rules:
|
||||
- Group related content into Themes. A Theme is a broad subject area.
|
||||
- Under each Theme, identify discrete Topics. A Topic covers one specific concept.
|
||||
- Identify relationships between Topics: related, prerequisite, or contrast.
|
||||
- related: Topics that complement each other
|
||||
- prerequisite: Topic A must be understood before Topic B
|
||||
- contrast: Topics that represent opposing approaches or concepts
|
||||
- For each Topic, extract key terms suitable for a glossary.
|
||||
- Assign a complexity weight (1–5) to each Topic.
|
||||
1 = introductory, 5 = advanced
|
||||
- Draft a body for each Topic (2–4 paragraphs) based on the source chunks.
|
||||
- Draft a description for each Theme (1–2 sentences).
|
||||
- Every Topic must reference the chunk IDs that contributed to it.
|
||||
```
|
||||
|
||||
User prompt:
|
||||
```
|
||||
Source document: {filename}
|
||||
Format: {format}
|
||||
|
||||
Chunks:
|
||||
{chunks mapped as: [CHUNK-{id}]\n{text}\n}
|
||||
|
||||
Extract the knowledge base structure from these chunks.
|
||||
```
|
||||
|
||||
**Output schema:**
|
||||
```typescript
|
||||
type DraftKB = {
|
||||
themes: DraftTheme[]
|
||||
}
|
||||
|
||||
type DraftTheme = {
|
||||
title: string
|
||||
description: string
|
||||
topics: DraftTopic[]
|
||||
}
|
||||
|
||||
type DraftTopic = {
|
||||
title: string
|
||||
body: string
|
||||
difficulty: 'introductory' | 'intermediate' | 'advanced'
|
||||
complexityWeight: number // 1–5
|
||||
keyTerms: string[]
|
||||
sourceChunkIds: string[] // references Chunk.id values
|
||||
relationships: {
|
||||
related: string[] // topic titles (resolved to IDs after write)
|
||||
prerequisites: string[]
|
||||
contrasts: string[]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**AI call configuration:**
|
||||
```typescript
|
||||
{
|
||||
model: 'claude-sonnet-4-20250514',
|
||||
max_tokens: 8000,
|
||||
temperature: 0 // deterministic output for structured extraction
|
||||
}
|
||||
```
|
||||
|
||||
**Chunking strategy for large documents:**
|
||||
If total chunk count exceeds 60 chunks, split into batches of 40 with 5-chunk
|
||||
overlap. Run one Claude call per batch. Merge resulting DraftKB objects:
|
||||
- Themes with identical titles → merge Topics
|
||||
- Duplicate Topic titles within a Theme → keep longer body, merge sourceChunkIds
|
||||
- Relationships are resolved after full merge
|
||||
|
||||
**Error handling:**
|
||||
- JSON parse failure → retry once with stricter prompt ("ensure valid JSON only")
|
||||
- Second failure → mark job `failed`, reason: `structure_extraction_failed`, log raw response
|
||||
- Empty themes array → mark job `failed`, reason: `no_structure_found`
|
||||
|
||||
---
|
||||
|
||||
### Stage 5 — Write to PocketBase
|
||||
|
||||
Input: DraftKB
|
||||
Output: written Theme + Topic records with status `draft`
|
||||
|
||||
```
|
||||
For each DraftTheme:
|
||||
create themes record {
|
||||
title, description,
|
||||
status: 'draft',
|
||||
source_documents: [documentId]
|
||||
}
|
||||
|
||||
For each DraftTopic under the theme:
|
||||
create topics record {
|
||||
theme: themeId,
|
||||
title, body, difficulty, complexity_weight, key_terms,
|
||||
status: 'draft',
|
||||
qdrant_chunk_ids: [] // populated in stage 6
|
||||
}
|
||||
|
||||
After all topics created:
|
||||
resolve relationship titles → topic IDs
|
||||
update topics.related_topics, prerequisite_topics, contrast_topics
|
||||
|
||||
If a relationship title cannot be resolved to an existing topic:
|
||||
skip silently (cross-document relationships resolved in a later pass)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Stage 6 — Embedding generation + Qdrant write
|
||||
|
||||
Input: Chunk[], written Topic records
|
||||
Output: vectors in Qdrant, qdrant_chunk_ids updated on Topic records
|
||||
|
||||
**Source chunk embeddings:**
|
||||
```
|
||||
For each Chunk (post-cleaning):
|
||||
embed Chunk.text → text-embedding-3-small (1536 dimensions)
|
||||
write to Qdrant collection: source_chunks {
|
||||
id: Chunk.id,
|
||||
vector: float[],
|
||||
payload: {
|
||||
source_document_id: documentId,
|
||||
chunk_index: Chunk.index,
|
||||
text: Chunk.text,
|
||||
theme_id: resolved themeId | null,
|
||||
topic_id: resolved topicId | null,
|
||||
format: Chunk.format
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Topic summary embeddings:**
|
||||
```
|
||||
For each published Topic:
|
||||
embed Topic.body → text-embedding-3-small
|
||||
write to Qdrant collection: topic_summaries {
|
||||
id: UUID,
|
||||
vector: float[],
|
||||
payload: {
|
||||
topic_id: Topic.id,
|
||||
theme_id: Topic.theme,
|
||||
title: Topic.title,
|
||||
text: Topic.body
|
||||
}
|
||||
}
|
||||
|
||||
Update Topic.qdrant_chunk_ids with all Chunk.ids that reference this topic
|
||||
```
|
||||
|
||||
**Batching:**
|
||||
OpenAI embeddings API: batch in groups of 100 texts per request to stay within
|
||||
rate limits and reduce latency.
|
||||
|
||||
---
|
||||
|
||||
## Job lifecycle
|
||||
|
||||
```
|
||||
POST /ingest received
|
||||
↓
|
||||
Job created → status: queued
|
||||
↓
|
||||
Stage 1: extracting
|
||||
↓
|
||||
Stage 2–3: chunking
|
||||
↓
|
||||
Stage 4: structuring
|
||||
↓
|
||||
Stage 5: writing to PocketBase
|
||||
↓
|
||||
Stage 6: embedding
|
||||
↓
|
||||
status: done
|
||||
↓
|
||||
Admin notification: "Document processed. N themes, N topics ready for review."
|
||||
↓
|
||||
Curriculum regeneration queued (status: pending_admin_confirm)
|
||||
```
|
||||
|
||||
On any stage failure:
|
||||
```
|
||||
status: failed
|
||||
error: { stage, reason, detail }
|
||||
Source document status → 'failed' in PocketBase
|
||||
Admin notification: "Ingestion failed: {reason}"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Environment variables required
|
||||
|
||||
```
|
||||
ANTHROPIC_API_KEY=
|
||||
OPENAI_API_KEY=
|
||||
POCKETBASE_URL=
|
||||
POCKETBASE_ADMIN_EMAIL=
|
||||
POCKETBASE_ADMIN_PASSWORD=
|
||||
QDRANT_URL=
|
||||
QDRANT_API_KEY= # empty string if running locally without auth
|
||||
INGESTION_PORT=3001
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
## Output shape (`emit_knowledge_graph`)
|
||||
|
||||
```json
|
||||
{
|
||||
"dependencies": {
|
||||
"fastify": "^4",
|
||||
"@anthropic-ai/sdk": "^0.24",
|
||||
"openai": "^4",
|
||||
"@qdrant/js-client-rest": "^1.9",
|
||||
"pocketbase": "^0.21",
|
||||
"pdfplumber": "NOT JS — see note below",
|
||||
"pdf-parse": "^1.1",
|
||||
"uuid": "^9",
|
||||
"zod": "^3"
|
||||
}
|
||||
"topics": [ { "id", "label", "type", "description", "learning_relevance" } ],
|
||||
"relations":[ { "source", "target", "type" } ]
|
||||
}
|
||||
```
|
||||
|
||||
**PDF extraction note:**
|
||||
`pdfplumber` is a Python library. Two options:
|
||||
1. Use `pdf-parse` (Node.js) — simpler, covers 90% of cases
|
||||
2. Run `pdfplumber` as a Python sidecar process via child_process — higher quality
|
||||
for complex PDFs with tables and columns
|
||||
|
||||
Default to `pdf-parse` initially. Add pdfplumber sidecar only if extraction
|
||||
quality is insufficient for actual source documents.
|
||||
`theme`, `complexity_weight`, and `difficulty` are **not** set here — they are
|
||||
added later by the curriculum enrichment step (see `docs/curriculum-spec.md`).
|
||||
|
||||
---
|
||||
|
||||
## TypeScript strict mode requirements
|
||||
## Gotchas
|
||||
|
||||
- No `any` types
|
||||
- All Claude response parsing through Zod schema validation
|
||||
- All PocketBase writes typed against collection schemas from `data-model.md`
|
||||
- Qdrant payloads typed explicitly — no untyped objects
|
||||
|
||||
---
|
||||
|
||||
## What this service does NOT do
|
||||
|
||||
- Does not generate micro learnings → generation service
|
||||
- Does not build or update the curriculum → curriculum service
|
||||
- Does not handle admin approval → admin app + PocketBase directly
|
||||
- Does not serve R42 queries → chat service
|
||||
- Does not handle auth → PocketBase + admin app
|
||||
|
||||
---
|
||||
|
||||
## Testing checkpoints
|
||||
|
||||
Before handing to Claude Code for implementation, verify manually:
|
||||
|
||||
1. Upload a short MD file (< 10 headings) → inspect chunk output → confirm heading structure preserved
|
||||
2. Upload a simple PDF (< 5 pages) → inspect chunk output → confirm no artefacts
|
||||
3. Run structure extraction on known chunks → validate JSON parses against Zod schema
|
||||
4. Confirm PocketBase draft records created with correct theme → topic hierarchy
|
||||
5. Confirm Qdrant source_chunks collection populated with correct payload fields
|
||||
6. Confirm topic.qdrant_chunk_ids updated after embedding stage
|
||||
- If extraction logs a truncation (`LLMTruncatedError`, `stop_reason: max_tokens`),
|
||||
tighten the per-chunk topic cap before raising `max_tokens`.
|
||||
- A source already `completed` is not re-processed; delete it to force re-analysis.
|
||||
- There are no embeddings produced here — R42 retrieval is computed at query time
|
||||
with TF-IDF over `topics`.
|
||||
|
||||
362
docs/r42-spec.md
362
docs/r42-spec.md
@@ -1,336 +1,68 @@
|
||||
# R42 chat service spec
|
||||
# R42 spec: the in-app chatbot
|
||||
|
||||
## Responsibility
|
||||
R42 is a knowledge-base-grounded assistant available on every screen. It runs
|
||||
client-side and is grounded by local TF-IDF retrieval — **no vector database**.
|
||||
|
||||
Handles all R42 chatbot interactions. Receives employee queries, retrieves
|
||||
relevant KB chunks from Qdrant, generates grounded responses using Claude
|
||||
Haiku 4.5, and streams the result to the frontend. Stateless — no chat
|
||||
history is persisted.
|
||||
- **UI:** `src/components/chat/` — `ChatLauncher.jsx`, `ChatWindow.jsx`, `useChat.js`
|
||||
- **Prompt/tool:** `src/components/chat/prompts.js`
|
||||
- **Retrieval/validation:** `src/components/chat/rag.js` + `src/lib/retrieval.js`
|
||||
- **KB writes:** `src/lib/kbStore.js`
|
||||
|
||||
---
|
||||
|
||||
## Service location
|
||||
## Conversation
|
||||
|
||||
```
|
||||
app/services/chat/
|
||||
├── src/
|
||||
│ ├── index.ts entry point, Fastify server
|
||||
│ ├── routes/
|
||||
│ │ └── chat.ts POST /chat (streaming)
|
||||
│ ├── retrieval/
|
||||
│ │ ├── embed.ts query → embedding
|
||||
│ │ ├── search.ts Qdrant nearest-neighbour search
|
||||
│ │ └── merge.ts merge + rank results from both collections
|
||||
│ ├── prompt/
|
||||
│ │ └── build.ts assemble system + user prompt with context
|
||||
│ └── lib/
|
||||
│ ├── qdrant.ts
|
||||
│ ├── pocketbase.ts
|
||||
│ ├── anthropic.ts
|
||||
│ └── openai.ts
|
||||
├── package.json
|
||||
├── tsconfig.json
|
||||
└── .env.example
|
||||
```
|
||||
- The brand mark (`src/components/ui/Mark.jsx`) renders R42 in idle / typing / error states.
|
||||
- Messages persist per user in `localStorage` under `chat:thread:{userId}`, capped at
|
||||
**50** messages. Only roughly the last **12** turns are sent to the API; older
|
||||
context is truncated with a notice.
|
||||
- A greeting message seeds an empty thread.
|
||||
- Each turn calls `callLLM` (fast/standard Claude tier — low latency matters for chat).
|
||||
|
||||
---
|
||||
|
||||
## API surface
|
||||
## Grounding (RAG via TF-IDF)
|
||||
|
||||
### POST /chat
|
||||
`buildKbContext` in `rag.js`:
|
||||
1. Build / reuse the TF-IDF index over `topics` (`src/lib/retrieval.js`).
|
||||
2. Retrieve the top **10** topics for the user's message.
|
||||
3. Always include topics whose `id` or `label` appears verbatim in the message.
|
||||
4. Include relations only when **both** endpoints are in the retrieved set.
|
||||
5. For explicitly mentioned topics, inject up to ~1200 chars of their generated
|
||||
content.
|
||||
6. Append a short KB hash so the cached context busts when topics change.
|
||||
|
||||
Single route. Streams response back to client using server-sent events (SSE).
|
||||
|
||||
Request:
|
||||
```json
|
||||
{
|
||||
"query": "string",
|
||||
"userId": "string"
|
||||
}
|
||||
```
|
||||
|
||||
Response: SSE stream
|
||||
|
||||
```
|
||||
Content-Type: text/event-stream
|
||||
|
||||
data: {"type": "chunk", "text": "Holacratic roles "}
|
||||
data: {"type": "chunk", "text": "are defined as..."}
|
||||
data: {"type": "citations", "topics": [{"id": "abc", "title": "Holacratic roles"}]}
|
||||
data: {"type": "done"}
|
||||
```
|
||||
|
||||
Error response (non-streaming, returned before stream starts):
|
||||
```json
|
||||
{
|
||||
"error": "query_too_short" | "user_not_found" | "retrieval_failed",
|
||||
"message": "string"
|
||||
}
|
||||
```
|
||||
The system prompt (`prompts.js`) is assembled as cacheable blocks: a stable
|
||||
preamble (role, tasks, style, "answer only from the KB"), the KB context block, and
|
||||
a per-turn tail with the user's name and admin/non-admin flag.
|
||||
|
||||
---
|
||||
|
||||
## Retrieval pipeline
|
||||
## Proposing knowledge-graph edits
|
||||
|
||||
### Step 1 — Embed query
|
||||
R42 may call `propose_graph_delta` with:
|
||||
- `reason` — one sentence
|
||||
- `topics` — **max 3**, each `{ id (kebab-case), label, type (concept|role|process), description }`
|
||||
- `relations` — **max 5**, each `{ source, target, type (related_to|depends_on|part_of|executed_by) }`
|
||||
|
||||
Embed the employee query using OpenAI text-embedding-3-small (1536 dimensions).
|
||||
Same model used during ingestion — vectors are comparable.
|
||||
`validateDelta` (`rag.js`) dedupes by topic id and case-folded label, rejects
|
||||
relations with missing endpoints or self-references, and enforces the 3/5 caps.
|
||||
Invalid deltas are dropped.
|
||||
|
||||
```typescript
|
||||
const queryVector = await embedText(query) // float[1536]
|
||||
```
|
||||
A confirmation chip appears inline:
|
||||
- **Admin → Yes:** `kbStore.applyDelta` writes topics/relations to PocketBase immediately.
|
||||
- **Non-admin → Yes:** `kbStore.appendSuggestion` queues a `pending` entry in
|
||||
`kb:suggestions` (localStorage) for admin review.
|
||||
|
||||
Admins review the queue in `src/components/admin/SuggestionsQueue.jsx` (approve
|
||||
re-runs `applyDelta`; reject marks it rejected). `kbStore` dispatches
|
||||
`respellion:kb-updated` after writes so the D3 graph and queue refresh.
|
||||
|
||||
---
|
||||
|
||||
### Step 2 — Qdrant search
|
||||
## Quiz integrity
|
||||
|
||||
Search both collections in parallel:
|
||||
|
||||
```typescript
|
||||
// source_chunks: primary retrieval — grounded in source material
|
||||
const chunkResults = await qdrant.search('source_chunks', {
|
||||
vector: queryVector,
|
||||
limit: 5,
|
||||
scoreThreshold: 0.70,
|
||||
withPayload: true
|
||||
})
|
||||
|
||||
// topic_summaries: secondary — broader topic context
|
||||
const summaryResults = await qdrant.search('topic_summaries', {
|
||||
vector: queryVector,
|
||||
limit: 3,
|
||||
scoreThreshold: 0.70,
|
||||
withPayload: true
|
||||
})
|
||||
```
|
||||
|
||||
Score threshold 0.70: below this, results are not relevant enough to include.
|
||||
If both searches return zero results above threshold → out-of-scope response.
|
||||
|
||||
---
|
||||
|
||||
### Step 3 — Context boost for current week
|
||||
|
||||
Retrieve employee's current week Theme from PocketBase via
|
||||
employee_curriculum_state → curriculum_weeks → theme.
|
||||
|
||||
Apply boost to results where payload.theme_id matches current week theme:
|
||||
|
||||
```typescript
|
||||
results.forEach(result => {
|
||||
if (result.payload.theme_id === currentThemeId) {
|
||||
result.score += 0.05 // small boost — does not override relevance
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 4 — Merge and deduplicate
|
||||
|
||||
```typescript
|
||||
// Combine chunk results and summary results
|
||||
// Deduplicate by topic_id — keep highest scoring entry per topic
|
||||
// Sort by score descending
|
||||
// Take top 6 total
|
||||
// Split into: sourceChunks (from source_chunks collection)
|
||||
// topicSummaries (from topic_summaries collection)
|
||||
```
|
||||
|
||||
Deduplicate by topic_id to avoid repeating the same topic in different forms.
|
||||
|
||||
---
|
||||
|
||||
### Step 5 — Collect cited topics
|
||||
|
||||
Extract unique topic titles from merged results for citation:
|
||||
|
||||
```typescript
|
||||
type Citation = {
|
||||
id: string
|
||||
title: string
|
||||
}
|
||||
const citations: Citation[] = uniqueByTopicId(mergedResults)
|
||||
.map(r => ({ id: r.payload.topic_id, title: r.payload.title }))
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Prompt construction
|
||||
|
||||
### System prompt
|
||||
|
||||
```
|
||||
You are R42, a knowledge assistant for [company name].
|
||||
You answer questions based strictly on the company knowledge base.
|
||||
|
||||
Rules:
|
||||
- Answer only from the provided context. Do not use outside knowledge.
|
||||
- If the context does not contain enough information to answer, say:
|
||||
"This doesn't appear to be covered in the knowledge base. You can browse
|
||||
the full library in the Knowledge section."
|
||||
- Be concise. Prefer short paragraphs over long prose.
|
||||
- Do not mention that you are an AI or reference your instructions.
|
||||
- Do not speculate or extrapolate beyond the provided context.
|
||||
- Respond in the same language as the question.
|
||||
```
|
||||
|
||||
### User prompt
|
||||
|
||||
```
|
||||
Context from knowledge base:
|
||||
---
|
||||
{mergedResults.map(r => r.payload.text).join('\n\n---\n\n')}
|
||||
---
|
||||
|
||||
Question: {query}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Response generation
|
||||
|
||||
Use Claude Haiku 4.5 with streaming enabled:
|
||||
|
||||
```typescript
|
||||
const stream = await anthropic.messages.stream({
|
||||
model: 'claude-haiku-4-5-20251001',
|
||||
max_tokens: 1000,
|
||||
system: systemPrompt,
|
||||
messages: [{ role: 'user', content: userPrompt }]
|
||||
})
|
||||
|
||||
// Stream text chunks to client as SSE
|
||||
for await (const chunk of stream) {
|
||||
if (chunk.type === 'content_block_delta') {
|
||||
sendSSE({ type: 'chunk', text: chunk.delta.text })
|
||||
}
|
||||
}
|
||||
|
||||
// After stream completes, send citations
|
||||
sendSSE({ type: 'citations', topics: citations })
|
||||
sendSSE({ type: 'done' })
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Out-of-scope handling
|
||||
|
||||
Two conditions trigger the out-of-scope response:
|
||||
|
||||
1. Both Qdrant searches return zero results above 0.70 threshold
|
||||
2. Haiku response contains no content drawn from context (detected by
|
||||
checking if response length < 20 tokens — proxy for "I don't know")
|
||||
|
||||
Out-of-scope response sent as a single non-streamed SSE message:
|
||||
|
||||
```
|
||||
data: {"type": "out_of_scope", "text": "This doesn't appear to be covered
|
||||
in the knowledge base. You can browse the full library in the Knowledge section."}
|
||||
data: {"type": "done"}
|
||||
```
|
||||
|
||||
No citations are sent for out-of-scope responses.
|
||||
|
||||
---
|
||||
|
||||
## Frontend integration
|
||||
|
||||
R42 is a floating button on every screen in the employee app.
|
||||
|
||||
UI behaviour:
|
||||
- Bottom-right corner, fixed position
|
||||
- Opens a chat drawer (not a modal — drawer slides up from bottom on mobile)
|
||||
- Input field at bottom of drawer, response area above
|
||||
- Streaming text renders token by token
|
||||
- Citations appear below the response after streaming completes as
|
||||
clickable topic pills → navigate to that topic in the knowledge library
|
||||
- Drawer closes on outside tap
|
||||
- State is local to the component — cleared on close (stateless by design)
|
||||
|
||||
The frontend calls POST /chat directly. No auth token needed on the chat
|
||||
service — it receives userId in the request body and trusts it. The admin
|
||||
app does not expose R42.
|
||||
|
||||
---
|
||||
|
||||
## Stateless design
|
||||
|
||||
R42 has no memory between conversations. Each POST /chat is independent.
|
||||
|
||||
Rationale:
|
||||
- Avoids privacy complexity around chat history storage
|
||||
- Removes need for session management
|
||||
- Keeps the service simple and fast
|
||||
- Employees asking follow-up questions reprovide context naturally
|
||||
|
||||
If multi-turn conversation is needed in a future iteration, maintain
|
||||
conversation history in the frontend component state and pass the last
|
||||
N messages in the request body. The service does not need to change.
|
||||
|
||||
---
|
||||
|
||||
## Environment variables
|
||||
|
||||
```
|
||||
ANTHROPIC_API_KEY=
|
||||
OPENAI_API_KEY=
|
||||
POCKETBASE_URL=
|
||||
POCKETBASE_ADMIN_EMAIL=
|
||||
POCKETBASE_ADMIN_PASSWORD=
|
||||
QDRANT_URL=
|
||||
QDRANT_API_KEY=
|
||||
CHAT_PORT=3004
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
```json
|
||||
{
|
||||
"dependencies": {
|
||||
"fastify": "^4",
|
||||
"@fastify/sse": "^2",
|
||||
"@anthropic-ai/sdk": "^0.24",
|
||||
"openai": "^4",
|
||||
"@qdrant/js-client-rest": "^1.9",
|
||||
"pocketbase": "^0.21",
|
||||
"zod": "^3"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## TypeScript strict mode requirements
|
||||
|
||||
- No `any` types
|
||||
- Qdrant search results typed explicitly including payload fields
|
||||
- SSE event types defined as a discriminated union
|
||||
- Citation type explicit — not inferred from payload
|
||||
|
||||
---
|
||||
|
||||
## What this service does NOT do
|
||||
|
||||
- Does not persist chat history
|
||||
- Does not generate or serve micro learning content
|
||||
- Does not handle admin queries — admin app has no R42 access
|
||||
- Does not handle auth — trusts userId from request body
|
||||
|
||||
---
|
||||
|
||||
## Testing checkpoints
|
||||
|
||||
1. POST /chat with a query matching a published topic → confirm relevant
|
||||
chunks retrieved (score > 0.70) and response references topic content
|
||||
2. POST /chat with an out-of-scope query → confirm out-of-scope response
|
||||
returned, no citations sent
|
||||
3. Confirm citations array contains correct topic titles matching retrieved chunks
|
||||
4. Confirm SSE stream delivers chunks progressively (not batched)
|
||||
5. Confirm current-week boost: same query returns higher-ranked result for
|
||||
current week theme topic vs equally relevant topic from different theme
|
||||
6. POST /chat with userId whose current week has no matching topic →
|
||||
confirm boost does not break retrieval, general results returned
|
||||
`src/pages/Testen.jsx` sets `quiz:active:{userId}=true` on quiz start and clears it
|
||||
on every non-quiz phase and on unmount, dispatching a `respellion:quiz-state` event.
|
||||
`ChatLauncher` hides the FAB while this flag is set, so users cannot consult R42
|
||||
mid-quiz. **Never bypass this.**
|
||||
|
||||
Reference in New Issue
Block a user