Add comprehensive documentation for employee learning platform
- Created handover document outlining design decisions and application functionality. - Developed implementation plan detailing phased approach for service development. - Specified ingestion service responsibilities, API surface, and processing pipeline.
This commit is contained in:
RaymondVerhoef
279
docs/architecture.md
Normal file
279
docs/architecture.md
Normal file
@@ -0,0 +1,279 @@
|
||||
# Architecture: employee learning platform
|
||||
|
||||
## Overview
|
||||
|
||||
A mobile-first progressive web application that provides employees with a structured knowledge library, a 26-week perpetual learning curriculum, and an AI-powered assistant (R42). The knowledge base is the single source of truth for all content, micro learnings, curriculum scheduling, and chat retrieval.
|
||||
|
||||
---
|
||||
|
||||
## System domains
|
||||
|
||||
### Admin app
|
||||
Browser-based interface for content administrators.
|
||||
|
||||
Responsibilities:
|
||||
- Upload source documents (PDF, MD, TXT)
|
||||
- Review and approve AI-generated Theme batches
|
||||
- Edit and finetune AI-generated curriculum
|
||||
- Confirm curriculum regeneration after KB updates
|
||||
- Monitor ingestion and generation job status
|
||||
|
||||
### Employee app
|
||||
Mobile-first PWA accessible on all devices.
|
||||
|
||||
Responsibilities:
|
||||
- Weekly session delivery (Theme + Topics + micro learning type selection)
|
||||
- Knowledge library (browse all published Topics)
|
||||
- Gamification profile (heatmap, badges, streak, leaderboard)
|
||||
- R42 chatbot (available on every screen)
|
||||
|
||||
### Backend services
|
||||
Six discrete services, each with a single responsibility.
|
||||
|
||||
| Service | Responsibility |
|
||||
|---|---|
|
||||
| Ingestion service | Document upload → chunk → extract KB structure |
|
||||
| Generation service | Topics → 10 micro learning types (structured JSON) |
|
||||
| Curriculum service | KB graph → 26-week schedule, versioning, regeneration |
|
||||
| Embedding service | Chunks + topic summaries → Qdrant |
|
||||
| Chat service (R42) | Query → vector retrieval → grounded response |
|
||||
| Progress service | Completions → XP → badges → streak |
|
||||
|
||||
---
|
||||
|
||||
## Deployment topology
|
||||
|
||||
```
|
||||
repo/
|
||||
├── .github/workflows/ ← pipeline (frozen)
|
||||
├── docker-compose.yml ← infrastructure (frozen)
|
||||
├── Dockerfile ← updated once to point at /app
|
||||
├── ansible/ ← provisioning (frozen)
|
||||
├── legacy/ ← original prototype (read-only reference)
|
||||
└── app/
|
||||
├── frontend/ ← Next.js PWA (admin + employee)
|
||||
└── services/
|
||||
├── ingestion/
|
||||
├── generation/
|
||||
├── curriculum/
|
||||
├── embedding/
|
||||
├── chat/
|
||||
└── progress/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Tech stack
|
||||
|
||||
| Layer | Technology | Rationale |
|
||||
|---|---|---|
|
||||
| Frontend | Next.js 14, TypeScript, Tailwind CSS | PWA support, single codebase for admin + employee |
|
||||
| Backend state | PocketBase | Auth, file storage, admin UI, SQLite — no infra overhead |
|
||||
| Vector store | Qdrant (Docker) | RAG retrieval, runs as single container |
|
||||
| AI generation | Claude Sonnet 4 via Anthropic API | Structured JSON output, long-form drafting, graph reasoning |
|
||||
| AI chat (R42) | Claude Haiku 4.5 via Anthropic API | Low latency, cost-effective, grounded by retrieval layer |
|
||||
| Embeddings | OpenAI text-embedding-3-small | Cost-effective, high quality at this scale |
|
||||
| Auth | PocketBase built-in | Role-based: admin / employee |
|
||||
|
||||
---
|
||||
|
||||
## AI model responsibilities
|
||||
|
||||
| Task | Model |
|
||||
|---|---|
|
||||
| Document → KB structure extraction | Claude Sonnet 4 |
|
||||
| Topic body drafting | Claude Sonnet 4 |
|
||||
| Micro learning generation (all 10 types) | Claude Sonnet 4 |
|
||||
| Curriculum generation + versioning | Claude Sonnet 4 |
|
||||
| R42 chat responses | Claude Haiku 4.5 |
|
||||
| Embeddings | text-embedding-3-small |
|
||||
|
||||
---
|
||||
|
||||
## Document ingestion pipeline
|
||||
|
||||
```
|
||||
Admin uploads file (PDF / MD / TXT)
|
||||
↓
|
||||
Format detection → text extraction
|
||||
MD: split on headings → preserve hierarchy
|
||||
PDF: pdfplumber → page + paragraph detection
|
||||
TXT: sliding window chunking with overlap
|
||||
↓
|
||||
Chunk cleaning (strip headers/footers/artefacts)
|
||||
↓
|
||||
Claude Sonnet 4 reads chunks → extracts:
|
||||
- candidate Themes
|
||||
- candidate Topics per Theme
|
||||
- Topic→Topic relationships (related, prerequisite, contrast)
|
||||
- key terms for glossary
|
||||
↓
|
||||
Draft KB written to PocketBase (status: draft)
|
||||
↓
|
||||
Embedding service: embed source chunks → write to Qdrant
|
||||
↓
|
||||
Admin reviews Theme batch → approves / edits / rejects
|
||||
↓
|
||||
On approval: Topics published, micro learning generation queued
|
||||
↓
|
||||
Curriculum regeneration notification queued for admin
|
||||
```
|
||||
|
||||
Note: embeddings are generated from **source chunks**, not only from AI-generated topic summaries. R42 retrieves from grounded source material.
|
||||
|
||||
MD source files are the preferred format for admins — heading structure maps directly to Theme → Topic hierarchy and improves extraction quality.
|
||||
|
||||
---
|
||||
|
||||
## Curriculum lifecycle
|
||||
|
||||
### Generation
|
||||
Input: all published Themes, Topics, relationship graph, complexity weights
|
||||
Process: cluster by Theme → sequence pedagogically (prerequisites first, complexity gradient) → distribute across 26 weeks → ensure full KB coverage
|
||||
Output: versioned 26-week draft schedule
|
||||
|
||||
### Perpetual cycling
|
||||
The curriculum runs continuously. After week 26, the employee begins cycle 2 on the latest curriculum version.
|
||||
|
||||
Second and subsequent cycles are not identical to cycle 1:
|
||||
- Theme sequence is varied
|
||||
- Recommended micro learning types surface types the employee has not yet used
|
||||
- Topics with low engagement in prior cycles receive increased coverage
|
||||
|
||||
### Versioning rules
|
||||
|
||||
| Event | Action |
|
||||
|---|---|
|
||||
| New source doc published to KB | Regenerate curriculum from week N+1 for all active employees |
|
||||
| Topic body edited | Micro learnings regenerated; curriculum unaffected |
|
||||
| Theme batch approved | Regeneration queued; admin confirms before it applies |
|
||||
|
||||
Completed weeks are immutable. Regeneration only affects future unstarted weeks.
|
||||
|
||||
### Admin regeneration flow
|
||||
Admin receives notification: "N new topics added. Regenerate curriculum? This will update unstarted weeks for all active employees."
|
||||
Admin can preview the proposed new schedule before confirming.
|
||||
|
||||
---
|
||||
|
||||
## Weekly session flow (employee)
|
||||
|
||||
```
|
||||
Week N opens
|
||||
↓
|
||||
Employee sees assigned Theme + Topics for the week
|
||||
↓
|
||||
Per Topic: employee selects micro learning type
|
||||
(all published types for that topic are available)
|
||||
↓
|
||||
Employee completes one or more types per topic
|
||||
↓
|
||||
Completion recorded → XP awarded → badges evaluated
|
||||
↓
|
||||
Progress visible on public leaderboard and activity feed
|
||||
```
|
||||
|
||||
Sessions support multiple micro learning types per topic in a single session.
|
||||
|
||||
---
|
||||
|
||||
## Micro learning types
|
||||
|
||||
All 10 types are generated by Claude Sonnet 4 as structured JSON, stored in PocketBase, and rendered by the frontend. One or more types may be published per topic.
|
||||
|
||||
| # | Type | Format |
|
||||
|---|---|---|
|
||||
| 1 | Concept explainer | 2–3 paragraphs + example |
|
||||
| 2 | Scenario quiz | situation + 3–4 MCQ options + explained answers |
|
||||
| 3 | Common misconceptions | 3–5 false beliefs + corrections |
|
||||
| 4 | Step-by-step how-to | numbered procedure |
|
||||
| 5 | Comparison card | side-by-side on 4–6 dimensions |
|
||||
| 6 | Reflection prompt | open question + model answer |
|
||||
| 7 | Spaced repetition flashcards | 5–10 Q&A pairs |
|
||||
| 8 | Case study mini-analysis | 150–200 word scenario + guiding questions |
|
||||
| 9 | Glossary anchor | term + definition + correct use + misuse |
|
||||
| 10 | Myth vs. evidence | false claim + evidence-based rebuttal |
|
||||
|
||||
---
|
||||
|
||||
## R42 — chat service design
|
||||
|
||||
R42 is a functional KB-grounded assistant available on every screen in the employee app.
|
||||
|
||||
Behaviour:
|
||||
- Stateless per session (no memory between conversations)
|
||||
- Retrieves relevant chunks from Qdrant using the employee's query
|
||||
- Knows the employee's current curriculum week → retrieval is context-weighted
|
||||
- Cites source topic in every response ("based on the **Holacratic roles** topic")
|
||||
- Explicitly refuses to answer outside KB scope rather than hallucinating
|
||||
- Scope: internal KB only
|
||||
|
||||
Implementation:
|
||||
- Employee query → embed → Qdrant nearest-neighbour retrieval → top-K chunks
|
||||
- Chunks + employee context injected into Haiku 4.5 prompt
|
||||
- Response streamed to frontend
|
||||
|
||||
UI: floating button bottom-right, unobtrusive on mobile.
|
||||
|
||||
---
|
||||
|
||||
## Gamification system
|
||||
|
||||
Inspired by the visual language of GitHub, Stack Overflow, and Duolingo. Mechanics use developer-native terminology.
|
||||
|
||||
### XP unit: commits
|
||||
Every completed topic earns commits. Quantity varies by micro learning type complexity.
|
||||
|
||||
### Levels
|
||||
`Intern → Junior → Medior → Senior → Staff → Principal`
|
||||
Based on cumulative commits across all cycles.
|
||||
|
||||
### Streak
|
||||
Counted in consecutive weeks, not days. Resets if a week is skipped entirely.
|
||||
|
||||
### Activity heatmap
|
||||
GitHub-style contribution graph spanning the full 26-week cycle. Cell darkness = number of types completed that week.
|
||||
|
||||
### Badges
|
||||
|
||||
| Tier | Condition |
|
||||
|---|---|
|
||||
| Bronze | Complete any session |
|
||||
| Silver | 5 sessions completed, 5 different types used |
|
||||
| Gold | 13 sessions without skipping a week |
|
||||
| Legendary | All 26 sessions, all 10 types used at least once |
|
||||
|
||||
Named content badges (examples):
|
||||
- `governance nerd` — all holacratic structure topics completed
|
||||
- `process architect` — all internal process topics completed
|
||||
- `deep reader` — case study type used 5+ times
|
||||
|
||||
### Milestone cards (public)
|
||||
At weeks 13 and 26, a public card is posted to the shared activity feed:
|
||||
|
||||
```
|
||||
🚀 [Name] shipped the full curriculum
|
||||
26 weeks · [N] commits · [badges]
|
||||
Longest streak: [N] weeks
|
||||
```
|
||||
|
||||
Language: shipping vocabulary, not school vocabulary.
|
||||
|
||||
### Leaderboard
|
||||
Not ranked 1–N by score. Displays multiple dimensions:
|
||||
|
||||
| Employee | Commits | Streak | Types used | Badges |
|
||||
|---|---|---|---|---|
|
||||
|
||||
Multiple paths to visibility. No single metric determines standing.
|
||||
|
||||
---
|
||||
|
||||
## Security and privacy
|
||||
|
||||
- Auth: PocketBase role-based (admin / employee)
|
||||
- Gamification data (commits, badges, streak) is public to all employees
|
||||
- Session completion data (which topic, which type, when) is public
|
||||
- Source documents are admin-only
|
||||
- No PII beyond display name stored in gamification context
|
||||
- R42 is stateless — no chat history persisted
|
||||
397
docs/data-model.md
Normal file
397
docs/data-model.md
Normal file
@@ -0,0 +1,397 @@
|
||||
# Data model: employee learning platform
|
||||
|
||||
## Overview
|
||||
|
||||
Two storage systems:
|
||||
- **PocketBase** — all structured relational data (SQLite under the hood)
|
||||
- **Qdrant** — all vector embeddings for RAG retrieval
|
||||
|
||||
---
|
||||
|
||||
## PocketBase collections
|
||||
|
||||
### `source_documents`
|
||||
Uploaded source files. Parent of all generated KB content.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| filename | string | original filename |
|
||||
| file | file | PocketBase file storage |
|
||||
| format | select | `pdf` `md` `txt` |
|
||||
| status | select | `processing` `processed` `failed` |
|
||||
| ingested_at | datetime | |
|
||||
| chunk_count | number | total chunks extracted |
|
||||
| created_by | relation → `users` | admin who uploaded |
|
||||
|
||||
---
|
||||
|
||||
### `themes`
|
||||
Top-level content groupings. One Theme = one weekly session.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| title | string | |
|
||||
| description | text | AI drafted, admin editable |
|
||||
| status | select | `draft` `published` |
|
||||
| source_documents | relation[] → `source_documents` | which docs contributed |
|
||||
| approved_by | relation → `users` | admin who approved batch |
|
||||
| approved_at | datetime | |
|
||||
| created_at | datetime | |
|
||||
| updated_at | datetime | |
|
||||
|
||||
---
|
||||
|
||||
### `topics`
|
||||
Atomic knowledge units. Always belong to a Theme.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| theme | relation → `themes` | parent theme |
|
||||
| title | string | |
|
||||
| body | text (rich) | AI drafted, admin editable |
|
||||
| difficulty | select | `introductory` `intermediate` `advanced` |
|
||||
| complexity_weight | number | 1–5, used by curriculum generator |
|
||||
| status | select | `draft` `published` |
|
||||
| related_topics | relation[] → `topics` | lateral relationships |
|
||||
| prerequisite_topics | relation[] → `topics` | must-complete-first |
|
||||
| contrast_topics | relation[] → `topics` | deliberate opposites |
|
||||
| key_terms | json | string[] — feeds glossary |
|
||||
| qdrant_chunk_ids | json | string[] — references to embedded chunks |
|
||||
| created_at | datetime | |
|
||||
| updated_at | datetime | |
|
||||
|
||||
Relationship types (related / prerequisite / contrast) are stored via the three explicit relation fields rather than a generic relationship table. This keeps queries simple at this scale.
|
||||
|
||||
---
|
||||
|
||||
### `micro_learnings`
|
||||
Generated content artifacts. One record per topic per type.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| topic | relation → `topics` | |
|
||||
| type | select | see type enum below |
|
||||
| content | json | structured output — schema varies per type |
|
||||
| status | select | `queued` `generated` `published` `rejected` |
|
||||
| generation_model | string | model version used |
|
||||
| generated_at | datetime | |
|
||||
| published_at | datetime | |
|
||||
| updated_at | datetime | |
|
||||
|
||||
**Type enum:**
|
||||
`concept_explainer` `scenario_quiz` `misconceptions` `how_to` `comparison_card` `reflection_prompt` `flashcard_set` `case_study` `glossary_anchor` `myth_vs_evidence`
|
||||
|
||||
**Content JSON schemas per type:**
|
||||
|
||||
```json
|
||||
// concept_explainer
|
||||
{
|
||||
"paragraphs": ["string", "string"],
|
||||
"example": "string"
|
||||
}
|
||||
|
||||
// scenario_quiz
|
||||
{
|
||||
"scenario": "string",
|
||||
"options": [
|
||||
{ "label": "A", "text": "string", "correct": true, "explanation": "string" }
|
||||
]
|
||||
}
|
||||
|
||||
// misconceptions
|
||||
{
|
||||
"items": [
|
||||
{ "misconception": "string", "correction": "string" }
|
||||
]
|
||||
}
|
||||
|
||||
// how_to
|
||||
{
|
||||
"steps": [
|
||||
{ "number": 1, "instruction": "string" }
|
||||
]
|
||||
}
|
||||
|
||||
// comparison_card
|
||||
{
|
||||
"subject_a": "string",
|
||||
"subject_b": "string",
|
||||
"dimensions": [
|
||||
{ "label": "string", "a": "string", "b": "string" }
|
||||
]
|
||||
}
|
||||
|
||||
// reflection_prompt
|
||||
{
|
||||
"prompt": "string",
|
||||
"model_answer": "string"
|
||||
}
|
||||
|
||||
// flashcard_set
|
||||
{
|
||||
"cards": [
|
||||
{ "question": "string", "answer": "string" }
|
||||
]
|
||||
}
|
||||
|
||||
// case_study
|
||||
{
|
||||
"scenario": "string",
|
||||
"questions": ["string"]
|
||||
}
|
||||
|
||||
// glossary_anchor
|
||||
{
|
||||
"term": "string",
|
||||
"definition": "string",
|
||||
"correct_use": "string",
|
||||
"misuse": "string"
|
||||
}
|
||||
|
||||
// myth_vs_evidence
|
||||
{
|
||||
"myth": "string",
|
||||
"evidence": "string",
|
||||
"sources": ["string"]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `curriculum_versions`
|
||||
Versioned 26-week schedule. New version created on each regeneration.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| version | number | increments on each regeneration |
|
||||
| status | select | `draft` `active` `superseded` |
|
||||
| generated_at | datetime | |
|
||||
| approved_by | relation → `users` | admin who confirmed regeneration |
|
||||
| approved_at | datetime | |
|
||||
| generation_notes | text | why this version was created |
|
||||
|
||||
Only one version has status `active` at any time.
|
||||
|
||||
---
|
||||
|
||||
### `curriculum_weeks`
|
||||
Individual week slots. Child of a curriculum version.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| curriculum_version | relation → `curriculum_versions` | |
|
||||
| week_number | number | 1–26 |
|
||||
| theme | relation → `themes` | |
|
||||
| topics | relation[] → `topics` | ordered subset of theme topics |
|
||||
| topic_order | json | number[] — explicit ordering |
|
||||
| estimated_duration_minutes | number | AI estimate |
|
||||
| admin_notes | text | freeform admin annotation |
|
||||
|
||||
---
|
||||
|
||||
### `employee_curriculum_state`
|
||||
Tracks each employee's position in the curriculum. One record per employee.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| user | relation → `users` | |
|
||||
| current_cycle | number | starts at 1, increments on loop |
|
||||
| current_week | number | 1–26 |
|
||||
| start_date | datetime | rolling start |
|
||||
| active_version | relation → `curriculum_versions` | version employee is on |
|
||||
| updated_at | datetime | |
|
||||
|
||||
When curriculum regenerates: `active_version` updates for all employees whose `current_week` is less than the first regenerated week.
|
||||
|
||||
---
|
||||
|
||||
### `session_completions`
|
||||
Immutable completion records. One record per employee per topic per type.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| user | relation → `users` | |
|
||||
| topic | relation → `topics` | |
|
||||
| micro_learning | relation → `micro_learnings` | specific type completed |
|
||||
| week_number | number | curriculum week at time of completion |
|
||||
| cycle | number | which cycle |
|
||||
| completed_at | datetime | |
|
||||
|
||||
Records are never updated or deleted. This is the canonical history.
|
||||
|
||||
---
|
||||
|
||||
### `gamification_profiles`
|
||||
One record per employee. Updated by progress service on each completion.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| user | relation → `users` | |
|
||||
| total_commits | number | cumulative XP |
|
||||
| current_level | select | `intern` `junior` `medior` `senior` `staff` `principal` |
|
||||
| current_streak_weeks | number | consecutive weeks with ≥1 completion |
|
||||
| longest_streak_weeks | number | all-time high |
|
||||
| types_used | json | string[] — which of 10 types used at least once |
|
||||
| last_active_week | number | used to detect streak breaks |
|
||||
| updated_at | datetime | |
|
||||
|
||||
---
|
||||
|
||||
### `badges`
|
||||
Badge definitions. Seeded at startup, not user-generated.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| key | string | unique slug e.g. `governance_nerd` |
|
||||
| tier | select | `bronze` `silver` `gold` `legendary` `content` |
|
||||
| label | string | display name |
|
||||
| description | string | award condition description |
|
||||
| icon | string | emoji or icon key |
|
||||
|
||||
---
|
||||
|
||||
### `employee_badges`
|
||||
Junction: which employees have earned which badges.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| user | relation → `users` | |
|
||||
| badge | relation → `badges` | |
|
||||
| earned_at | datetime | |
|
||||
| cycle | number | which cycle it was earned in |
|
||||
|
||||
---
|
||||
|
||||
### `milestone_cards`
|
||||
Public milestone events at weeks 13 and 26.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | PocketBase auto |
|
||||
| user | relation → `users` | |
|
||||
| cycle | number | |
|
||||
| week | number | 13 or 26 |
|
||||
| total_commits | number | snapshot at time of milestone |
|
||||
| streak_weeks | number | snapshot |
|
||||
| badge_keys | json | string[] — badges held at milestone |
|
||||
| created_at | datetime | public feed ordered by this |
|
||||
|
||||
---
|
||||
|
||||
## PocketBase users collection (extended)
|
||||
|
||||
Standard PocketBase `users` collection with additional fields:
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| role | select | `admin` `employee` |
|
||||
| display_name | string | used in gamification feed |
|
||||
| avatar | file | optional |
|
||||
|
||||
---
|
||||
|
||||
## Qdrant collections
|
||||
|
||||
### `source_chunks`
|
||||
Embeddings of raw source document chunks. Primary retrieval target for R42.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | UUID |
|
||||
| vector | float[] | 1536 dimensions (text-embedding-3-small) |
|
||||
| source_document_id | string | reference to PocketBase |
|
||||
| chunk_index | number | position within document |
|
||||
| text | string | raw chunk text |
|
||||
| theme_id | string | assigned theme (post-extraction) |
|
||||
| topic_id | string | assigned topic (post-extraction, nullable) |
|
||||
| format | string | pdf / md / txt |
|
||||
|
||||
### `topic_summaries`
|
||||
Embeddings of AI-generated topic body text. Secondary retrieval target.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | string | UUID |
|
||||
| vector | float[] | 1536 dimensions |
|
||||
| topic_id | string | reference to PocketBase |
|
||||
| theme_id | string | |
|
||||
| title | string | for display in R42 citation |
|
||||
| text | string | full topic body |
|
||||
|
||||
---
|
||||
|
||||
## Retrieval strategy for R42
|
||||
|
||||
R42 queries both Qdrant collections and merges results:
|
||||
|
||||
```
|
||||
Employee query
|
||||
↓
|
||||
Embed query → text-embedding-3-small
|
||||
↓
|
||||
Qdrant search: source_chunks (top 5) + topic_summaries (top 3)
|
||||
↓
|
||||
Filter: boost chunks from employee's current week theme
|
||||
↓
|
||||
Merge + deduplicate by topic_id
|
||||
↓
|
||||
Top-K context injected into Haiku 4.5 prompt
|
||||
↓
|
||||
Response includes: answer + cited topic title(s)
|
||||
```
|
||||
|
||||
Source chunks are weighted higher than topic summaries to keep R42 grounded in original source material rather than AI-generated abstractions.
|
||||
|
||||
---
|
||||
|
||||
## Indexes and query patterns
|
||||
|
||||
Critical query patterns the data model must support efficiently:
|
||||
|
||||
| Query | Collection | Index |
|
||||
|---|---|---|
|
||||
| All published topics for a theme | topics | theme + status |
|
||||
| All micro learnings for a topic | micro_learnings | topic + status |
|
||||
| Employee's current week | employee_curriculum_state | user |
|
||||
| Weeks for a curriculum version | curriculum_weeks | curriculum_version + week_number |
|
||||
| Employee completion history | session_completions | user + cycle |
|
||||
| Public leaderboard | gamification_profiles | total_commits + streak |
|
||||
| Milestone feed | milestone_cards | created_at DESC |
|
||||
| Badges earned by employee | employee_badges | user |
|
||||
|
||||
PocketBase creates indexes on relation fields by default. Composite indexes on `status` fields should be added manually where query frequency warrants it.
|
||||
|
||||
---
|
||||
|
||||
## Data flow summary
|
||||
|
||||
```
|
||||
source_documents
|
||||
└── (ingestion service)
|
||||
└── qdrant: source_chunks
|
||||
└── themes (draft)
|
||||
└── topics (draft)
|
||||
└── (approval)
|
||||
└── topics (published)
|
||||
└── qdrant: topic_summaries
|
||||
└── micro_learnings (queued → published)
|
||||
└── (curriculum service)
|
||||
└── curriculum_versions
|
||||
└── curriculum_weeks
|
||||
└── (employee progress)
|
||||
└── session_completions
|
||||
└── gamification_profiles
|
||||
└── employee_badges
|
||||
└── milestone_cards
|
||||
```
|
||||
231
docs/handover.md
Normal file
231
docs/handover.md
Normal file
@@ -0,0 +1,231 @@
|
||||
# Handover: employee learning platform
|
||||
|
||||
## Purpose of this document
|
||||
|
||||
This document captures every design decision made before implementation started.
|
||||
It is the authoritative source for rationale. When a spec file is ambiguous,
|
||||
resolve it against this document. Do not ask the human — the answers are here.
|
||||
|
||||
---
|
||||
|
||||
## What this application does
|
||||
|
||||
Employees of a tech company use this platform to build and maintain knowledge of
|
||||
the employee handbook, holacratic structures, and internal processes.
|
||||
|
||||
Core mechanics:
|
||||
- Admins upload source documents → AI extracts a structured knowledge base
|
||||
- The KB is organised into Themes (broad) and Topics (specific)
|
||||
- An AI generates 10 types of micro learning content per Topic
|
||||
- Employees follow a 26-week curriculum of weekly sessions
|
||||
- Each session covers one Theme (multiple related Topics)
|
||||
- Employees choose which micro learning type to use per Topic
|
||||
- After 26 weeks the curriculum restarts, varied to reinforce rather than repeat
|
||||
- A chatbot called R42 answers KB-grounded questions on every screen
|
||||
- A gamification system using developer-native language motivates completion
|
||||
|
||||
---
|
||||
|
||||
## All confirmed design decisions
|
||||
|
||||
### Knowledge base
|
||||
|
||||
**Decision: KB is extracted from source documents, not manually authored**
|
||||
Admins upload raw source material. Claude Sonnet 4 extracts Themes, Topics, and
|
||||
relationships. Admins review and approve in batches (one Theme at a time, not
|
||||
one Topic at a time). Topic bodies are AI-drafted and admin-editable after approval.
|
||||
|
||||
**Decision: two-level hierarchy — Theme → Topic**
|
||||
A Theme is a broad subject area. A Topic is one specific concept within a Theme.
|
||||
One weekly session = one Theme. Multiple Topics within that Theme per session.
|
||||
|
||||
**Decision: three relationship types between Topics**
|
||||
- related: Topics that complement each other
|
||||
- prerequisite: Topic A should be understood before Topic B
|
||||
- contrast: Topics representing opposing approaches
|
||||
|
||||
These relationships are stored as explicit PocketBase relation fields, not a
|
||||
generic junction table.
|
||||
|
||||
**Decision: source material format priority**
|
||||
Accepted formats: PDF, MD, TXT only. MD is the highest quality input —
|
||||
heading structure maps directly to Theme → Topic hierarchy. Admins should be
|
||||
recommended to provide MD where possible.
|
||||
|
||||
**Decision: embeddings from source chunks, not topic summaries only**
|
||||
R42 retrieves from original source material chunks as primary source, with
|
||||
topic summaries as secondary. This keeps R42 grounded and reduces hallucination.
|
||||
|
||||
---
|
||||
|
||||
### Micro learnings
|
||||
|
||||
**Decision: 10 types, all generated by AI as structured JSON**
|
||||
Types:
|
||||
1. concept_explainer
|
||||
2. scenario_quiz
|
||||
3. misconceptions
|
||||
4. how_to
|
||||
5. comparison_card
|
||||
6. reflection_prompt
|
||||
7. flashcard_set
|
||||
8. case_study
|
||||
9. glossary_anchor
|
||||
10. myth_vs_evidence
|
||||
|
||||
Each type has a defined JSON schema in data-model.md. Generation uses
|
||||
Claude Sonnet 4. Output is validated against Zod schemas before storage.
|
||||
|
||||
**Decision: employees choose type per topic per session**
|
||||
Employees are not locked to one type globally. Each session, per Topic, the
|
||||
employee selects from all published types for that topic. Multiple types can
|
||||
be completed in one session.
|
||||
|
||||
**Decision: pre-generate, don't generate on demand**
|
||||
All 10 types are generated when a Topic is approved, not when an employee
|
||||
requests them. This controls quality and cost. R42 is the only on-demand
|
||||
AI interaction.
|
||||
|
||||
---
|
||||
|
||||
### Curriculum
|
||||
|
||||
**Decision: AI generates curriculum, admin edits**
|
||||
Claude Sonnet 4 reads the full KB graph (Themes, Topics, relationships,
|
||||
complexity weights) and produces a 26-week schedule. Admin reviews, reorders,
|
||||
and finetunes. Admin does not build from scratch.
|
||||
|
||||
**Decision: one Theme per week session**
|
||||
A session covers all Topics under one Theme. Topics within the session are
|
||||
ordered by the curriculum generator based on prerequisites and complexity.
|
||||
|
||||
**Decision: perpetual curriculum with versioning**
|
||||
The curriculum runs indefinitely. After week 26, cycle 2 begins on the latest
|
||||
curriculum version. Cycle 2+ varies sequence, surfaces unused micro learning
|
||||
types, and increases coverage of low-engagement topics.
|
||||
|
||||
**Decision: completed weeks are immutable**
|
||||
Regeneration only affects future unstarted weeks. An employee's completion
|
||||
history is never altered regardless of curriculum version changes.
|
||||
|
||||
**Decision: regeneration requires admin confirmation**
|
||||
When new Topics are approved, the system queues a regeneration but does not
|
||||
apply it until the admin explicitly confirms. Admin sees a preview of the
|
||||
proposed new schedule before confirming.
|
||||
|
||||
**Decision: rolling starts**
|
||||
Each employee has their own start date. There are no cohorts or shared
|
||||
start dates.
|
||||
|
||||
---
|
||||
|
||||
### Gamification
|
||||
|
||||
**Decision: developer-native visual language**
|
||||
Inspired by GitHub (heatmap), Stack Overflow (badges, reputation), and
|
||||
Duolingo (streak, XP, levels). Language uses developer vocabulary throughout.
|
||||
|
||||
**Decision: XP unit is called commits**
|
||||
Every completed Topic earns commits. Quantity varies by micro learning type.
|
||||
|
||||
**Decision: levels use developer rank names**
|
||||
Intern → Junior → Medior → Senior → Staff → Principal
|
||||
Based on cumulative commits across all cycles.
|
||||
|
||||
**Decision: streak is weekly, not daily**
|
||||
Consecutive weeks with at least one completion. Resets on a skipped week.
|
||||
|
||||
**Decision: activity heatmap covers 26-week cycle**
|
||||
GitHub-style contribution graph. Cell darkness = number of types completed
|
||||
that week.
|
||||
|
||||
**Decision: no social layer**
|
||||
No comments, reactions, or direct messaging. Gamification is visible but
|
||||
not interactive between employees.
|
||||
|
||||
**Decision: public milestone cards, not ranked leaderboard**
|
||||
At weeks 13 and 26, a public card is posted to the shared activity feed.
|
||||
Language: "shipped", not "graduated". The leaderboard shows multiple
|
||||
dimensions (commits, streak, types used, badges) — not a single ranking.
|
||||
|
||||
**Decision: named content badges**
|
||||
Examples: governance_nerd, process_architect, deep_reader. These are seeded
|
||||
at startup, not user-generated. See data-model.md for badge schema.
|
||||
|
||||
---
|
||||
|
||||
### R42 chatbot
|
||||
|
||||
**Decision: functional only, no personality**
|
||||
R42 answers questions grounded in the KB. It does not have a defined persona,
|
||||
tone, or name story beyond the label R42.
|
||||
|
||||
**Decision: stateless per session**
|
||||
No chat history is persisted between sessions. This avoids privacy complexity
|
||||
and keeps the implementation simple.
|
||||
|
||||
**Decision: internal KB scope only**
|
||||
R42 cannot search external sources. If a question cannot be answered from the
|
||||
KB, R42 says so explicitly.
|
||||
|
||||
**Decision: context-weighted retrieval**
|
||||
R42 knows the employee's current curriculum week. Retrieval boosts chunks
|
||||
from the current week's Theme. General KB questions are not restricted.
|
||||
|
||||
**Decision: always cites source Topic**
|
||||
Every R42 response includes the Topic title(s) its answer draws from.
|
||||
|
||||
**Decision: Haiku 4.5 for R42, Sonnet 4 for generation**
|
||||
Low latency matters for chat. The retrieval layer compensates for Haiku's
|
||||
smaller knowledge base. Sonnet 4 is reserved for generation tasks where
|
||||
structure and quality matter more than speed.
|
||||
|
||||
---
|
||||
|
||||
### Infrastructure
|
||||
|
||||
**Decision: PocketBase as primary backend**
|
||||
Auth, file storage, structured data, and admin UI in one binary. SQLite is
|
||||
sufficient for ~150 users. No PostgreSQL needed at this scale.
|
||||
|
||||
**Decision: Qdrant for vector storage**
|
||||
Separate Docker container. Keeps vector operations out of SQLite.
|
||||
pgvector was rejected — adding Postgres just for vectors is unnecessary overhead.
|
||||
|
||||
**Decision: Next.js 14 PWA for frontend**
|
||||
Single codebase for admin and employee app. PWA covers mobile without a native
|
||||
app. Learning platforms do not require native device APIs.
|
||||
|
||||
**Decision: five discrete backend services**
|
||||
Ingestion, generation, curriculum, chat, progress. Each is a separate Fastify
|
||||
service with its own port and responsibility. They do not call each other
|
||||
directly — they read/write shared PocketBase collections.
|
||||
|
||||
**Decision: PDF parsing starts with pdf-parse (Node.js)**
|
||||
Switch to pdfplumber Python sidecar only if pdf-parse quality is insufficient
|
||||
for actual source documents. Do not over-engineer the extraction layer upfront.
|
||||
|
||||
---
|
||||
|
||||
## What is not yet specced
|
||||
|
||||
The following spec files still need to be written before their phases begin:
|
||||
- /docs/generation-spec.md — micro learning generation service
|
||||
- /docs/curriculum-spec.md — curriculum generator + versioning
|
||||
- /docs/r42-spec.md — chat service
|
||||
- /docs/gamification-spec.md — progress service + gamification mechanics
|
||||
- /docs/frontend-spec.md — employee app, admin app, PWA config
|
||||
|
||||
Do not begin a phase without its spec file. Flag the gap if you reach it.
|
||||
|
||||
---
|
||||
|
||||
## Source of truth hierarchy
|
||||
|
||||
When sources conflict, resolve in this order:
|
||||
1. This handover document (rationale and decisions)
|
||||
2. The relevant spec file (implementation detail)
|
||||
3. data-model.md (schema is authoritative)
|
||||
4. architecture.md (system structure)
|
||||
|
||||
Do not use legacy/ code as a source of truth for anything.
|
||||
394
docs/implementation-plan.md
Normal file
394
docs/implementation-plan.md
Normal file
@@ -0,0 +1,394 @@
|
||||
# Implementation plan
|
||||
|
||||
## How to use this document
|
||||
|
||||
Work through phases in order. Do not start phase N+1 before phase N passes
|
||||
all acceptance criteria. Each phase lists the spec file to read, the steps
|
||||
to execute, and the criteria that define done.
|
||||
|
||||
At the start of each session: state the phase and step.
|
||||
At the end of each session: state completed steps and next starting point.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Infrastructure + ingestion service
|
||||
|
||||
**Spec to read:** /docs/ingestion-spec.md, /docs/data-model.md
|
||||
|
||||
### Steps
|
||||
|
||||
**1.1 — Repo scaffold**
|
||||
```
|
||||
app/
|
||||
frontend/ (empty, Next.js init comes in phase 4)
|
||||
services/
|
||||
ingestion/
|
||||
generation/ (empty placeholder)
|
||||
curriculum/ (empty placeholder)
|
||||
chat/ (empty placeholder)
|
||||
progress/ (empty placeholder)
|
||||
```
|
||||
|
||||
Create `app/services/ingestion/` with:
|
||||
- package.json (dependencies from ingestion-spec.md)
|
||||
- tsconfig.json (strict mode)
|
||||
- .env.example (all env vars from ingestion-spec.md)
|
||||
- .gitignore
|
||||
|
||||
**1.2 — PocketBase collections**
|
||||
PocketBase runs as a binary. Create a migration script at
|
||||
`app/services/ingestion/migrations/001_initial_schema.ts` that uses the
|
||||
PocketBase JS SDK to create all collections defined in data-model.md:
|
||||
|
||||
Collections to create:
|
||||
- source_documents
|
||||
- themes
|
||||
- topics
|
||||
- micro_learnings (schema only — no data yet)
|
||||
- curriculum_versions (schema only)
|
||||
- curriculum_weeks (schema only)
|
||||
- employee_curriculum_state (schema only)
|
||||
- session_completions (schema only)
|
||||
- gamification_profiles (schema only)
|
||||
- badges (schema only)
|
||||
- employee_badges (schema only)
|
||||
- milestone_cards (schema only)
|
||||
|
||||
Seed the badges collection with all badge definitions from data-model.md.
|
||||
|
||||
**1.3 — Qdrant collections**
|
||||
Create `app/services/ingestion/migrations/002_qdrant_setup.ts` that
|
||||
initialises both Qdrant collections:
|
||||
- source_chunks (1536 dimensions, cosine distance)
|
||||
- topic_summaries (1536 dimensions, cosine distance)
|
||||
|
||||
**1.4 — Ingestion service scaffold**
|
||||
Build the Fastify server with two routes:
|
||||
- POST /ingest
|
||||
- GET /status/:jobId
|
||||
|
||||
Use the file structure from ingestion-spec.md exactly.
|
||||
|
||||
**1.5 — Stage 1: text extraction**
|
||||
Implement extract.ts per ingestion-spec.md:
|
||||
- TXT: direct UTF-8 read
|
||||
- MD: direct UTF-8 read, preserve heading markers
|
||||
- PDF: pdf-parse, page break markers
|
||||
|
||||
**1.6 — Stage 2–3: chunking + cleaning**
|
||||
Implement chunk.ts and clean.ts per ingestion-spec.md:
|
||||
- MD: heading-based splitting
|
||||
- TXT: sliding window (800 chars, 150 overlap)
|
||||
- PDF: page + paragraph splitting
|
||||
- Cleaning: whitespace, artefacts, minimum length filter
|
||||
|
||||
**1.7 — Stage 4: structure extraction**
|
||||
Implement structure.ts per ingestion-spec.md:
|
||||
- Claude Sonnet 4 call with system + user prompt from spec
|
||||
- Zod validation of DraftKB output
|
||||
- Batch handling for documents > 60 chunks
|
||||
- Retry logic on parse failure
|
||||
- Error handling: failed job status + reason
|
||||
|
||||
**1.8 — Stage 5: PocketBase write**
|
||||
Implement the PocketBase write logic:
|
||||
- Create Theme records (status: draft)
|
||||
- Create Topic records under each Theme (status: draft)
|
||||
- Resolve relationships between Topics after all records created
|
||||
|
||||
**1.9 — Stage 6: embeddings + Qdrant write**
|
||||
Implement embed.ts:
|
||||
- OpenAI text-embedding-3-small, batches of 100
|
||||
- Write to Qdrant source_chunks collection
|
||||
- Write to Qdrant topic_summaries collection
|
||||
- Update Topic.qdrant_chunk_ids in PocketBase
|
||||
|
||||
**1.10 — Job status tracking**
|
||||
Wire all stages into the job queue (jobs/queue.ts):
|
||||
- Status transitions: queued → extracting → chunking → structuring →
|
||||
writing → embedding → done / failed
|
||||
- Progress counters (chunksTotal, chunksEmbedded, themesFound, topicsFound)
|
||||
- GET /status/:jobId returns current state
|
||||
|
||||
### Acceptance criteria
|
||||
|
||||
- [ ] POST /ingest with a small MD file completes without error
|
||||
- [ ] GET /status/:jobId returns `done` after processing
|
||||
- [ ] PocketBase contains draft Theme + Topic records with correct hierarchy
|
||||
- [ ] Topic.body contains AI-drafted content (not empty)
|
||||
- [ ] Topic relationships are resolved (related_topics populated where applicable)
|
||||
- [ ] Qdrant source_chunks contains vectors with correct payload fields
|
||||
- [ ] Qdrant topic_summaries contains vectors for each published topic
|
||||
- [ ] Topic.qdrant_chunk_ids is populated
|
||||
- [ ] POST /ingest with a PDF file completes without error
|
||||
- [ ] POST /ingest with a TXT file completes without error
|
||||
- [ ] A document > 60 chunks triggers batch processing without error
|
||||
- [ ] A malformed PDF returns status `failed` with reason, not an uncaught exception
|
||||
- [ ] All Zod validations pass — no `any` types in codebase
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Generation service
|
||||
|
||||
**Spec to read:** /docs/generation-spec.md (write this spec before starting)
|
||||
|
||||
### Steps
|
||||
|
||||
**2.1 — Generation service scaffold**
|
||||
Fastify service at app/services/generation/
|
||||
Routes: POST /generate, GET /status/:jobId
|
||||
|
||||
**2.2 — Generate all 10 types per topic**
|
||||
One Claude Sonnet 4 call per type per topic.
|
||||
Structured JSON output validated against Zod schemas from data-model.md.
|
||||
Write to micro_learnings collection (status: generated).
|
||||
|
||||
**2.3 — Batch generation on theme approval**
|
||||
When admin approves a Theme batch, queue generation for all Topics in that Theme.
|
||||
All 10 types per Topic.
|
||||
|
||||
**2.4 — Admin publish flow**
|
||||
Route to update micro_learning status from generated → published or rejected.
|
||||
This is called by the admin app (built in phase 4).
|
||||
|
||||
### Acceptance criteria (to be detailed in generation-spec.md)
|
||||
|
||||
- [ ] All 10 micro learning types generated for a test topic
|
||||
- [ ] All 10 JSON outputs validate against their Zod schemas
|
||||
- [ ] Generated content written to PocketBase with status: generated
|
||||
- [ ] Admin can publish or reject individual micro learnings
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Curriculum service
|
||||
|
||||
**Spec to read:** /docs/curriculum-spec.md (write this spec before starting)
|
||||
|
||||
### Steps
|
||||
|
||||
**3.1 — Curriculum service scaffold**
|
||||
Fastify service at app/services/curriculum/
|
||||
|
||||
**3.2 — Curriculum generator**
|
||||
Claude Sonnet 4 reads full KB graph → produces 26-week schedule.
|
||||
Written to curriculum_versions + curriculum_weeks.
|
||||
|
||||
**3.3 — Versioning logic**
|
||||
- New version created on regeneration
|
||||
- Completed weeks frozen (employee_curriculum_state.current_week used as boundary)
|
||||
- Admin confirmation required before applying new version
|
||||
|
||||
**3.4 — Perpetual cycling**
|
||||
On week 26 completion, cycle increments, new cycle starts on latest version.
|
||||
Second cycle: varied sequence, surfaces unused micro learning types.
|
||||
|
||||
### Acceptance criteria (to be detailed in curriculum-spec.md)
|
||||
|
||||
- [ ] Curriculum generated from a populated KB
|
||||
- [ ] 26 weeks produced, all Themes covered
|
||||
- [ ] Prerequisites respected in ordering
|
||||
- [ ] Regeneration does not alter completed weeks
|
||||
- [ ] Admin confirmation flow works correctly
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — Frontend: admin app
|
||||
|
||||
**Spec to read:** /docs/frontend-spec.md (write this spec before starting)
|
||||
|
||||
### Steps
|
||||
|
||||
**4.1 — Next.js 14 scaffold**
|
||||
Mobile-first, TypeScript strict, Tailwind CSS, PWA config.
|
||||
Role-based routing: /admin/* and /app/* from single Next.js codebase.
|
||||
|
||||
**4.2 — Auth**
|
||||
PocketBase auth integration. Admin role routes to /admin/*.
|
||||
|
||||
**4.3 — Document upload + ingestion status**
|
||||
Upload UI → calls ingestion service → polls job status → shows progress.
|
||||
|
||||
**4.4 — Theme batch review**
|
||||
Display draft Themes with their Topic list.
|
||||
Approve batch / edit individual topics / reject batch.
|
||||
Triggers generation service on approval.
|
||||
|
||||
**4.5 — Curriculum editor**
|
||||
Display AI-generated curriculum (26 weeks).
|
||||
Drag-to-reorder weeks. Edit Theme assignment per week.
|
||||
Confirm regeneration with preview.
|
||||
|
||||
### Acceptance criteria (to be detailed in frontend-spec.md)
|
||||
|
||||
- [ ] Admin can upload a document and see ingestion progress
|
||||
- [ ] Admin can approve a Theme batch
|
||||
- [ ] Admin can edit a Topic before approval
|
||||
- [ ] Admin can view and reorder the curriculum
|
||||
- [ ] Admin can confirm a curriculum regeneration with preview
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — Frontend: employee app
|
||||
|
||||
**Spec to read:** /docs/frontend-spec.md (same file, employee section)
|
||||
|
||||
### Steps
|
||||
|
||||
**5.1 — Employee auth + onboarding**
|
||||
PocketBase auth. Employee role routes to /app/*.
|
||||
Set start date on first login → creates employee_curriculum_state record.
|
||||
|
||||
**5.2 — Weekly session flow**
|
||||
Current week's Theme displayed.
|
||||
Topics listed with available micro learning types per topic.
|
||||
Employee selects type → content rendered → mark complete.
|
||||
|
||||
**5.3 — Knowledge library**
|
||||
Browse all published Topics.
|
||||
Filter by Theme, difficulty, key terms.
|
||||
|
||||
**5.4 — R42 chatbot**
|
||||
Floating button, every screen.
|
||||
Calls chat service → streams response.
|
||||
Cites source topic in response.
|
||||
|
||||
**5.5 — Gamification profile**
|
||||
GitHub-style heatmap (26-week view).
|
||||
Badge display.
|
||||
Streak + level + commit count.
|
||||
Public leaderboard (multi-dimension).
|
||||
Milestone cards in activity feed.
|
||||
|
||||
### Acceptance criteria (to be detailed in frontend-spec.md)
|
||||
|
||||
- [ ] Employee sees correct week based on start date
|
||||
- [ ] Employee can complete a topic with a chosen micro learning type
|
||||
- [ ] Completion is recorded and XP awarded
|
||||
- [ ] Knowledge library shows all published topics with filters
|
||||
- [ ] R42 responds with grounded answer and source citation
|
||||
- [ ] R42 is accessible from every screen
|
||||
- [ ] Heatmap renders correctly on mobile (375px)
|
||||
- [ ] Leaderboard shows all employees with multi-dimension data
|
||||
|
||||
---
|
||||
|
||||
## Phase 6 — Chat service (R42)
|
||||
|
||||
**Spec to read:** /docs/r42-spec.md (write this spec before starting)
|
||||
|
||||
### Steps
|
||||
|
||||
**6.1 — Chat service scaffold**
|
||||
Fastify service at app/services/chat/
|
||||
|
||||
**6.2 — Query → embed → retrieve**
|
||||
Employee query embedded → Qdrant nearest-neighbour on both collections.
|
||||
Boost chunks from employee's current Theme.
|
||||
|
||||
**6.3 — Response generation**
|
||||
Top-K chunks injected into Haiku 4.5 prompt.
|
||||
Response streamed to frontend.
|
||||
Source Topic titles included in response.
|
||||
|
||||
**6.4 — Out-of-scope handling**
|
||||
If retrieval confidence is below threshold, R42 responds:
|
||||
"I can only answer questions based on the internal knowledge base.
|
||||
This topic doesn't appear to be covered."
|
||||
|
||||
### Acceptance criteria (to be detailed in r42-spec.md)
|
||||
|
||||
- [ ] R42 answers a question about a published topic correctly
|
||||
- [ ] R42 cites the source topic in its response
|
||||
- [ ] R42 refuses to answer out-of-scope questions explicitly
|
||||
- [ ] Response streams to frontend (not batch)
|
||||
- [ ] Response latency < 3 seconds for typical queries
|
||||
|
||||
---
|
||||
|
||||
## Phase 7 — Progress service
|
||||
|
||||
**Spec to read:** /docs/gamification-spec.md (write this spec before starting)
|
||||
|
||||
### Steps
|
||||
|
||||
**7.1 — Progress service scaffold**
|
||||
Fastify service at app/services/progress/
|
||||
|
||||
**7.2 — Completion recording**
|
||||
Write session_completions record on topic completion.
|
||||
Calculate XP (commits) per type.
|
||||
|
||||
**7.3 — Gamification updates**
|
||||
Update gamification_profiles: commits, level, streak, types_used.
|
||||
Evaluate badge conditions → write employee_badges on award.
|
||||
|
||||
**7.4 — Milestone cards**
|
||||
Generate milestone_cards record at weeks 13 and 26.
|
||||
|
||||
**7.5 — Leaderboard query**
|
||||
Endpoint returning all gamification_profiles for leaderboard rendering.
|
||||
|
||||
### Acceptance criteria (to be detailed in gamification-spec.md)
|
||||
|
||||
- [ ] Completion writes to session_completions
|
||||
- [ ] Commits calculated and added to gamification_profile
|
||||
- [ ] Level updates correctly at commit thresholds
|
||||
- [ ] Streak increments on weekly completion, resets on skip
|
||||
- [ ] Badge awarded when condition is met
|
||||
- [ ] Milestone card created at weeks 13 and 26
|
||||
- [ ] Leaderboard endpoint returns all employees with correct data
|
||||
|
||||
---
|
||||
|
||||
## Phase 8 — Integration + hardening
|
||||
|
||||
No new spec required.
|
||||
|
||||
### Steps
|
||||
|
||||
**8.1 — Service wiring**
|
||||
Verify all services communicate through PocketBase correctly.
|
||||
No direct service-to-service calls — all state through PocketBase.
|
||||
|
||||
**8.2 — Error handling audit**
|
||||
Review all services for unhandled promise rejections, missing error states,
|
||||
and uncaught exceptions. Every external call (AI API, PocketBase, Qdrant,
|
||||
OpenAI) wrapped in try/catch with meaningful error logging.
|
||||
|
||||
**8.3 — Mobile QA**
|
||||
Test all employee app flows at 375px width.
|
||||
R42 floating button must not obscure content.
|
||||
Heatmap must render without horizontal scroll.
|
||||
|
||||
**8.4 — Environment variable audit**
|
||||
Verify no hardcoded values. All .env.example files complete.
|
||||
|
||||
**8.5 — Dockerfile update**
|
||||
Update COPY path from legacy app root to /app.
|
||||
This is the one manual change that connects the rebuild to the existing pipeline.
|
||||
|
||||
### Acceptance criteria
|
||||
|
||||
- [ ] Full flow works end-to-end: upload doc → approve → curriculum → employee completes session → R42 answers question → gamification updates
|
||||
- [ ] No uncaught exceptions in any service under normal operating conditions
|
||||
- [ ] All screens render correctly on 375px mobile
|
||||
- [ ] Dockerfile builds successfully pointing at /app
|
||||
- [ ] Existing pipeline deploys the rebuilt app without modification
|
||||
|
||||
---
|
||||
|
||||
## Spec files still to be written
|
||||
|
||||
Before starting each phase, write the corresponding spec file.
|
||||
Use ingestion-spec.md as the template for structure and detail level.
|
||||
|
||||
| Phase | Spec file needed |
|
||||
|---|---|
|
||||
| 2 | /docs/generation-spec.md |
|
||||
| 3 | /docs/curriculum-spec.md |
|
||||
| 4–5 | /docs/frontend-spec.md |
|
||||
| 6 | /docs/r42-spec.md |
|
||||
| 7 | /docs/gamification-spec.md |
|
||||
|
||||
When you reach a phase without a spec: stop, draft the spec, then proceed.
|
||||
Do not implement without a spec.
|
||||
483
docs/ingestion-spec.md
Normal file
483
docs/ingestion-spec.md
Normal file
@@ -0,0 +1,483 @@
|
||||
# Ingestion service spec
|
||||
|
||||
## Responsibility
|
||||
|
||||
Accepts uploaded source documents (PDF, MD, TXT), extracts clean text, chunks it,
|
||||
generates embeddings, and produces a structured draft KB (Themes + Topics +
|
||||
relationships) ready for admin review.
|
||||
|
||||
This service runs entirely server-side. The admin app calls it via REST. All AI
|
||||
calls go through the Anthropic API. No ingestion logic lives in the frontend.
|
||||
|
||||
---
|
||||
|
||||
## Service location
|
||||
|
||||
```
|
||||
app/services/ingestion/
|
||||
├── index.ts entry point, Fastify server
|
||||
├── routes/
|
||||
│ └── documents.ts POST /ingest, GET /status/:jobId
|
||||
├── pipeline/
|
||||
│ ├── extract.ts format detection + text extraction
|
||||
│ ├── chunk.ts chunking strategies per format
|
||||
│ ├── clean.ts chunk cleaning
|
||||
│ ├── structure.ts Claude call → Theme/Topic extraction
|
||||
│ └── embed.ts embedding generation + Qdrant write
|
||||
├── jobs/
|
||||
│ └── queue.ts async job queue (in-memory, BullMQ later if needed)
|
||||
├── lib/
|
||||
│ ├── pocketbase.ts PocketBase client
|
||||
│ ├── qdrant.ts Qdrant client
|
||||
│ ├── anthropic.ts Anthropic client
|
||||
│ └── openai.ts OpenAI embeddings client
|
||||
└── types.ts shared TypeScript types
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API surface
|
||||
|
||||
### POST /ingest
|
||||
|
||||
Triggered by admin app on document upload.
|
||||
|
||||
Request:
|
||||
```json
|
||||
{
|
||||
"documentId": "string",
|
||||
"filename": "string",
|
||||
"format": "pdf" | "md" | "txt",
|
||||
"filePath": "string"
|
||||
}
|
||||
```
|
||||
|
||||
Response (202 Accepted):
|
||||
```json
|
||||
{
|
||||
"jobId": "string",
|
||||
"status": "queued"
|
||||
}
|
||||
```
|
||||
|
||||
Processing is async. The admin app polls job status.
|
||||
|
||||
---
|
||||
|
||||
### GET /status/:jobId
|
||||
|
||||
Returns current job progress.
|
||||
|
||||
Response:
|
||||
```json
|
||||
{
|
||||
"jobId": "string",
|
||||
"status": "queued" | "extracting" | "chunking" | "structuring" | "embedding" | "done" | "failed",
|
||||
"progress": {
|
||||
"chunksTotal": 42,
|
||||
"chunksEmbedded": 18,
|
||||
"themesFound": 3,
|
||||
"topicsFound": 14
|
||||
},
|
||||
"error": "string | null"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Pipeline stages
|
||||
|
||||
### Stage 1 — Text extraction
|
||||
|
||||
Input: file path + format
|
||||
Output: raw text string
|
||||
|
||||
```
|
||||
format === 'txt'
|
||||
→ read file directly as UTF-8
|
||||
|
||||
format === 'md'
|
||||
→ read file directly as UTF-8
|
||||
→ preserve heading markers (# ## ###) — used in chunking
|
||||
|
||||
format === 'pdf'
|
||||
→ pdfplumber: extract text page by page
|
||||
→ concatenate with page break markers: \n\n---PAGE---\n\n
|
||||
→ strip known PDF artefacts: headers/footers repeating on every page,
|
||||
page numbers, watermarks
|
||||
```
|
||||
|
||||
Failure handling:
|
||||
- PDF extraction returns empty string → mark job `failed`, reason: `pdf_extraction_empty`
|
||||
- File not found → mark job `failed`, reason: `file_not_found`
|
||||
|
||||
---
|
||||
|
||||
### Stage 2 — Chunking
|
||||
|
||||
Input: raw text + format
|
||||
Output: Chunk[]
|
||||
|
||||
Chunking strategy differs per format.
|
||||
|
||||
**MD chunking — heading-based (preferred)**
|
||||
```
|
||||
Split on heading markers: #, ##, ###
|
||||
Each heading + its following content = one chunk
|
||||
Minimum chunk size: 100 characters
|
||||
→ if heading section is < 100 chars, merge with next sibling
|
||||
Maximum chunk size: 1500 characters
|
||||
→ if section exceeds limit, split on paragraph breaks within section
|
||||
Metadata preserved per chunk:
|
||||
heading_level: 1 | 2 | 3
|
||||
heading_text: string
|
||||
parent_heading: string | null
|
||||
```
|
||||
|
||||
MD chunking produces the highest quality structural signal for Theme/Topic extraction.
|
||||
Admins should be advised to provide source material as MD where possible.
|
||||
|
||||
**TXT chunking — sliding window**
|
||||
```
|
||||
Window size: 800 characters
|
||||
Overlap: 150 characters
|
||||
Split on: paragraph breaks (\n\n) first, then sentence boundaries, then hard cut
|
||||
Metadata per chunk:
|
||||
chunk_index: number
|
||||
approximate_position: 'start' | 'middle' | 'end'
|
||||
```
|
||||
|
||||
**PDF chunking — page + paragraph**
|
||||
```
|
||||
Split on ---PAGE--- markers from extraction stage
|
||||
Within each page: split on paragraph breaks (\n\n)
|
||||
Minimum chunk size: 100 characters
|
||||
→ merge sub-threshold paragraphs with adjacent chunk
|
||||
Maximum chunk size: 1200 characters
|
||||
→ hard split at sentence boundary
|
||||
Metadata per chunk:
|
||||
page_number: number
|
||||
chunk_index_on_page: number
|
||||
```
|
||||
|
||||
**Chunk type:**
|
||||
```typescript
|
||||
type Chunk = {
|
||||
id: string // UUID generated at chunking
|
||||
documentId: string
|
||||
text: string
|
||||
format: 'pdf' | 'md' | 'txt'
|
||||
index: number // global position in document
|
||||
metadata: {
|
||||
// MD-specific
|
||||
headingLevel?: number
|
||||
headingText?: string
|
||||
parentHeading?: string
|
||||
// TXT-specific
|
||||
approximatePosition?: 'start' | 'middle' | 'end'
|
||||
// PDF-specific
|
||||
pageNumber?: number
|
||||
chunkIndexOnPage?: number
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Stage 3 — Chunk cleaning
|
||||
|
||||
Input: Chunk[]
|
||||
Output: Chunk[] (cleaned)
|
||||
|
||||
Applied to all formats:
|
||||
```
|
||||
- trim leading/trailing whitespace
|
||||
- collapse 3+ consecutive newlines to 2
|
||||
- remove null bytes and non-printable characters
|
||||
- remove chunks where text.length < 80 after cleaning
|
||||
→ these are likely artefacts (page numbers, standalone headers)
|
||||
- normalise unicode: NFC normalisation
|
||||
- do not strip punctuation or alter sentence structure
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Stage 4 — Structure extraction (AI)
|
||||
|
||||
Input: Chunk[]
|
||||
Output: DraftKB
|
||||
|
||||
This is the core AI call. Claude Sonnet 4 reads all chunks and returns a structured
|
||||
KB draft as JSON.
|
||||
|
||||
**Prompt strategy:**
|
||||
|
||||
System prompt:
|
||||
```
|
||||
You are a knowledge architect. Your task is to analyse a set of text chunks from
|
||||
a source document and extract a structured knowledge base.
|
||||
|
||||
Output ONLY valid JSON matching the schema provided. No preamble, no explanation,
|
||||
no markdown fences.
|
||||
|
||||
Rules:
|
||||
- Group related content into Themes. A Theme is a broad subject area.
|
||||
- Under each Theme, identify discrete Topics. A Topic covers one specific concept.
|
||||
- Identify relationships between Topics: related, prerequisite, or contrast.
|
||||
- related: Topics that complement each other
|
||||
- prerequisite: Topic A must be understood before Topic B
|
||||
- contrast: Topics that represent opposing approaches or concepts
|
||||
- For each Topic, extract key terms suitable for a glossary.
|
||||
- Assign a complexity weight (1–5) to each Topic.
|
||||
1 = introductory, 5 = advanced
|
||||
- Draft a body for each Topic (2–4 paragraphs) based on the source chunks.
|
||||
- Draft a description for each Theme (1–2 sentences).
|
||||
- Every Topic must reference the chunk IDs that contributed to it.
|
||||
```
|
||||
|
||||
User prompt:
|
||||
```
|
||||
Source document: {filename}
|
||||
Format: {format}
|
||||
|
||||
Chunks:
|
||||
{chunks mapped as: [CHUNK-{id}]\n{text}\n}
|
||||
|
||||
Extract the knowledge base structure from these chunks.
|
||||
```
|
||||
|
||||
**Output schema:**
|
||||
```typescript
|
||||
type DraftKB = {
|
||||
themes: DraftTheme[]
|
||||
}
|
||||
|
||||
type DraftTheme = {
|
||||
title: string
|
||||
description: string
|
||||
topics: DraftTopic[]
|
||||
}
|
||||
|
||||
type DraftTopic = {
|
||||
title: string
|
||||
body: string
|
||||
difficulty: 'introductory' | 'intermediate' | 'advanced'
|
||||
complexityWeight: number // 1–5
|
||||
keyTerms: string[]
|
||||
sourceChunkIds: string[] // references Chunk.id values
|
||||
relationships: {
|
||||
related: string[] // topic titles (resolved to IDs after write)
|
||||
prerequisites: string[]
|
||||
contrasts: string[]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**AI call configuration:**
|
||||
```typescript
|
||||
{
|
||||
model: 'claude-sonnet-4-20250514',
|
||||
max_tokens: 8000,
|
||||
temperature: 0 // deterministic output for structured extraction
|
||||
}
|
||||
```
|
||||
|
||||
**Chunking strategy for large documents:**
|
||||
If total chunk count exceeds 60 chunks, split into batches of 40 with 5-chunk
|
||||
overlap. Run one Claude call per batch. Merge resulting DraftKB objects:
|
||||
- Themes with identical titles → merge Topics
|
||||
- Duplicate Topic titles within a Theme → keep longer body, merge sourceChunkIds
|
||||
- Relationships are resolved after full merge
|
||||
|
||||
**Error handling:**
|
||||
- JSON parse failure → retry once with stricter prompt ("ensure valid JSON only")
|
||||
- Second failure → mark job `failed`, reason: `structure_extraction_failed`, log raw response
|
||||
- Empty themes array → mark job `failed`, reason: `no_structure_found`
|
||||
|
||||
---
|
||||
|
||||
### Stage 5 — Write to PocketBase
|
||||
|
||||
Input: DraftKB
|
||||
Output: written Theme + Topic records with status `draft`
|
||||
|
||||
```
|
||||
For each DraftTheme:
|
||||
create themes record {
|
||||
title, description,
|
||||
status: 'draft',
|
||||
source_documents: [documentId]
|
||||
}
|
||||
|
||||
For each DraftTopic under the theme:
|
||||
create topics record {
|
||||
theme: themeId,
|
||||
title, body, difficulty, complexity_weight, key_terms,
|
||||
status: 'draft',
|
||||
qdrant_chunk_ids: [] // populated in stage 6
|
||||
}
|
||||
|
||||
After all topics created:
|
||||
resolve relationship titles → topic IDs
|
||||
update topics.related_topics, prerequisite_topics, contrast_topics
|
||||
|
||||
If a relationship title cannot be resolved to an existing topic:
|
||||
skip silently (cross-document relationships resolved in a later pass)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Stage 6 — Embedding generation + Qdrant write
|
||||
|
||||
Input: Chunk[], written Topic records
|
||||
Output: vectors in Qdrant, qdrant_chunk_ids updated on Topic records
|
||||
|
||||
**Source chunk embeddings:**
|
||||
```
|
||||
For each Chunk (post-cleaning):
|
||||
embed Chunk.text → text-embedding-3-small (1536 dimensions)
|
||||
write to Qdrant collection: source_chunks {
|
||||
id: Chunk.id,
|
||||
vector: float[],
|
||||
payload: {
|
||||
source_document_id: documentId,
|
||||
chunk_index: Chunk.index,
|
||||
text: Chunk.text,
|
||||
theme_id: resolved themeId | null,
|
||||
topic_id: resolved topicId | null,
|
||||
format: Chunk.format
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Topic summary embeddings:**
|
||||
```
|
||||
For each published Topic:
|
||||
embed Topic.body → text-embedding-3-small
|
||||
write to Qdrant collection: topic_summaries {
|
||||
id: UUID,
|
||||
vector: float[],
|
||||
payload: {
|
||||
topic_id: Topic.id,
|
||||
theme_id: Topic.theme,
|
||||
title: Topic.title,
|
||||
text: Topic.body
|
||||
}
|
||||
}
|
||||
|
||||
Update Topic.qdrant_chunk_ids with all Chunk.ids that reference this topic
|
||||
```
|
||||
|
||||
**Batching:**
|
||||
OpenAI embeddings API: batch in groups of 100 texts per request to stay within
|
||||
rate limits and reduce latency.
|
||||
|
||||
---
|
||||
|
||||
## Job lifecycle
|
||||
|
||||
```
|
||||
POST /ingest received
|
||||
↓
|
||||
Job created → status: queued
|
||||
↓
|
||||
Stage 1: extracting
|
||||
↓
|
||||
Stage 2–3: chunking
|
||||
↓
|
||||
Stage 4: structuring
|
||||
↓
|
||||
Stage 5: writing to PocketBase
|
||||
↓
|
||||
Stage 6: embedding
|
||||
↓
|
||||
status: done
|
||||
↓
|
||||
Admin notification: "Document processed. N themes, N topics ready for review."
|
||||
↓
|
||||
Curriculum regeneration queued (status: pending_admin_confirm)
|
||||
```
|
||||
|
||||
On any stage failure:
|
||||
```
|
||||
status: failed
|
||||
error: { stage, reason, detail }
|
||||
Source document status → 'failed' in PocketBase
|
||||
Admin notification: "Ingestion failed: {reason}"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Environment variables required
|
||||
|
||||
```
|
||||
ANTHROPIC_API_KEY=
|
||||
OPENAI_API_KEY=
|
||||
POCKETBASE_URL=
|
||||
POCKETBASE_ADMIN_EMAIL=
|
||||
POCKETBASE_ADMIN_PASSWORD=
|
||||
QDRANT_URL=
|
||||
QDRANT_API_KEY= # empty string if running locally without auth
|
||||
INGESTION_PORT=3001
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
```json
|
||||
{
|
||||
"dependencies": {
|
||||
"fastify": "^4",
|
||||
"@anthropic-ai/sdk": "^0.24",
|
||||
"openai": "^4",
|
||||
"@qdrant/js-client-rest": "^1.9",
|
||||
"pocketbase": "^0.21",
|
||||
"pdfplumber": "NOT JS — see note below",
|
||||
"pdf-parse": "^1.1",
|
||||
"uuid": "^9",
|
||||
"zod": "^3"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**PDF extraction note:**
|
||||
`pdfplumber` is a Python library. Two options:
|
||||
1. Use `pdf-parse` (Node.js) — simpler, covers 90% of cases
|
||||
2. Run `pdfplumber` as a Python sidecar process via child_process — higher quality
|
||||
for complex PDFs with tables and columns
|
||||
|
||||
Default to `pdf-parse` initially. Add pdfplumber sidecar only if extraction
|
||||
quality is insufficient for actual source documents.
|
||||
|
||||
---
|
||||
|
||||
## TypeScript strict mode requirements
|
||||
|
||||
- No `any` types
|
||||
- All Claude response parsing through Zod schema validation
|
||||
- All PocketBase writes typed against collection schemas from `data-model.md`
|
||||
- Qdrant payloads typed explicitly — no untyped objects
|
||||
|
||||
---
|
||||
|
||||
## What this service does NOT do
|
||||
|
||||
- Does not generate micro learnings → generation service
|
||||
- Does not build or update the curriculum → curriculum service
|
||||
- Does not handle admin approval → admin app + PocketBase directly
|
||||
- Does not serve R42 queries → chat service
|
||||
- Does not handle auth → PocketBase + admin app
|
||||
|
||||
---
|
||||
|
||||
## Testing checkpoints
|
||||
|
||||
Before handing to Claude Code for implementation, verify manually:
|
||||
|
||||
1. Upload a short MD file (< 10 headings) → inspect chunk output → confirm heading structure preserved
|
||||
2. Upload a simple PDF (< 5 pages) → inspect chunk output → confirm no artefacts
|
||||
3. Run structure extraction on known chunks → validate JSON parses against Zod schema
|
||||
4. Confirm PocketBase draft records created with correct theme → topic hierarchy
|
||||
5. Confirm Qdrant source_chunks collection populated with correct payload fields
|
||||
6. Confirm topic.qdrant_chunk_ids updated after embedding stage
|
||||
Reference in New Issue
Block a user