Add comprehensive documentation for key organizational aspects

- Introduced "Pension Scheme & Benefits" detailing secondary employment benefits and pension specifics. - Created "Roles & Accountabilities" outlining the Holacracy role structure and responsibilities within Respellion. - Added "Security" section covering GDPR compliance and workplace safety protocols. - Established "Spending and Contracting" policy detailing expense categories and submission processes. - Documented "Who We Are" to define Respellion's identity, services, and operational model under Holacracy and ISO 9001.
2026-05-27 08:24:56 +02:00
parent 7066f881f9
commit 07af2783dc
26 changed files with 2631 additions and 4598 deletions
--- a/docs/data-model.md
+++ b/docs/data-model.md
@@ -1,397 +1,252 @@
-# Data model: employee learning platform
+# Data model: Respellion Learning Platform

 ## Overview

-Two storage systems:
- **PocketBase** — all structured relational data (SQLite under the hood)
- **Qdrant** — all vector embeddings for RAG retrieval
+All structured data lives in **PocketBase** (SQLite). There is **no vector store**
+— retrieval is computed at runtime with a local TF-IDF index over `topics`
+(`src/lib/retrieval.js`).
+
+Schema is defined by JS migrations in `pb_migrations/` (applied automatically by
+the PocketBase binary) and mirrored for local bootstrap in
+`scripts/setup-pb-collections.mjs`. The data-access layer is `src/lib/db.js`.
+
+All collections use PocketBase's auto `id`, plus `created` / `updated` autodate
+fields unless noted otherwise.

 ---

 ## PocketBase collections

-### `source_documents`
-Uploaded source files. Parent of all generated KB content.
-
-| Field | Type | Notes |
-|---|---|---|
-| id | string | PocketBase auto |
-| filename | string | original filename |
-| file | file | PocketBase file storage |
-| format | select | `pdf` `md` `txt` |
-| status | select | `processing` `processed` `failed` |
-| ingested_at | datetime | |
-| chunk_count | number | total chunks extracted |
-| created_by | relation → `users` | admin who uploaded |
-
---
-
-### `themes`
-Top-level content groupings. One Theme = one weekly session.
-
-| Field | Type | Notes |
-|---|---|---|
-| id | string | PocketBase auto |
-| title | string | |
-| description | text | AI drafted, admin editable |
-| status | select | `draft` `published` |
-| source_documents | relation[] → `source_documents` | which docs contributed |
-| approved_by | relation → `users` | admin who approved batch |
-| approved_at | datetime | |
-| created_at | datetime | |
-| updated_at | datetime | |
-
---
-
 ### `topics`
-Atomic knowledge units. Always belong to a Theme.
+Knowledge graph nodes. Created during ingestion, enriched for curriculum.

 | Field | Type | Notes |
 |---|---|---|
-| id | string | PocketBase auto |
-| theme | relation → `themes` | parent theme |
-| title | string | |
-| body | text (rich) | AI drafted, admin editable |
-| difficulty | select | `introductory` `intermediate` `advanced` |
-| complexity_weight | number | 1–5, used by curriculum generator |
-| status | select | `draft` `published` |
-| related_topics | relation[] → `topics` | lateral relationships |
-| prerequisite_topics | relation[] → `topics` | must-complete-first |
-| contrast_topics | relation[] → `topics` | deliberate opposites |
-| key_terms | json | string[] — feeds glossary |
-| qdrant_chunk_ids | json | string[] — references to embedded chunks |
-| created_at | datetime | |
-| updated_at | datetime | |
+| id | text | kebab-case slug (e.g. `holacratic-roles`) |
+| label | text | display name |
+| type | text | `concept` · `role` · `process` (ingestion); `fact` is excluded from learning |
+| description | text | 1–2 sentence summary |
+| learning_relevance | text | `core` · `standard` · `peripheral` · `exclude` |
+| relevance_locked | bool | if true, re-ingestion will not overwrite `learning_relevance` |
+| theme | text | subject grouping (used by curriculum generation) |
+| complexity_weight | number | 1–5 (curriculum ordering) |
+| difficulty | text | `introductory` · `intermediate` · `advanced` |

-Relationship types (related / prerequisite / contrast) are stored via the three explicit relation fields rather than a generic relationship table. This keeps queries simple at this scale.
+Topics with `type='fact'` or `learning_relevance='exclude'` are filtered out of
+learning, micro-learning, curriculum, and test selection.
+
+---
+
+### `relations`
+Knowledge graph edges between topics.
+
+| Field | Type | Notes |
+|---|---|---|
+| source | text | topic id |
+| target | text | topic id |
+| type | text | `related_to` · `depends_on` · `part_of` · `executed_by` |
+
+Edges are de-duplicated on the `(source, target, type)` tuple.
+
+---
+
+### `content`
+On-demand long-form learning content, one record per topic.
+
+| Field | Type | Notes |
+|---|---|---|
+| topic_id | text | topic this content belongs to |
+| data | json | merged object — only generated types are present |
+
+`data` shape (each key generated independently and shallow-merged):
+
+```json
+{
+  "article":     { "title", "intro", "sections": [{ "heading", "body" }], "keyTakeaways": [] },
+  "slides":      [ { "title", "bullets": [], "speakerNote" } ],
+  "infographic": { "headline", "tagline", "stats": [{ "value", "label", "icon" }],
+                   "steps": [{ "number", "title", "description", "icon" }], "quote", "colorTheme" }
+}
+```
+
+> There is no `podcast` key. The podcast type was removed.

 ---

 ### `micro_learnings`
-Generated content artifacts. One record per topic per type.
+Generated micro-learning artifacts. One record per topic per type.

 | Field | Type | Notes |
 |---|---|---|
-| id | string | PocketBase auto |
-| topic | relation → `topics` | |
-| type | select | see type enum below |
-| content | json | structured output — schema varies per type |
-| status | select | `queued` `generated` `published` `rejected` |
-| generation_model | string | model version used |
-| generated_at | datetime | |
-| published_at | datetime | |
-| updated_at | datetime | |
+| topic_id | relation → `topics` | cascade delete |
+| type | select | `concept_explainer` · `scenario_quiz` · `flashcard_set` |
+| content | json | structured output, schema varies per type |
+| status | select | `draft` · `published` (only `published` is visible to employees) |

-**Type enum:**
-`concept_explainer` `scenario_quiz` `misconceptions` `how_to` `comparison_card` `reflection_prompt` `flashcard_set` `case_study` `glossary_anchor` `myth_vs_evidence`
-
-**Content JSON schemas per type:**
+**Content JSON per type:**

 ```json
 // concept_explainer
-{
-  "paragraphs": ["string", "string"],
-  "example": "string"
-}
+{ "sections": [ { "title": "string", "content": "string (HTML: <p>, <ul>, <li>, <strong>)" } ] }   // ≥3 sections

 // scenario_quiz
-{
-  "scenario": "string",
-  "options": [
-    { "label": "A", "text": "string", "correct": true, "explanation": "string" }
-  ]
-}
-
-// misconceptions
-{
-  "items": [
-    { "misconception": "string", "correction": "string" }
-  ]
-}
-
-// how_to
-{
-  "steps": [
-    { "number": 1, "instruction": "string" }
-  ]
-}
-
-// comparison_card
-{
-  "subject_a": "string",
-  "subject_b": "string",
-  "dimensions": [
-    { "label": "string", "a": "string", "b": "string" }
-  ]
-}
-
-// reflection_prompt
-{
-  "prompt": "string",
-  "model_answer": "string"
-}
+{ "scenario": "string",
+  "options": [ { "text": "string", "isCorrect": true, "explanation": "string" } ] }                 // 3–4 options, exactly 1 correct

 // flashcard_set
-{
-  "cards": [
-    { "question": "string", "answer": "string" }
-  ]
-}
-
-// case_study
-{
-  "scenario": "string",
-  "questions": ["string"]
-}
-
-// glossary_anchor
-{
-  "term": "string",
-  "definition": "string",
-  "correct_use": "string",
-  "misuse": "string"
-}
-
-// myth_vs_evidence
-{
-  "myth": "string",
-  "evidence": "string",
-  "sources": ["string"]
-}
+{ "cards": [ { "front": "string", "back": "string" } ] }                                            // 5–10 cards
 ```

+> A former `reflection_prompt` type was dropped and is no longer generated.
+
+---
+
+### `micro_learning_completions`
+Append-only completion events. Never updated or deleted.
+
+| Field | Type | Notes |
+|---|---|---|
+| team_member_id | relation → `team_members` | the employee |
+| micro_learning_id | relation → `micro_learnings` | the artifact completed |
+| topic_id | relation → `topics` | denormalized topic |
+| type | text | type at time of completion |
+| session_week | number | the user's **absolute** curriculum week (week 1 = day they enrolled) |
+
+The 26-week slot and cycle are derived from `session_week`; there is no stored
+`cycle` field.
+
 ---

 ### `curriculum_versions`
-Versioned 26-week schedule. New version created on each regeneration.
+Versioned 26-week schedules. New version on each (re)generation.

 | Field | Type | Notes |
 |---|---|---|
-| id | string | PocketBase auto |
-| version | number | increments on each regeneration |
-| status | select | `draft` `active` `superseded` |
-| generated_at | datetime | |
-| approved_by | relation → `users` | admin who confirmed regeneration |
-| approved_at | datetime | |
-| generation_notes | text | why this version was created |
+| version_number | number | increments per generation |
+| status | text | `draft` · `active` · `superseded` (exactly one `active`) |
+| generation_reason | text | why this version was created |
+| confirmed_by | text | admin id who activated it |
+| confirmed_at | text | ISO datetime |
+| schedule | json | array of 26 week objects (below) |
+| coverage_stats | json | `{ themes_kb, themes_scheduled, topics_kb, topics_scheduled }` |

-Only one version has status `active` at any time.
+**`schedule[]` week object:**

---
-
-### `curriculum_weeks`
-Individual week slots. Child of a curriculum version.
-
-| Field | Type | Notes |
-|---|---|---|
-| id | string | PocketBase auto |
-| curriculum_version | relation → `curriculum_versions` | |
-| week_number | number | 1–26 |
-| theme | relation → `themes` | |
-| topics | relation[] → `topics` | ordered subset of theme topics |
-| topic_order | json | number[] — explicit ordering |
-| estimated_duration_minutes | number | AI estimate |
-| admin_notes | text | freeform admin annotation |
-
---
-
-### `employee_curriculum_state`
-Tracks each employee's position in the curriculum. One record per employee.
-
-| Field | Type | Notes |
-|---|---|---|
-| id | string | PocketBase auto |
-| user | relation → `users` | |
-| current_cycle | number | starts at 1, increments on loop |
-| current_week | number | 1–26 |
-| start_date | datetime | rolling start |
-| active_version | relation → `curriculum_versions` | version employee is on |
-| updated_at | datetime | |
-
-When curriculum regenerates: `active_version` updates for all employees whose `current_week` is less than the first regenerated week.
-
---
-
-### `session_completions`
-Immutable completion records. One record per employee per topic per type.
-
-| Field | Type | Notes |
-|---|---|---|
-| id | string | PocketBase auto |
-| user | relation → `users` | |
-| topic | relation → `topics` | |
-| micro_learning | relation → `micro_learnings` | specific type completed |
-| week_number | number | curriculum week at time of completion |
-| cycle | number | which cycle |
-| completed_at | datetime | |
-
-Records are never updated or deleted. This is the canonical history.
-
---
-
-### `gamification_profiles`
-One record per employee. Updated by progress service on each completion.
-
-| Field | Type | Notes |
-|---|---|---|
-| id | string | PocketBase auto |
-| user | relation → `users` | |
-| total_commits | number | cumulative XP |
-| current_level | select | `intern` `junior` `medior` `senior` `staff` `principal` |
-| current_streak_weeks | number | consecutive weeks with ≥1 completion |
-| longest_streak_weeks | number | all-time high |
-| types_used | json | string[] — which of 10 types used at least once |
-| last_active_week | number | used to detect streak breaks |
-| updated_at | datetime | |
-
---
-
-### `badges`
-Badge definitions. Seeded at startup, not user-generated.
-
-| Field | Type | Notes |
-|---|---|---|
-| id | string | PocketBase auto |
-| key | string | unique slug e.g. `governance_nerd` |
-| tier | select | `bronze` `silver` `gold` `legendary` `content` |
-| label | string | display name |
-| description | string | award condition description |
-| icon | string | emoji or icon key |
-
---
-
-### `employee_badges`
-Junction: which employees have earned which badges.
-
-| Field | Type | Notes |
-|---|---|---|
-| id | string | PocketBase auto |
-| user | relation → `users` | |
-| badge | relation → `badges` | |
-| earned_at | datetime | |
-| cycle | number | which cycle it was earned in |
-
---
-
-### `milestone_cards`
-Public milestone events at weeks 13 and 26.
-
-| Field | Type | Notes |
-|---|---|---|
-| id | string | PocketBase auto |
-| user | relation → `users` | |
-| cycle | number | |
-| week | number | 13 or 26 |
-| total_commits | number | snapshot at time of milestone |
-| streak_weeks | number | snapshot |
-| badge_keys | json | string[] — badges held at milestone |
-| created_at | datetime | public feed ordered by this |
-
---
-
-## PocketBase users collection (extended)
-
-Standard PocketBase `users` collection with additional fields:
-
-| Field | Type | Notes |
-|---|---|---|
-| role | select | `admin` `employee` |
-| display_name | string | used in gamification feed |
-| avatar | file | optional |
-
---
-
-## Qdrant collections
-
-### `source_chunks`
-Embeddings of raw source document chunks. Primary retrieval target for R42.
-
-| Field | Type | Notes |
-|---|---|---|
-| id | string | UUID |
-| vector | float[] | 1536 dimensions (text-embedding-3-small) |
-| source_document_id | string | reference to PocketBase |
-| chunk_index | number | position within document |
-| text | string | raw chunk text |
-| theme_id | string | assigned theme (post-extraction) |
-| topic_id | string | assigned topic (post-extraction, nullable) |
-| format | string | pdf / md / txt |
-
-### `topic_summaries`
-Embeddings of AI-generated topic body text. Secondary retrieval target.
-
-| Field | Type | Notes |
-|---|---|---|
-| id | string | UUID |
-| vector | float[] | 1536 dimensions |
-| topic_id | string | reference to PocketBase |
-| theme_id | string | |
-| title | string | for display in R42 citation |
-| text | string | full topic body |
-
---
-
-## Retrieval strategy for R42
-
-R42 queries both Qdrant collections and merges results:
-
-```
-Employee query
-  ↓
-Embed query → text-embedding-3-small
-  ↓
-Qdrant search: source_chunks (top 5) + topic_summaries (top 3)
-  ↓
-Filter: boost chunks from employee's current week theme
-  ↓
-Merge + deduplicate by topic_id
-  ↓
-Top-K context injected into Haiku 4.5 prompt
-  ↓
-Response includes: answer + cited topic title(s)
+```json
+{ "week_number": 1,              // 1..26
+  "theme": "string",
+  "topic_ids": ["topic-id"],     // 1+ topic ids
+  "estimated_duration": 30,      // 15..45 minutes
+  "week_rationale": "string" }
 ```

-Source chunks are weighted higher than topic summaries to keep R42 grounded in original source material rather than AI-generated abstractions.
-
 ---

-## Indexes and query patterns
+### `team_members`
+Registered users with PIN auth. This is the auth + employee record.

-Critical query patterns the data model must support efficiently:
-
-| Query | Collection | Index |
+| Field | Type | Notes |
 |---|---|---|
-| All published topics for a theme | topics | theme + status |
-| All micro learnings for a topic | micro_learnings | topic + status |
-| Employee's current week | employee_curriculum_state | user |
-| Weeks for a curriculum version | curriculum_weeks | curriculum_version + week_number |
-| Employee completion history | session_completions | user + cycle |
-| Public leaderboard | gamification_profiles | total_commits + streak |
-| Milestone feed | milestone_cards | created_at DESC |
-| Badges earned by employee | employee_badges | user |
+| name | text | display name |
+| pin | text | login PIN |
+| role | text | `admin` or empty/`employee` |
+| curriculum_started_at | date | timestamp the user enrolled (week 1 anchor); empty until enrolled |
+| enrollment_status | text | `not_started` · `active` |

-PocketBase creates indexes on relation fields by default. Composite indexes on `status` fields should be added manually where query frequency warrants it.
+A user is gated through the `/onboarding` screen until `enrollment_status='active'`
+(admins are exempt when heading to the admin panel).

 ---

-## Data flow summary
+### `sources`
+Uploaded source documents and their extraction status.
+
+| Field | Type | Notes |
+|---|---|---|
+| name | text | original filename |
+| status | text | `processing` · `completed` · `failed` · `cancelled` |
+| error | text | failure message, if any |
+| progress | json | `{ current, total, message }` during chunked extraction |
+
+---
+
+### `leaderboard`
+Points ledger, one row per user.
+
+| Field | Type | Notes |
+|---|---|---|
+| user_id | text | team member id |
+| name | text | display name |
+| points | number | cumulative (+2 per correct quiz answer) |
+| tests_completed | number | count of completed tests |
+| learnings_completed | number | reserved counter |
+
+Admins are filtered out of the public leaderboard at render time.
+
+---
+
+### `settings`
+App-wide key/value store.
+
+| Field | Type | Notes |
+|---|---|---|
+| key | text | setting key |
+| value | text | stringified value |
+
+---
+
+### `llm_calls`
+Best-effort telemetry for every Anthropic call (written by `callLLM`).
+
+| Field | Type | Notes |
+|---|---|---|
+| task | text | logging label (e.g. `learning.article`, `chat.r42`) |
+| model | text | resolved model string |
+| tier | text | `fast` · `standard` · `reasoning` |
+| duration_ms | number | wall-clock |
+| input_tokens / output_tokens | number | usage |
+| cache_read_tokens / cache_create_tokens | number | prompt-cache usage |
+| stop_reason | text | `end_turn` · `tool_use` · `max_tokens` |
+| ok | bool | success flag |
+| error_msg | text | error, if any |
+
+---
+
+## Dropped / legacy collections
+
+These existed in earlier iterations and have been removed. Their `db.js` helpers
+remain as deprecated no-op stubs — do not build on them:
+
+`quiz_banks`, `quiz_results`, `quiz_cache`, `learn_progress`, and the v1
+`curriculum` collection.
+
+---
+
+## Client-side storage (not PocketBase)
+
+`localStorage` is used only for admin/browser-local state:
+
+| Key | Purpose |
+|---|---|
+| `admin:model:{fast,standard,reasoning}` | per-tier model overrides (legacy `admin:model`) |
+| `admin:use_simulation` | stub LLM responses instead of calling Anthropic |
+| `kb:suggestions` | R42 graph-delta suggestion queue (managed via `kbStore`) |
+| `quiz:active:{userId}` | mid-quiz flag (hides R42) |
+| `chat:thread:{userId}` | R42 conversation, capped at 50 messages |
+
+`sessionStorage.respellion_session` holds the logged-in team member id.
+
+---
+
+## Retrieval (no vector DB)
+
+R42 context is built by `src/lib/retrieval.js`:

 ```
-source_documents
-  └── (ingestion service)
-        └── qdrant: source_chunks
-        └── themes (draft)
-              └── topics (draft)
-                    └── (approval)
-                          └── topics (published)
-                                └── qdrant: topic_summaries
-                                └── micro_learnings (queued → published)
-                                └── (curriculum service)
-                                      └── curriculum_versions
-                                            └── curriculum_weeks
-                                                  └── (employee progress)
-                                                        └── session_completions
-                                                        └── gamification_profiles
-                                                        └── employee_badges
-                                                        └── milestone_cards
+buildIndex(topics)         → TF-IDF index over (label + description), cached by array ref
+retrieveTopK(index, q, k)  → top-K topics, score = Σ (1 + log(tf)) · log((N+1)/(df+1))
 ```
+
+`src/components/chat/rag.js` combines top-K results with verbatim topic mentions,
+filters relations to the retrieved set, and injects limited deep content for
+explicitly named topics.