dda20612e9
- Created handover document outlining design decisions and application functionality. - Developed implementation plan detailing phased approach for service development. - Specified ingestion service responsibilities, API surface, and processing pipeline.
398 lines
11 KiB
Markdown
398 lines
11 KiB
Markdown
# Data model: employee learning platform
|
||
|
||
## Overview
|
||
|
||
Two storage systems:
|
||
- **PocketBase** — all structured relational data (SQLite under the hood)
|
||
- **Qdrant** — all vector embeddings for RAG retrieval
|
||
|
||
---
|
||
|
||
## PocketBase collections
|
||
|
||
### `source_documents`
|
||
Uploaded source files. Parent of all generated KB content.
|
||
|
||
| Field | Type | Notes |
|
||
|---|---|---|
|
||
| id | string | PocketBase auto |
|
||
| filename | string | original filename |
|
||
| file | file | PocketBase file storage |
|
||
| format | select | `pdf` `md` `txt` |
|
||
| status | select | `processing` `processed` `failed` |
|
||
| ingested_at | datetime | |
|
||
| chunk_count | number | total chunks extracted |
|
||
| created_by | relation → `users` | admin who uploaded |
|
||
|
||
---
|
||
|
||
### `themes`
|
||
Top-level content groupings. One Theme = one weekly session.
|
||
|
||
| Field | Type | Notes |
|
||
|---|---|---|
|
||
| id | string | PocketBase auto |
|
||
| title | string | |
|
||
| description | text | AI drafted, admin editable |
|
||
| status | select | `draft` `published` |
|
||
| source_documents | relation[] → `source_documents` | which docs contributed |
|
||
| approved_by | relation → `users` | admin who approved batch |
|
||
| approved_at | datetime | |
|
||
| created_at | datetime | |
|
||
| updated_at | datetime | |
|
||
|
||
---
|
||
|
||
### `topics`
|
||
Atomic knowledge units. Always belong to a Theme.
|
||
|
||
| Field | Type | Notes |
|
||
|---|---|---|
|
||
| id | string | PocketBase auto |
|
||
| theme | relation → `themes` | parent theme |
|
||
| title | string | |
|
||
| body | text (rich) | AI drafted, admin editable |
|
||
| difficulty | select | `introductory` `intermediate` `advanced` |
|
||
| complexity_weight | number | 1–5, used by curriculum generator |
|
||
| status | select | `draft` `published` |
|
||
| related_topics | relation[] → `topics` | lateral relationships |
|
||
| prerequisite_topics | relation[] → `topics` | must-complete-first |
|
||
| contrast_topics | relation[] → `topics` | deliberate opposites |
|
||
| key_terms | json | string[] — feeds glossary |
|
||
| qdrant_chunk_ids | json | string[] — references to embedded chunks |
|
||
| created_at | datetime | |
|
||
| updated_at | datetime | |
|
||
|
||
Relationship types (related / prerequisite / contrast) are stored via the three explicit relation fields rather than a generic relationship table. This keeps queries simple at this scale.
|
||
|
||
---
|
||
|
||
### `micro_learnings`
|
||
Generated content artifacts. One record per topic per type.
|
||
|
||
| Field | Type | Notes |
|
||
|---|---|---|
|
||
| id | string | PocketBase auto |
|
||
| topic | relation → `topics` | |
|
||
| type | select | see type enum below |
|
||
| content | json | structured output — schema varies per type |
|
||
| status | select | `queued` `generated` `published` `rejected` |
|
||
| generation_model | string | model version used |
|
||
| generated_at | datetime | |
|
||
| published_at | datetime | |
|
||
| updated_at | datetime | |
|
||
|
||
**Type enum:**
|
||
`concept_explainer` `scenario_quiz` `misconceptions` `how_to` `comparison_card` `reflection_prompt` `flashcard_set` `case_study` `glossary_anchor` `myth_vs_evidence`
|
||
|
||
**Content JSON schemas per type:**
|
||
|
||
```json
|
||
// concept_explainer
|
||
{
|
||
"paragraphs": ["string", "string"],
|
||
"example": "string"
|
||
}
|
||
|
||
// scenario_quiz
|
||
{
|
||
"scenario": "string",
|
||
"options": [
|
||
{ "label": "A", "text": "string", "correct": true, "explanation": "string" }
|
||
]
|
||
}
|
||
|
||
// misconceptions
|
||
{
|
||
"items": [
|
||
{ "misconception": "string", "correction": "string" }
|
||
]
|
||
}
|
||
|
||
// how_to
|
||
{
|
||
"steps": [
|
||
{ "number": 1, "instruction": "string" }
|
||
]
|
||
}
|
||
|
||
// comparison_card
|
||
{
|
||
"subject_a": "string",
|
||
"subject_b": "string",
|
||
"dimensions": [
|
||
{ "label": "string", "a": "string", "b": "string" }
|
||
]
|
||
}
|
||
|
||
// reflection_prompt
|
||
{
|
||
"prompt": "string",
|
||
"model_answer": "string"
|
||
}
|
||
|
||
// flashcard_set
|
||
{
|
||
"cards": [
|
||
{ "question": "string", "answer": "string" }
|
||
]
|
||
}
|
||
|
||
// case_study
|
||
{
|
||
"scenario": "string",
|
||
"questions": ["string"]
|
||
}
|
||
|
||
// glossary_anchor
|
||
{
|
||
"term": "string",
|
||
"definition": "string",
|
||
"correct_use": "string",
|
||
"misuse": "string"
|
||
}
|
||
|
||
// myth_vs_evidence
|
||
{
|
||
"myth": "string",
|
||
"evidence": "string",
|
||
"sources": ["string"]
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
### `curriculum_versions`
|
||
Versioned 26-week schedule. New version created on each regeneration.
|
||
|
||
| Field | Type | Notes |
|
||
|---|---|---|
|
||
| id | string | PocketBase auto |
|
||
| version | number | increments on each regeneration |
|
||
| status | select | `draft` `active` `superseded` |
|
||
| generated_at | datetime | |
|
||
| approved_by | relation → `users` | admin who confirmed regeneration |
|
||
| approved_at | datetime | |
|
||
| generation_notes | text | why this version was created |
|
||
|
||
Only one version has status `active` at any time.
|
||
|
||
---
|
||
|
||
### `curriculum_weeks`
|
||
Individual week slots. Child of a curriculum version.
|
||
|
||
| Field | Type | Notes |
|
||
|---|---|---|
|
||
| id | string | PocketBase auto |
|
||
| curriculum_version | relation → `curriculum_versions` | |
|
||
| week_number | number | 1–26 |
|
||
| theme | relation → `themes` | |
|
||
| topics | relation[] → `topics` | ordered subset of theme topics |
|
||
| topic_order | json | number[] — explicit ordering |
|
||
| estimated_duration_minutes | number | AI estimate |
|
||
| admin_notes | text | freeform admin annotation |
|
||
|
||
---
|
||
|
||
### `employee_curriculum_state`
|
||
Tracks each employee's position in the curriculum. One record per employee.
|
||
|
||
| Field | Type | Notes |
|
||
|---|---|---|
|
||
| id | string | PocketBase auto |
|
||
| user | relation → `users` | |
|
||
| current_cycle | number | starts at 1, increments on loop |
|
||
| current_week | number | 1–26 |
|
||
| start_date | datetime | rolling start |
|
||
| active_version | relation → `curriculum_versions` | version employee is on |
|
||
| updated_at | datetime | |
|
||
|
||
When curriculum regenerates: `active_version` updates for all employees whose `current_week` is less than the first regenerated week.
|
||
|
||
---
|
||
|
||
### `session_completions`
|
||
Immutable completion records. One record per employee per topic per type.
|
||
|
||
| Field | Type | Notes |
|
||
|---|---|---|
|
||
| id | string | PocketBase auto |
|
||
| user | relation → `users` | |
|
||
| topic | relation → `topics` | |
|
||
| micro_learning | relation → `micro_learnings` | specific type completed |
|
||
| week_number | number | curriculum week at time of completion |
|
||
| cycle | number | which cycle |
|
||
| completed_at | datetime | |
|
||
|
||
Records are never updated or deleted. This is the canonical history.
|
||
|
||
---
|
||
|
||
### `gamification_profiles`
|
||
One record per employee. Updated by progress service on each completion.
|
||
|
||
| Field | Type | Notes |
|
||
|---|---|---|
|
||
| id | string | PocketBase auto |
|
||
| user | relation → `users` | |
|
||
| total_commits | number | cumulative XP |
|
||
| current_level | select | `intern` `junior` `medior` `senior` `staff` `principal` |
|
||
| current_streak_weeks | number | consecutive weeks with ≥1 completion |
|
||
| longest_streak_weeks | number | all-time high |
|
||
| types_used | json | string[] — which of 10 types used at least once |
|
||
| last_active_week | number | used to detect streak breaks |
|
||
| updated_at | datetime | |
|
||
|
||
---
|
||
|
||
### `badges`
|
||
Badge definitions. Seeded at startup, not user-generated.
|
||
|
||
| Field | Type | Notes |
|
||
|---|---|---|
|
||
| id | string | PocketBase auto |
|
||
| key | string | unique slug e.g. `governance_nerd` |
|
||
| tier | select | `bronze` `silver` `gold` `legendary` `content` |
|
||
| label | string | display name |
|
||
| description | string | award condition description |
|
||
| icon | string | emoji or icon key |
|
||
|
||
---
|
||
|
||
### `employee_badges`
|
||
Junction: which employees have earned which badges.
|
||
|
||
| Field | Type | Notes |
|
||
|---|---|---|
|
||
| id | string | PocketBase auto |
|
||
| user | relation → `users` | |
|
||
| badge | relation → `badges` | |
|
||
| earned_at | datetime | |
|
||
| cycle | number | which cycle it was earned in |
|
||
|
||
---
|
||
|
||
### `milestone_cards`
|
||
Public milestone events at weeks 13 and 26.
|
||
|
||
| Field | Type | Notes |
|
||
|---|---|---|
|
||
| id | string | PocketBase auto |
|
||
| user | relation → `users` | |
|
||
| cycle | number | |
|
||
| week | number | 13 or 26 |
|
||
| total_commits | number | snapshot at time of milestone |
|
||
| streak_weeks | number | snapshot |
|
||
| badge_keys | json | string[] — badges held at milestone |
|
||
| created_at | datetime | public feed ordered by this |
|
||
|
||
---
|
||
|
||
## PocketBase users collection (extended)
|
||
|
||
Standard PocketBase `users` collection with additional fields:
|
||
|
||
| Field | Type | Notes |
|
||
|---|---|---|
|
||
| role | select | `admin` `employee` |
|
||
| display_name | string | used in gamification feed |
|
||
| avatar | file | optional |
|
||
|
||
---
|
||
|
||
## Qdrant collections
|
||
|
||
### `source_chunks`
|
||
Embeddings of raw source document chunks. Primary retrieval target for R42.
|
||
|
||
| Field | Type | Notes |
|
||
|---|---|---|
|
||
| id | string | UUID |
|
||
| vector | float[] | 1536 dimensions (text-embedding-3-small) |
|
||
| source_document_id | string | reference to PocketBase |
|
||
| chunk_index | number | position within document |
|
||
| text | string | raw chunk text |
|
||
| theme_id | string | assigned theme (post-extraction) |
|
||
| topic_id | string | assigned topic (post-extraction, nullable) |
|
||
| format | string | pdf / md / txt |
|
||
|
||
### `topic_summaries`
|
||
Embeddings of AI-generated topic body text. Secondary retrieval target.
|
||
|
||
| Field | Type | Notes |
|
||
|---|---|---|
|
||
| id | string | UUID |
|
||
| vector | float[] | 1536 dimensions |
|
||
| topic_id | string | reference to PocketBase |
|
||
| theme_id | string | |
|
||
| title | string | for display in R42 citation |
|
||
| text | string | full topic body |
|
||
|
||
---
|
||
|
||
## Retrieval strategy for R42
|
||
|
||
R42 queries both Qdrant collections and merges results:
|
||
|
||
```
|
||
Employee query
|
||
↓
|
||
Embed query → text-embedding-3-small
|
||
↓
|
||
Qdrant search: source_chunks (top 5) + topic_summaries (top 3)
|
||
↓
|
||
Filter: boost chunks from employee's current week theme
|
||
↓
|
||
Merge + deduplicate by topic_id
|
||
↓
|
||
Top-K context injected into Haiku 4.5 prompt
|
||
↓
|
||
Response includes: answer + cited topic title(s)
|
||
```
|
||
|
||
Source chunks are weighted higher than topic summaries to keep R42 grounded in original source material rather than AI-generated abstractions.
|
||
|
||
---
|
||
|
||
## Indexes and query patterns
|
||
|
||
Critical query patterns the data model must support efficiently:
|
||
|
||
| Query | Collection | Index |
|
||
|---|---|---|
|
||
| All published topics for a theme | topics | theme + status |
|
||
| All micro learnings for a topic | micro_learnings | topic + status |
|
||
| Employee's current week | employee_curriculum_state | user |
|
||
| Weeks for a curriculum version | curriculum_weeks | curriculum_version + week_number |
|
||
| Employee completion history | session_completions | user + cycle |
|
||
| Public leaderboard | gamification_profiles | total_commits + streak |
|
||
| Milestone feed | milestone_cards | created_at DESC |
|
||
| Badges earned by employee | employee_badges | user |
|
||
|
||
PocketBase creates indexes on relation fields by default. Composite indexes on `status` fields should be added manually where query frequency warrants it.
|
||
|
||
---
|
||
|
||
## Data flow summary
|
||
|
||
```
|
||
source_documents
|
||
└── (ingestion service)
|
||
└── qdrant: source_chunks
|
||
└── themes (draft)
|
||
└── topics (draft)
|
||
└── (approval)
|
||
└── topics (published)
|
||
└── qdrant: topic_summaries
|
||
└── micro_learnings (queued → published)
|
||
└── (curriculum service)
|
||
└── curriculum_versions
|
||
└── curriculum_weeks
|
||
└── (employee progress)
|
||
└── session_completions
|
||
└── gamification_profiles
|
||
└── employee_badges
|
||
└── milestone_cards
|
||
```
|