learning-platform/docs/ingestion-spec.md

Ingestion service spec

Responsibility

Accepts uploaded source documents (PDF, MD, TXT), extracts clean text, chunks it, generates embeddings, and produces a structured draft KB (Themes + Topics + relationships) ready for admin review.

This service runs entirely server-side. The admin app calls it via REST. All AI calls go through the Anthropic API. No ingestion logic lives in the frontend.

---

Service location

app/services/ingestion/
  ├── index.ts              entry point, Fastify server
  ├── routes/
  │     └── documents.ts    POST /ingest, GET /status/:jobId
  ├── pipeline/
  │     ├── extract.ts      format detection + text extraction
  │     ├── chunk.ts        chunking strategies per format
  │     ├── clean.ts        chunk cleaning
  │     ├── structure.ts    Claude call → Theme/Topic extraction
  │     └── embed.ts        embedding generation + Qdrant write
  ├── jobs/
  │     └── queue.ts        async job queue (in-memory, BullMQ later if needed)
  ├── lib/
  │     ├── pocketbase.ts   PocketBase client
  │     ├── qdrant.ts       Qdrant client
  │     ├── anthropic.ts    Anthropic client
  │     └── openai.ts       OpenAI embeddings client
  └── types.ts              shared TypeScript types

---

API surface

POST /ingest

Triggered by admin app on document upload.

Request:

{
  "documentId": "string",
  "filename": "string",
  "format": "pdf" | "md" | "txt",
  "filePath": "string"
}

Response (202 Accepted):

{
  "jobId": "string",
  "status": "queued"
}

Processing is async. The admin app polls job status.

---

GET /status/:jobId

Returns current job progress.

Response:

{
  "jobId": "string",
  "status": "queued" | "extracting" | "chunking" | "structuring" | "embedding" | "done" | "failed",
  "progress": {
    "chunksTotal": 42,
    "chunksEmbedded": 18,
    "themesFound": 3,
    "topicsFound": 14
  },
  "error": "string | null"
}

---

Pipeline stages

Stage 1 — Text extraction

Input: file path + format Output: raw text string

format === 'txt'
  → read file directly as UTF-8

format === 'md'
  → read file directly as UTF-8
  → preserve heading markers (# ## ###) — used in chunking

format === 'pdf'
  → pdfplumber: extract text page by page
  → concatenate with page break markers: \n\n---PAGE---\n\n
  → strip known PDF artefacts: headers/footers repeating on every page,
    page numbers, watermarks

Failure handling:

• PDF extraction returns empty string → mark job failed, reason: pdf_extraction_empty
• File not found → mark job failed, reason: file_not_found

---

Stage 2 — Chunking

Input: raw text + format Output: Chunk[]

Chunking strategy differs per format.

MD chunking — heading-based (preferred)

Split on heading markers: #, ##, ###
Each heading + its following content = one chunk
Minimum chunk size: 100 characters
  → if heading section is < 100 chars, merge with next sibling
Maximum chunk size: 1500 characters
  → if section exceeds limit, split on paragraph breaks within section
Metadata preserved per chunk:
  heading_level: 1 | 2 | 3
  heading_text: string
  parent_heading: string | null

MD chunking produces the highest quality structural signal for Theme/Topic extraction. Admins should be advised to provide source material as MD where possible.

TXT chunking — sliding window

Window size: 800 characters
Overlap: 150 characters
Split on: paragraph breaks (\n\n) first, then sentence boundaries, then hard cut
Metadata per chunk:
  chunk_index: number
  approximate_position: 'start' | 'middle' | 'end'

PDF chunking — page + paragraph

Split on ---PAGE--- markers from extraction stage
Within each page: split on paragraph breaks (\n\n)
Minimum chunk size: 100 characters
  → merge sub-threshold paragraphs with adjacent chunk
Maximum chunk size: 1200 characters
  → hard split at sentence boundary
Metadata per chunk:
  page_number: number
  chunk_index_on_page: number

Chunk type:

type Chunk = {
  id: string               // UUID generated at chunking
  documentId: string
  text: string
  format: 'pdf' | 'md' | 'txt'
  index: number            // global position in document
  metadata: {
    // MD-specific
    headingLevel?: number
    headingText?: string
    parentHeading?: string
    // TXT-specific
    approximatePosition?: 'start' | 'middle' | 'end'
    // PDF-specific
    pageNumber?: number
    chunkIndexOnPage?: number
  }
}

---

Stage 3 — Chunk cleaning

Input: Chunk[] Output: Chunk[] (cleaned)

Applied to all formats:

- trim leading/trailing whitespace
- collapse 3+ consecutive newlines to 2
- remove null bytes and non-printable characters
- remove chunks where text.length < 80 after cleaning
  → these are likely artefacts (page numbers, standalone headers)
- normalise unicode: NFC normalisation
- do not strip punctuation or alter sentence structure

---

Stage 4 — Structure extraction (AI)

Input: Chunk[] Output: DraftKB

This is the core AI call. Claude Sonnet 4 reads all chunks and returns a structured KB draft as JSON.

Prompt strategy:

System prompt:

You are a knowledge architect. Your task is to analyse a set of text chunks from
a source document and extract a structured knowledge base.

Output ONLY valid JSON matching the schema provided. No preamble, no explanation,
no markdown fences.

Rules:
- Group related content into Themes. A Theme is a broad subject area.
- Under each Theme, identify discrete Topics. A Topic covers one specific concept.
- Identify relationships between Topics: related, prerequisite, or contrast.
  - related: Topics that complement each other
  - prerequisite: Topic A must be understood before Topic B
  - contrast: Topics that represent opposing approaches or concepts
- For each Topic, extract key terms suitable for a glossary.
- Assign a complexity weight (1–5) to each Topic.
  1 = introductory, 5 = advanced
- Draft a body for each Topic (2–4 paragraphs) based on the source chunks.
- Draft a description for each Theme (1–2 sentences).
- Every Topic must reference the chunk IDs that contributed to it.

User prompt:

Source document: {filename}
Format: {format}

Chunks:
{chunks mapped as: [CHUNK-{id}]\n{text}\n}

Extract the knowledge base structure from these chunks.

Output schema:

type DraftKB = {
  themes: DraftTheme[]
}

type DraftTheme = {
  title: string
  description: string
  topics: DraftTopic[]
}

type DraftTopic = {
  title: string
  body: string
  difficulty: 'introductory' | 'intermediate' | 'advanced'
  complexityWeight: number    // 1–5
  keyTerms: string[]
  sourceChunkIds: string[]    // references Chunk.id values
  relationships: {
    related: string[]         // topic titles (resolved to IDs after write)
    prerequisites: string[]
    contrasts: string[]
  }
}

AI call configuration:

{
  model: 'claude-sonnet-4-20250514',
  max_tokens: 8000,
  temperature: 0            // deterministic output for structured extraction
}

Chunking strategy for large documents: If total chunk count exceeds 60 chunks, split into batches of 40 with 5-chunk overlap. Run one Claude call per batch. Merge resulting DraftKB objects:

• Themes with identical titles → merge Topics
• Duplicate Topic titles within a Theme → keep longer body, merge sourceChunkIds
• Relationships are resolved after full merge

Error handling:

• JSON parse failure → retry once with stricter prompt ("ensure valid JSON only")
• Second failure → mark job failed, reason: structure_extraction_failed, log raw response
• Empty themes array → mark job failed, reason: no_structure_found

---

Stage 5 — Write to PocketBase

Input: DraftKB Output: written Theme + Topic records with status draft

For each DraftTheme:
  create themes record {
    title, description,
    status: 'draft',
    source_documents: [documentId]
  }

For each DraftTopic under the theme:
  create topics record {
    theme: themeId,
    title, body, difficulty, complexity_weight, key_terms,
    status: 'draft',
    qdrant_chunk_ids: []     // populated in stage 6
  }

After all topics created:
  resolve relationship titles → topic IDs
  update topics.related_topics, prerequisite_topics, contrast_topics

If a relationship title cannot be resolved to an existing topic:
  skip silently (cross-document relationships resolved in a later pass)

---

Stage 6 — Embedding generation + Qdrant write

Input: Chunk[], written Topic records Output: vectors in Qdrant, qdrant_chunk_ids updated on Topic records

Source chunk embeddings:

For each Chunk (post-cleaning):
  embed Chunk.text → text-embedding-3-small (1536 dimensions)
  write to Qdrant collection: source_chunks {
    id: Chunk.id,
    vector: float[],
    payload: {
      source_document_id: documentId,
      chunk_index: Chunk.index,
      text: Chunk.text,
      theme_id: resolved themeId | null,
      topic_id: resolved topicId | null,
      format: Chunk.format
    }
  }

Topic summary embeddings:

For each published Topic:
  embed Topic.body → text-embedding-3-small
  write to Qdrant collection: topic_summaries {
    id: UUID,
    vector: float[],
    payload: {
      topic_id: Topic.id,
      theme_id: Topic.theme,
      title: Topic.title,
      text: Topic.body
    }
  }

Update Topic.qdrant_chunk_ids with all Chunk.ids that reference this topic

Batching: OpenAI embeddings API: batch in groups of 100 texts per request to stay within rate limits and reduce latency.

---

Job lifecycle

POST /ingest received
  ↓
Job created → status: queued
  ↓
Stage 1: extracting
  ↓
Stage 2–3: chunking
  ↓
Stage 4: structuring
  ↓
Stage 5: writing to PocketBase
  ↓
Stage 6: embedding
  ↓
status: done
  ↓
Admin notification: "Document processed. N themes, N topics ready for review."
  ↓
Curriculum regeneration queued (status: pending_admin_confirm)

On any stage failure:

status: failed
error: { stage, reason, detail }
Source document status → 'failed' in PocketBase
Admin notification: "Ingestion failed: {reason}"

---

Environment variables required

ANTHROPIC_API_KEY=
OPENAI_API_KEY=
POCKETBASE_URL=
POCKETBASE_ADMIN_EMAIL=
POCKETBASE_ADMIN_PASSWORD=
QDRANT_URL=
QDRANT_API_KEY=             # empty string if running locally without auth
INGESTION_PORT=3001

---

Dependencies

{
  "dependencies": {
    "fastify": "^4",
    "@anthropic-ai/sdk": "^0.24",
    "openai": "^4",
    "@qdrant/js-client-rest": "^1.9",
    "pocketbase": "^0.21",
    "pdfplumber": "NOT JS — see note below",
    "pdf-parse": "^1.1",
    "uuid": "^9",
    "zod": "^3"
  }
}

PDF extraction note: pdfplumber is a Python library. Two options:

1. Use pdf-parse (Node.js) — simpler, covers 90% of cases
2. Run pdfplumber as a Python sidecar process via child_process — higher quality for complex PDFs with tables and columns

Default to pdf-parse initially. Add pdfplumber sidecar only if extraction quality is insufficient for actual source documents.

---

TypeScript strict mode requirements

• No any types
• All Claude response parsing through Zod schema validation
• All PocketBase writes typed against collection schemas from data-model.md
• Qdrant payloads typed explicitly — no untyped objects

---

What this service does NOT do

• Does not generate micro learnings → generation service
• Does not build or update the curriculum → curriculum service
• Does not handle admin approval → admin app + PocketBase directly
• Does not serve R42 queries → chat service
• Does not handle auth → PocketBase + admin app

---

Testing checkpoints

Before handing to Claude Code for implementation, verify manually:

1. Upload a short MD file (< 10 headings) → inspect chunk output → confirm heading structure preserved
2. Upload a simple PDF (< 5 pages) → inspect chunk output → confirm no artefacts
3. Run structure extraction on known chunks → validate JSON parses against Zod schema
4. Confirm PocketBase draft records created with correct theme → topic hierarchy
5. Confirm Qdrant source_chunks collection populated with correct payload fields
6. Confirm topic.qdrant_chunk_ids updated after embedding stage