learning-platform/docs/ingestion-spec.md

Ingestion spec: source documents → knowledge graph

Turns admin-uploaded text into topics and relations using Claude. Runs entirely client-side; there is no ingestion service.

• UI: src/components/admin/UploadZone.jsx (Admin → Sources tab)
• Pipeline: src/lib/extractionPipeline.js
• Tool/schema: emit_knowledge_graph (src/lib/llmTools.js) validated by extractionResultSchema (src/lib/llmSchemas.js)

---

Upload

• Accepted formats: .txt and .md, max 5 MB per file.
• Drag-and-drop or click-to-browse. Unsupported files are skipped with a toast.
• The queue tracks each file: pending → processing → done / failed / cancelled.
• Progress is polled every ~2s from the sources.progress field ({ current, total, message }), shown as "Chunk N/total".
• Orphan detection: sources stuck in processing for >5 minutes (e.g. a closed tab) can be marked failed or deleted.

---

Pipeline: processSourceText(textContent, sourceName, { signal })

1. Create source record with status='processing'.
2. Chunk the text (chunkText): target ~8000 chars per chunk with ~800 chars overlap, splitting on sentence/paragraph boundaries (hard-splitting oversized sentences). Overlap preserves cross-boundary context.
3. Known-topics hint (buildKnownIdsHint): up to the 200 most recent existing topics are listed so the model reuses existing ids instead of duplicating them.
4. Per-chunk extraction via callLLM:
   • tier standard, maxTokens: 8192, timeoutMs: 180_000
   • forced toolChoice on emit_knowledge_graph
   • rate-limited (~20 req/min, burst 2) to protect quota across many chunks
   • system prompt instructs: ≤15 topics per chunk; topic type ∈ {concept, role, process}; learning_relevance ∈ {core, standard, peripheral, exclude}; relation type ∈ {related_to, depends_on, part_of, executed_by}
5. Update progress before each chunk.
6. Merge (mergeKnowledgeGraph): topics keyed by id (new data updates existing, but learning_relevance is preserved when relevance_locked is true); relations de-duplicated on (source, target, type). Persisted via db.saveTopics / db.saveRelations.
7. Finalize: status='completed', or failed (error) / cancelled (abort).

Aborting via the signal stops the run and marks the source cancelled.

---

Output shape (emit_knowledge_graph)

{
  "topics":   [ { "id", "label", "type", "description", "learning_relevance" } ],
  "relations":[ { "source", "target", "type" } ]
}

theme, complexity_weight, and difficulty are not set here — they are added later by the curriculum enrichment step (see docs/curriculum-spec.md).

---

Gotchas

• If extraction logs a truncation (LLMTruncatedError, stop_reason: max_tokens), tighten the per-chunk topic cap before raising max_tokens.
• A source already completed is not re-processed; delete it to force re-analysis.
• There are no embeddings produced here — R42 retrieval is computed at query time with TF-IDF over topics.