# Ingestion spec: source documents → knowledge graph

Turns admin-uploaded text into `topics` and `relations` using Claude. Runs
entirely client-side; there is no ingestion service.

- **UI:** `src/components/admin/UploadZone.jsx` (Admin → Sources tab)
- **Pipeline:** `src/lib/extractionPipeline.js`
- **Tool/schema:** `emit_knowledge_graph` (`src/lib/llmTools.js`) validated by
  `extractionResultSchema` (`src/lib/llmSchemas.js`)

---

## Upload

- Accepted formats: **`.txt` and `.md`**, max **5 MB** per file.
- Drag-and-drop or click-to-browse. Unsupported files are skipped with a toast.
- The queue tracks each file: `pending → processing → done / failed / cancelled`.
- Progress is polled every ~2s from the `sources.progress` field
  (`{ current, total, message }`), shown as "Chunk N/total".
- **Orphan detection:** sources stuck in `processing` for >5 minutes (e.g. a closed
  tab) can be marked failed or deleted.

---

## Pipeline: `processSourceText(textContent, sourceName, { signal })`

1. **Create source record** with `status='processing'`.
2. **Chunk** the text (`chunkText`): target ~8000 chars per chunk with ~800 chars
   overlap, splitting on sentence/paragraph boundaries (hard-splitting oversized
   sentences). Overlap preserves cross-boundary context.
3. **Known-topics hint** (`buildKnownIdsHint`): up to the 200 most recent existing
   topics are listed so the model reuses existing ids instead of duplicating them.
4. **Per-chunk extraction** via `callLLM`:
   - tier `standard`, `maxTokens: 8192`, `timeoutMs: 180_000`
   - forced `toolChoice` on `emit_knowledge_graph`
   - rate-limited (~20 req/min, burst 2) to protect quota across many chunks
   - system prompt instructs: ≤15 topics per chunk; topic `type` ∈
     {`concept`, `role`, `process`}; `learning_relevance` ∈
     {`core`, `standard`, `peripheral`, `exclude`}; relation `type` ∈
     {`related_to`, `depends_on`, `part_of`, `executed_by`}
5. **Update progress** before each chunk.
6. **Merge** (`mergeKnowledgeGraph`): topics keyed by `id` (new data updates
   existing, but `learning_relevance` is preserved when `relevance_locked` is true);
   relations de-duplicated on `(source, target, type)`. Persisted via
   `db.saveTopics` / `db.saveRelations`.
7. **Finalize:** `status='completed'`, or `failed` (error) / `cancelled` (abort).

Aborting via the `signal` stops the run and marks the source `cancelled`.

---

## Output shape (`emit_knowledge_graph`)

```json
{
  "topics":   [ { "id", "label", "type", "description", "learning_relevance" } ],
  "relations":[ { "source", "target", "type" } ]
}
```

`theme`, `complexity_weight`, and `difficulty` are **not** set here — they are
added later by the curriculum enrichment step (see `docs/curriculum-spec.md`).

---

## Gotchas

- If extraction logs a truncation (`LLMTruncatedError`, `stop_reason: max_tokens`),
  tighten the per-chunk topic cap before raising `max_tokens`.
- A source already `completed` is not re-processed; delete it to force re-analysis.
- There are no embeddings produced here — R42 retrieval is computed at query time
  with TF-IDF over `topics`.