learning-platform/docs/generation-spec.md

# Generation service spec

## Responsibility

Accepts a Theme ID from the admin app (on batch approval) and generates all 10
micro learning types for every published Topic in that Theme. One Claude Sonnet 4
call per type per topic. All outputs validated through Zod schemas before write.

This service runs entirely server-side. The admin app calls it via REST. All AI
calls go through the Anthropic API. No generation logic lives in the frontend.

---

## Service location

```
app/services/generation/
  ├── src/
  │     ├── index.ts              entry point, Fastify server
  │     ├── routes/
  │     │     ├── generate.ts     POST /generate, GET /status/:jobId
  │     │     └── publish.ts      PATCH /micro-learnings/:id
  │     ├── pipeline/
  │     │     └── generate.ts     per-type generation logic
  │     ├── jobs/
  │     │     └── queue.ts        async job queue (in-memory)
  │     ├── lib/
  │     │     ├── pocketbase.ts   PocketBase client
  │     │     └── anthropic.ts    Anthropic client
  │     └── types.ts              shared TypeScript types + Zod schemas
  ├── package.json
  ├── tsconfig.json
  ├── .env.example
  └── .gitignore
```

---

## API surface

### POST /generate

Triggered by admin app when a Theme batch is approved.

Request:
```json
{
  "themeId": "string"
}
```

Response (202 Accepted):
```json
{
  "jobId": "string",
  "status": "queued",
  "topicsFound": 5,
  "totalItems": 50
}
```

Processing is async. The admin app polls job status.

Behaviour:
- Fetches all published Topics for the given themeId
- Creates one micro_learnings record per topic per type with status `queued`
- Generates each item sequentially; updates status to `generated` on success
- On failure: sets individual item status to `failed`, continues remaining items
- Job completes when all items are either `generated` or `failed`

---

### GET /status/:jobId

Returns current job progress.

Response:
```json
{
  "jobId": "string",
  "status": "queued" | "running" | "done" | "failed",
  "progress": {
    "topicsTotal": 5,
    "topicsProcessed": 3,
    "itemsTotal": 50,
    "itemsGenerated": 28,
    "itemsFailed": 2
  },
  "error": "string | null"
}
```

---

### PATCH /micro-learnings/:id

Admin publishes or rejects an individual micro learning.

Request:
```json
{
  "status": "published" | "rejected"
}
```

Response (200 OK):
```json
{
  "id": "string",
  "status": "published" | "rejected",
  "published_at": "datetime | null"
}
```

Rules:
- Only `generated` records can be published or rejected
- `published_at` set on publish, left null on reject
- Returns 400 if record is not in `generated` status
- Returns 404 if record not found

---

## Generation pipeline

### Input

For each Topic in the approved Theme:
```
topic.title: string
topic.body: string
topic.key_terms: string[]
topic.difficulty: 'introductory' | 'intermediate' | 'advanced'
```

### Output

10 micro_learnings records per topic, one per type.

---

## AI call configuration

```typescript
{
  model: 'claude-sonnet-4-20250514',
  max_tokens: 2000,
  temperature: 0          // deterministic structured output
}
```

One call per type per topic. Do not batch multiple types into one call — isolated
calls are easier to retry and validate independently.

---

## Prompt strategy

### System prompt (all types)

```
You are a learning content designer. Your task is to generate structured learning
content for a specific topic in an employee learning platform.

Output ONLY valid JSON matching the schema provided. No preamble, no explanation,
no markdown fences.

The content should be accurate, practical, and appropriate for the stated
difficulty level. Tone: professional but accessible.
```

### User prompt template (all types)

```
Topic: {topic.title}
Difficulty: {topic.difficulty}
Body:
{topic.body}

Key terms: {topic.key_terms.join(', ')}

Generate a {type_label} for this topic.

Output schema:
{JSON.stringify(schemaDescription)}
```

---

## Type-specific prompts and schemas

### concept_explainer

Type label: `Concept Explainer`

Schema description:
```json
{
  "paragraphs": ["2 to 3 paragraphs explaining the concept in plain language"],
  "example": "one concrete real-world example"
}
```

Zod schema:
```typescript
z.object({
  paragraphs: z.array(z.string()).min(2).max(3),
  example: z.string().min(20)
})
```

---

### scenario_quiz

Type label: `Scenario Quiz`

Schema description:
```json
{
  "scenario": "a realistic workplace scenario",
  "options": [
    { "label": "A", "text": "answer text", "correct": false, "explanation": "why" },
    { "label": "B", "text": "answer text", "correct": true,  "explanation": "why" },
    { "label": "C", "text": "answer text", "correct": false, "explanation": "why" },
    { "label": "D", "text": "answer text", "correct": false, "explanation": "why" }
  ]
}
```

Rules: exactly 4 options, exactly 1 correct.

Zod schema:
```typescript
z.object({
  scenario: z.string().min(30),
  options: z.array(z.object({
    label: z.enum(['A', 'B', 'C', 'D']),
    text: z.string().min(5),
    correct: z.boolean(),
    explanation: z.string().min(10)
  })).length(4).refine(
    opts => opts.filter(o => o.correct).length === 1,
    { message: 'exactly one correct option required' }
  )
})
```

---

### misconceptions

Type label: `Misconceptions`

Schema description:
```json
{
  "items": [
    { "misconception": "common wrong belief", "correction": "accurate explanation" }
  ]
}
```

Rules: 3 to 5 items.

Zod schema:
```typescript
z.object({
  items: z.array(z.object({
    misconception: z.string().min(10),
    correction: z.string().min(10)
  })).min(3).max(5)
})
```

---

### how_to

Type label: `How-To Guide`

Schema description:
```json
{
  "steps": [
    { "number": 1, "instruction": "what to do" }
  ]
}
```

Rules: 3 to 8 steps.

Zod schema:
```typescript
z.object({
  steps: z.array(z.object({
    number: z.number().int().positive(),
    instruction: z.string().min(10)
  })).min(3).max(8)
})
```

---

### comparison_card

Type label: `Comparison Card`

Schema description:
```json
{
  "subject_a": "first concept or approach",
  "subject_b": "second concept or approach",
  "dimensions": [
    { "label": "dimension name", "a": "how A differs", "b": "how B differs" }
  ]
}
```

Rules: 3 to 6 dimensions.

Zod schema:
```typescript
z.object({
  subject_a: z.string().min(2),
  subject_b: z.string().min(2),
  dimensions: z.array(z.object({
    label: z.string().min(2),
    a: z.string().min(5),
    b: z.string().min(5)
  })).min(3).max(6)
})
```

---

### reflection_prompt

Type label: `Reflection Prompt`

Schema description:
```json
{
  "prompt": "open-ended question for the employee to reflect on",
  "model_answer": "a thoughtful example answer the employee can compare against"
}
```

Zod schema:
```typescript
z.object({
  prompt: z.string().min(20),
  model_answer: z.string().min(50)
})
```

---

### flashcard_set

Type label: `Flashcard Set`

Schema description:
```json
{
  "cards": [
    { "question": "question text", "answer": "answer text" }
  ]
}
```

Rules: 5 to 10 cards.

Zod schema:
```typescript
z.object({
  cards: z.array(z.object({
    question: z.string().min(5),
    answer: z.string().min(5)
  })).min(5).max(10)
})
```

---

### case_study

Type label: `Case Study`

Schema description:
```json
{
  "scenario": "a detailed real-world scenario (150+ words)",
  "questions": ["discussion or reflection question 1", "discussion or reflection question 2"]
}
```

Rules: 2 to 4 questions.

Zod schema:
```typescript
z.object({
  scenario: z.string().min(150),
  questions: z.array(z.string().min(10)).min(2).max(4)
})
```

---

### glossary_anchor

Type label: `Glossary Anchor`

Schema description:
```json
{
  "term": "the key term",
  "definition": "precise definition",
  "correct_use": "example sentence showing correct use",
  "misuse": "common incorrect usage to avoid"
}
```

Prompt addition: use the first key term from `topic.key_terms` as the anchor term.

Zod schema:
```typescript
z.object({
  term: z.string().min(2),
  definition: z.string().min(20),
  correct_use: z.string().min(20),
  misuse: z.string().min(20)
})
```

---

### myth_vs_evidence

Type label: `Myth vs Evidence`

Schema description:
```json
{
  "myth": "a commonly held misconception about this topic",
  "evidence": "the evidence-based counterpoint",
  "sources": ["source or reference if applicable — leave empty array if none"]
}
```

Zod schema:
```typescript
z.object({
  myth: z.string().min(20),
  evidence: z.string().min(30),
  sources: z.array(z.string())
})
```

---

## Error handling

**Per item:**
- JSON parse failure → retry once with stricter prompt ("respond with valid JSON only, no other text")
- Second failure → set micro_learning status to `failed`, log raw response, continue to next item
- Zod validation failure → same as parse failure: retry once, then `failed`
- Anthropic API error (rate limit / timeout) → exponential backoff, 3 retries, then `failed`

**Per job:**
- If all items for a topic fail → log, continue to next topic
- Job status becomes `done` when all items processed, regardless of individual failures
- Job status becomes `failed` only if the initial topic fetch fails (PocketBase error before generation starts)

---

## PocketBase write

For each generated item:
```typescript
{
  topic: topicId,
  type: type,            // one of the 10 type enum values
  content: validatedContent,  // JSON, validated by Zod
  status: 'generated',
  generation_model: 'claude-sonnet-4-20250514',
  generated_at: new Date().toISOString()
}
```

Create record with status `queued` before generation starts.
Update to `generated` (with content) or `failed` after attempt.

---

## Job lifecycle

```
POST /generate received
  ↓
Fetch published Topics for Theme
  ↓
Create micro_learning records: status = queued
  ↓
Job created → status: running
  ↓
For each topic:
  For each of 10 types:
    Claude call → validate → write content → status = generated
  ↓
All items processed
  ↓
Job status: done
```

On topic fetch failure:
```
status: failed
error: { reason: 'topic_fetch_failed', detail: ... }
```

---

## Environment variables required

```
ANTHROPIC_API_KEY=
POCKETBASE_URL=
POCKETBASE_ADMIN_EMAIL=
POCKETBASE_ADMIN_PASSWORD=
GENERATION_PORT=3002
```

---

## Dependencies

```json
{
  "dependencies": {
    "fastify": "^4",
    "@anthropic-ai/sdk": "^0.24",
    "pocketbase": "^0.21",
    "uuid": "^9",
    "zod": "^3"
  },
  "devDependencies": {
    "typescript": "^5",
    "@types/node": "^20",
    "tsx": "^4"
  }
}
```

---

## TypeScript strict mode requirements

- No `any` types
- All Claude response parsing through Zod schema validation before PocketBase write
- All PocketBase writes typed against micro_learnings schema from data-model.md
- Content type is `unknown` after JSON.parse — always validate through Zod before use

---

## What this service does NOT do

- Does not extract or chunk source documents → ingestion service
- Does not build or schedule the curriculum → curriculum service
- Does not handle admin auth → PocketBase + admin app
- Does not embed content into Qdrant → ingestion service handles all embeddings
- Does not serve R42 queries → chat service

---

## Testing checkpoints

1. Call POST /generate with a themeId that has 2 published topics → verify 20 micro_learning records created
2. All 10 types generated for each topic → verify content JSON parses correctly
3. All Zod schemas pass for each of the 10 types
4. PATCH /micro-learnings/:id with `published` → verify status + published_at updated
5. PATCH /micro-learnings/:id with `rejected` → verify status updated, published_at null
6. Force a JSON parse error (mock) → verify retry logic fires once, then sets status to `failed`
7. GET /status/:jobId during processing → verify progress counters increment correctly