rve/learning-platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: phase 3 of AI pipeline hardening — extraction quality #5

Merged
rve merged 1 commits from feat/ai-pipeline-hardening-phase-3 into main 2026-05-20 15:57:41 +00:00
Conversation 0 Commits 1 Files Changed 10
+509 -48
rve commented 2026-05-20 15:57:34 +00:00
Owner
Copy Link
Copy Source

Replace stateless one-shot extraction with a stateful, paced, cancellable
pipeline. Six subtasks:

  • 3.1 Sentence-aware chunking with 800-char overlap (was paragraph-only
    at 4000 chars). Hard-split fallback for runaway sentences.
  • 3.2 Stateful extraction: chunks 2+ receive an "already-extracted topic
    IDs" hint capped at 200 IDs, so the model reuses IDs instead of
    inventing variants like software-developer vs software-engineer.
  • 3.3 Token-bucket limiter in llmRetry.js (extractionLimiter, 5 req/min).
    callLLM awaits the limiter before fetch; 429+Retry-After calls
    pauseUntil. Replaces hard setTimeout(12000) and setTimeout(15000).
  • 3.4 relevance_locked column on topics — admin edits to relevance are
    sticky across re-extraction. Migration + merge respects the flag +
    unlock checkbox in KnowledgeGraph edit form.
  • 3.5 Unify relation vocabulary — handbook prompt no longer mentions
    legacy "executes"; one-shot migration rewrites existing executes rows
    to executed_by with source/target swapped.
  • 3.6 Cancellation — Cancel button on UploadZone wired to an
    AbortController threaded into callLLM; aborted runs persist status =
    "cancelled" rather than "failed".

Tests: 16 new unit tests for chunkText, buildKnownIdsHint, and
createLimiter. All 61 tests pass, 0 lint errors, build clean.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

Replace stateless one-shot extraction with a stateful, paced, cancellable pipeline. Six subtasks: - 3.1 Sentence-aware chunking with 800-char overlap (was paragraph-only at 4000 chars). Hard-split fallback for runaway sentences. - 3.2 Stateful extraction: chunks 2+ receive an "already-extracted topic IDs" hint capped at 200 IDs, so the model reuses IDs instead of inventing variants like software-developer vs software-engineer. - 3.3 Token-bucket limiter in llmRetry.js (extractionLimiter, 5 req/min). callLLM awaits the limiter before fetch; 429+Retry-After calls pauseUntil. Replaces hard setTimeout(12000) and setTimeout(15000). - 3.4 relevance_locked column on topics — admin edits to relevance are sticky across re-extraction. Migration + merge respects the flag + unlock checkbox in KnowledgeGraph edit form. - 3.5 Unify relation vocabulary — handbook prompt no longer mentions legacy "executes"; one-shot migration rewrites existing executes rows to executed_by with source/target swapped. - 3.6 Cancellation — Cancel button on UploadZone wired to an AbortController threaded into callLLM; aborted runs persist status = "cancelled" rather than "failed". Tests: 16 new unit tests for chunkText, buildKnownIdsHint, and createLimiter. All 61 tests pass, 0 lint errors, build clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
rve added 1 commit 2026-05-20 15:57:34 +00:00
feat: phase 3 of AI pipeline hardening — extraction quality
On Pull Request to Main / publish (pull_request) Successful in 1m1s
Details
On Pull Request to Main / deploy-dev (pull_request) Successful in 1m32s
Details
On Pull Request to Main / test (pull_request) Successful in 31s
Details
aeb197d5f4 
Replace stateless one-shot extraction with a stateful, paced, cancellable
pipeline. Six subtasks:

- 3.1 Sentence-aware chunking with 800-char overlap (was paragraph-only
  at 4000 chars). Hard-split fallback for runaway sentences.
- 3.2 Stateful extraction: chunks 2+ receive an "already-extracted topic
  IDs" hint capped at 200 IDs, so the model reuses IDs instead of
  inventing variants like software-developer vs software-engineer.
- 3.3 Token-bucket limiter in llmRetry.js (extractionLimiter, 5 req/min).
  callLLM awaits the limiter before fetch; 429+Retry-After calls
  pauseUntil. Replaces hard setTimeout(12000) and setTimeout(15000).
- 3.4 relevance_locked column on topics — admin edits to relevance are
  sticky across re-extraction. Migration + merge respects the flag +
  unlock checkbox in KnowledgeGraph edit form.
- 3.5 Unify relation vocabulary — handbook prompt no longer mentions
  legacy "executes"; one-shot migration rewrites existing executes rows
  to executed_by with source/target swapped.
- 3.6 Cancellation — Cancel button on UploadZone wired to an
  AbortController threaded into callLLM; aborted runs persist status =
  "cancelled" rather than "failed".

Tests: 16 new unit tests for chunkText, buildKnownIdsHint, and
createLimiter. All 61 tests pass, 0 lint errors, build clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
rve merged commit fd3b849c19 into main 2026-05-20 15:57:41 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
eho (Edwin van den Houdt) lmi (Leroy Mies) lpr (Leon de Pruyssenaere de la Woestijne) not (Niek Otten) rdi (Robert van Diest) rve (Raymond Verhoef)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: rve/learning-platform#5