Islam Timeline Research Pipeline

Overview

This is a local-first research scaffold that processes scanned Islamic source material into structured, reviewable data for a future history timeline. A locally hosted, OpenAI-compatible language model does the extraction; everything around it exists to keep that extraction honest.

The pipeline moves through explicit stages, each writing JSONL to disk:

PDF source → raw pages → hadith units → report extractions
→ similarity candidates → event candidates → event clusters
→ review decisions → approved research data → review dashboard

The problem

Language models are good at reading unstructured text and bad at admitting uncertainty. That combination is dangerous here. The source material is religious scholarship, the OCR is imperfect, and a fluent, confident, wrong answer is worse than no answer at all.

I also did not want a system that produced a finished-looking artifact from an unreviewed model output. The dashboard states plainly that it contains unreviewed AI-assisted material and is not a citation source or a source of religious guidance.

Approach

The design treats the model as one untrusted stage in a pipeline rather than as the pipeline.

Every stage writes to disk. Each step reads the previous stage’s JSONL and writes its own. Any stage can be re-run over a range with --offset and --limit without redoing the ones before it.

Output is schema-validated. Nine JSON Schemas (raw page, hadith unit, report extraction, similarity candidate, event candidate, event cluster, review decision, approved event, source manifest) are enforced with Ajv. Output that does not match its schema does not advance.

Prompts are versioned artifacts. Extraction prompts live in prompts/ under explicit versions, and .env names the active one. A prompt change is a tracked change, so an extraction can be attributed to the exact instructions that produced it.

The model runs at temperature 0 against a local endpoint, with bounded retries and an input character cap. Nothing leaves the machine.

Designing for refusal

Most of the prompt engineering is about what the model must not do. The extraction prompt opens by stating that its job is not to issue rulings, authenticate hadith, or make final scholarly judgments, only to extract what a report says, preserve uncertainty, and flag anything needing review.

That translates into concrete schema behavior. Legal and jurisprudential reports are classified as not-events and forced to needs_human_review: true with a red review priority. Dates are only numeric when the text itself supplies one or a standard historical anchor exists; otherwise the record carries undated, approximate, or broad_period_only along with the reasoning. OCR-damaged input is still extracted, but with the damage recorded in the certainty notes.

The point is a model that produces auditable uncertainty instead of confident guesses.

What I directed versus what the model produced

I designed the stage boundaries, the schemas, the review gates, and the classification rules the prompt enforces. AI coding tools wrote much of the surrounding script code to my specification, and the language model performs the extraction itself. The judgment about what the system is allowed to assert, and where it must stop and ask a human, is mine.

Quality control

Separate QC commands audit the accumulated data rather than any single run: counts per stage, orphaned records with no parent, review-state coverage, and provenance completeness. A build step compiles approved data into a SQLite database for the private dashboard.

Current state

The pipeline runs end to end over source volumes and produces reviewed research data. Extraction quality work and the review backlog are ongoing. The source is private, and no source PDFs are committed.

What I learned

The interesting engineering in an LLM system is almost never the prompt. It is the scaffolding that decides what happens when the model is wrong: schema validation, resumable stages, versioned prompts, forced review paths. Building that first made the model a component I could reason about rather than a black box I had to trust.