• Numeric range checks (validate/numeric_rules.py): market-calibrated bounds per field type — leverage ratios must be 0–20x, margins 0–20%, pledge percentages 0–100%, run-rate periods 0–60 months. These catch gross errors like a dollar amount misread as a leverage ratio.
• Citation verification (validate/citation_checks.py): for every found field, verifies that the evidence_text actually exists in the source chunks at the claimed page numbers. Uses exact substring matching first, then a 60% word-overlap fallback for minor LLM paraphrasing. This catches hallucinated citations — the model cannot claim evidence from text that doesn't exist in the document.

This is the system's key agentic capability: after validation, the pipeline inspects its own output and decides what to re-examine. The design is informed by self-correction literature (surveyed in Kamoi et al. 2024, Huang et al. 2024, Dhuliawala et al. 2023) which shows that naive "please reconsider" prompts suffer from anchoring bias, while structured evidence-based verification improves accuracy reliably.

Three triggers identify fields needing review: validation warnings (failed numeric/citation checks), low confidence (< 0.7 on found fields), and ambiguous status.

For each flagged field, the system escalates through three stages:

Stage A — Informed verification (temperature 0.2, precise): The prior answer is treated as a falsifiable hypothesis, not a fact to confirm. The prompt asks the LLM to find the exact verbatim quote from the source text that proves or disproves the hypothesis. The response is then validated with deterministic evidence checks — the same citation verification and numeric range checks from Layer 2. If the returned quote is an exact substring of the source chunks (evidence score ≥ 3), the field is resolved without further LLM calls.

Stage B — Blind re-extraction (temperature 0.7, exploratory): Only reached when Stage A's evidence score is too low. The field is extracted from scratch with no prior answer shown and reversed chunk ordering to mitigate the "lost in the middle" positional bias documented in long-context retrieval research. Higher temperature encourages the model to explore different readings rather than repeating the same systematic error. The result goes through the same deterministic evidence checks.

Stage C — Source-grounded judge (temperature 0.0, deterministic): Only reached when both A and B produce weak evidence. This is not a generic "pick A or B" comparison — it is an informed re-extraction that sees all prior candidates AND the actual source text. The judge verifies each candidate's claimed evidence quote against the source, identifies which (if any) is correct, and can extract a corrected answer itself. This avoids the LLM judge reliability problems documented in the literature (position bias, verbosity bias) by grounding the decision in verifiable source text.

Evidence scoring rubric (deterministic, no LLM involved):

A score of ≥ 3 means the evidence is grounded in the actual document text — safe to accept without an LLM judge.

Why three stages with different temperatures?

• Low temperature (0.2) for Stage A: be precise about evidence, stick to what the text says.
• High temperature (0.7) for Stage B: explore different interpretations, break out of self-consistent errors.
• Zero temperature (0.0) for Stage C: make the most deterministic and consistent final judgment possible.

Why deterministic fast-pass filtering? The literature (Kamoi et al. 2024) shows that self-correction works reliably when feedback is external and concrete rather than the model's own belief. Our citation checks and numeric validators are exactly this: deterministic external signals that don't depend on LLM judgment. By using them as fast-pass filters, most fields are resolved cheaply (1 LLM call, 0 judge calls), and the expensive judge only runs for genuinely hard cases.

Benchmark on Daseke term loan (GPT-5.1):

• 5 fields flagged → 3 resolved at Stage A, 1 at Stage B, 1 at Stage C
• Only 1 out of 5 fields required a judge call (80% resolved by deterministic checks)
• 97K extra tokens ($0.26) — significantly cheaper than a naive retry-everything approach
• Fields that failed both A and B are marked permanently uncertain to avoid wasting cost on genuinely ambiguous provisions

The --no-verify flag disables verification when speed or cost matters more than accuracy. The verification_summary in the output JSON records exactly what was flagged, which stage resolved it, and the evidence scores, making the self-correction process fully auditable.

• Ground-truth annotations (eval/ground_truth.py): human-annotated expected values for all 44 fields across 3 documents (132 total), with expected status, numeric value, page numbers, and tolerance thresholds.
• Per-field graders (eval/graders.py): status match (found/not_found), numeric accuracy (within configurable tolerance), and page overlap (F1-style harmonic mean with +/-3 page tolerance).
• Evaluation reporter (eval/run_eval.py): computes precision, recall, F1 per extraction family and overall, with detailed mismatch listings.

python eval/run_eval.py

Deal type is auto-detected from extraction patterns: has EBITDA add-backs + collateral = leveraged; no collateral + no EBITDA = IG unsecured. Missing fields receive contextual annotations — e.g., "IG unsecured: no collateral expected (not a protection gap)" — so scores reflect deal economics rather than extraction gaps.

The 2024 prototype used prompt-based extraction with create_extraction_chain, which frequently produced malformed JSON, hallucinated fields, and required brittle parser workarounds. OpenAI's Structured Outputs guarantee schema adherence — the response is always valid, typed, and parseable. This eliminated an entire class of runtime errors.

Pure embedding-based semantic search has a known blind spot: when a short, high-value definition (e.g., "Applicable Rate" means 4.00% for Term Benchmark Loans) is embedded in a 4000-character chunk containing 20+ other unrelated definitions, the embedding becomes a diluted average of all topics. The target definition's semantic signal is overwhelmed. In testing, the chunk containing "Applicable Rate" ranked 19th out of 475 chunks for our best semantic query — well below the top-10 cutoff.

Keyword search solves this with zero additional API cost: it scans chunk text for exact term matches (e.g., "Applicable Rate", "Material Adverse Effect", "Incremental Facility") and returns chunks ranked by keyword density. Each extractor defines a set of domain-standard keywords that cover the terminology variants used across credit agreements. The semantic and keyword results are merged and deduplicated before being sent to the LLM.

This is a lightweight implementation of the hybrid retrieval pattern (dense + sparse) widely used in production RAG systems (e.g., BM25 + dense retrieval with reciprocal rank fusion). The keyword component adds ~0ms latency (simple string matching over hundreds of chunks) and typically supplements the semantic results with 2-5 additional chunks containing exact defined terms that embeddings miss.

Fixed-character chunking (the 2024 approach) splits text at arbitrary positions, often breaking a covenant definition across two chunks. Section-aware chunking respects document structure: chunks never cross Article or Section boundaries, so retrieval returns topically coherent context.

Every ExtractedField carries evidence_text and page_numbers. This serves two purposes:

1. Validation: the citation check layer verifies that the evidence text actually appears in the source chunks.
2. Trust: users can click "View Evidence" in the UI to see exactly what the model read.

Raw extracted terms are useful for individual deal review, but portfolio-level analysis requires quantitative comparison. The scoring module converts extracted terms into normalized 0–100 scores across three dimensions (covenant tightness, add-back aggressiveness, lender protection), enabling cross-deal ranking and distribution analysis.

Both providers achieve structured extraction through different mechanisms: OpenAI uses JSON Schema via text.format, while Claude uses tool use with input_schema. The same JSON schema works for both — no duplication needed. Running both providers on the same document enables cross-model consensus: fields where both models agree have high confidence, while disagreements flag fields for human review. This provides a confidence signal without requiring ground-truth labels.

The 6 extraction families are independent — each retrieves its own chunks and makes its own LLM call. Running them concurrently via ThreadPoolExecutor cuts extraction time by ~2.5x (135s → 48s for a 182-page document). The ChunkIndex is made thread-safe with a threading.Lock on lazy embedding initialization, and embeddings are pre-warmed before concurrent access.

A naive approach to improving accuracy would be to run extraction twice and take the consensus. But this doubles cost without targeting the actual problems. The self-correction literature also shows that simply asking a model to "reconsider" its answer often reinforces mistakes through anchoring bias (Huang et al. 2024), and that blind retries can repeat systematic errors when retrieval and context are unchanged.

Our staged design addresses both problems:

1. Selective: only re-extracts fields that failed validation, have low confidence, or are ambiguous — typically 5-15% of all fields, not 100%.
2. Evidence-constrained: Stage A treats the prior answer as a hypothesis to be proven from the text, not a fact to confirm. This converts self-correction from an introspective task ("am I right?") into a verification task ("can I find the evidence?"), which the literature shows is more reliable.
3. Independent second opinion: Stage B uses blind re-extraction with reversed chunk ordering and higher temperature — maximally independent from the first attempt.
4. Deterministic fast-pass: most fields are resolved by checking whether the evidence quote actually exists in the source text. No LLM judge needed for clear-cut cases.
5. Source-grounded judge as fallback: when deterministic checks can't resolve, the judge sees all candidates AND the actual source text, so it can verify quotes rather than just comparing "which sounds better."

This is the core agentic behavior in the system: the pipeline inspects its own output, identifies weaknesses, and takes corrective action — the hallmark of an agent versus a static pipeline. The escalation design means easy cases are cheap (1 LLM call) and only hard cases pay for the full 3-call pipeline.

Agent orchestration adds complexity without proportional value for this pipeline. The architecture is a clean sequential pipeline (ingest → retrieve → extract → validate → verify → score) with no branching or dynamic routing needed. The self-verification loop adds targeted agentic self-correction where it matters most, without requiring a full agent orchestration framework. Keeping it simple makes the code auditable and the cost predictable.

The staged self-verification loop is informed by findings from the LLM self-correction, anchoring bias, and information extraction literature. Below are the key papers that shaped the design and how each influenced specific engineering decisions.

Why naive self-correction fails (motivating the staged design)

• Kamoi et al. (2024) — "When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs." TACL. Key finding: self-correction reliably improves only when there is reliable external feedback, not when the model merely critiques its own output. This motivated our deterministic citation/numeric checks as the primary decision mechanism. Paper

• Huang et al. (2024) — "Large Language Models Cannot Self-Correct Reasoning Yet." ICLR. LLMs often cannot self-correct without external feedback, and performance can degrade. This is why Stage A reframes correction as an evidence verification task. Paper

• Tyen et al. (2024) — "LLMs Cannot Find Reasoning Errors, But Can Correct Them!" Models are poor at detecting errors but good at correcting them once located. This validates using deterministic validators to locate problems, then asking the LLM to fix flagged fields. Paper

Why evidence-constrained verification works (motivating Stage A)

• Dhuliawala et al. (2023) — "Chain-of-Verification (CoVe) Reduces Hallucination in Large Language Models." Verification questions must be answered independently to avoid anchoring — directly inspired Stage A's evidence-first design. Paper

• Gero et al. (2023) — "Self-Verification Improves Few-Shot Clinical Information Extraction." Reports F1 gain of +0.056 with evidence-grounded verification in information extraction. Paper

Why blind retry uses reversed chunks (motivating Stage B)

• Liu et al. (2024) — "Lost in the Middle: How Language Models Use Long Contexts." LLMs have degraded recall for middle-positioned information. Stage B reverses chunk ordering to break positional bias. Paper

Why we minimize LLM judge reliance (motivating deterministic adjudication)

• Zheng et al. (2023) — "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." Documents position, verbosity, and self-enhancement bias in LLM judges. Paper

Why sampling diversity helps (motivating temperature variation)

• Wang et al. (2023) — "Self-Consistency Improves Chain of Thought Reasoning in Language Models." ICLR. Multiple reasoning paths improve accuracy — supports temperature variation across stages. Paper

• Manakul et al. (2023) — "SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection." Independent re-sampling reveals instability correlated with errors. Paper