arsvendg/Stirling-PDF
1
0
Fork 0
mirror of https://github.com/arsvendg/Stirling-PDF.git synced 2026-07-16 11:23:10 +02:00
Code Issues Actions 3 Packages Projects Releases 1 Wiki Activity

feat(ai): add Contradiction Agent on a new ChunkedMapper primitive (#6369)

Browse Source 
## Summary

Adds a new AI specialist that finds **textual contradictions** across
one or more PDFs — conflicting claims, recommendations, points of view,
contested facts — built entirely in Python on top of the new
`DocumentService` + `ChunkedReasoner` stack from #6314. Replaces the
closed #6304, which was started before #6314 landed and therefore
over-engineered (Java orchestrator, two-round handshake, resume
artifact, discriminated-union lift).

Two commits:
1. **`refactor(engine): extract ChunkedMapper[T] from ChunkedReasoner`**
— pure refactor, public API of ChunkedReasoner unchanged. New
`ChunkedMapper[T: BaseModel]` is a generic parallel-chunk primitive
(slicing, semaphore, time-bounded extraction, cancellation drain,
progress events) that's now a peer to ChunkedReasoner rather than locked
inside it. The compression loop stays on ChunkedReasoner where it
belongs.
2. **`feat(ai): add Contradiction Agent on ChunkedMapper`** — the agent
itself, plus integrations into `PdfReviewAgent` and `PdfQuestionAgent`.

## Architecture

- **Python-only.** No Java code. No `AgentToolId.CONTRADICTION_AGENT`.
No dedicated HTTP endpoint. No resume artifact, no discriminated-union
lift in `contracts/common.py`. Detector runs inside the Python engine
and the Python engine alone.
- **Review path** (`PdfReviewAgent`): a new
`ContradictionIntentClassifier` fires on contradiction-flavoured
prompts; agent runs detection synchronously and emits a single
`EditPlanResponse(steps=[ADD_COMMENTS])`. Single-turn flow — no resume.
- **Question path** (`PdfQuestionAgent`): a new
`ContradictionCapability` joins `RagCapability` and
`WholeDocReaderCapability` in the smart-model toolset, exposing
`find_contradictions(query)`. The smart model picks it from the toolset
alongside `search_knowledge` and `read_full_document`.

## Inside `ContradictionDetector.detect()`

1. `DocumentService.read_pages(file_id)` → ordered `list[Page]`.
2. `ChunkedMapper[_ExtractedClaims].map_pages(...)` — char-budgeted
multi-page slicing; each slice runs the claim-extractor LLM in parallel
under a semaphore.
3. Page-traceability: the extractor returns `_ExtractedClaim.page`
(which `[Page N]` marker the claim came from). The wrapper validates
`page ∈ chunk.pages`; if not, mechanical fallback searches the chunk's
page text for the verbatim quote and reassigns. If still no match, drop
the claim.
4. `Claim.anchor_quality: Literal[\"verbatim\", \"paraphrased\"]` is set
by a substring check against the declared page's text. Verbatim quotes
feed `anchor_text` for snap-to-quote add-comments placement; paraphrased
ones fall back to margin geometry.
5. Subject canonicalisation: ONE fast-model LLM call collapses synonyms
across the document. Fails open to lexical bucketing.
6. Pre-filters: drop identical-quote pairs; drop same-page same-polarity
paraphrases.
7. Per-bucket pair detection in parallel (separate semaphore, cap 5).
Buckets > 12 claims chunk into windows of 12 with overlap 2; pairs
deduped across overlapping windows by frozen `(i, j)` index pair.
8. Summary fast-model call with fallback string on error.

## Prompt-injection hardening

Every prompt that interpolates user-supplied or PDF-extracted text wraps
content in `<user_message>` / `<verdict>` / `<content>` tags with an
explicit SECURITY preamble instructing the model to treat tagged content
as data only.

## Limitations

- **Combined math + contradiction intent**: when both intent classifiers
fire on the same prompt, contradiction takes precedence and the math
intent is silently dropped. Documented in the Review module docstring
and pinned by
`test_review_integration.py::test_contradiction_precedence_over_math`.
- **Cross-window contradiction reach**: within a subject bucket, pairs
more than ~10 claim indices apart in the same chunked window may be
missed by the overlap-2 strategy. Documented in `test_detector.py`.
Acceptable for v1.

## Settings (engine/src/stirling/config/settings.py)

```python
contradiction_detect_concurrency = 5     # per-bucket detector semaphore
contradiction_bucket_chunk_size = 12     # max claims per detector call
contradiction_bucket_chunk_overlap = 2   # overlap for >threshold buckets
```

`chars_per_slice` and extraction concurrency are reused from the
existing `chunked_reasoner_*` settings.

## Test plan

- [x] `uv run pytest tests/ -v` — **245/245 pass** (210 pre-existing +
35 new)
- [x] `uv run ruff check src/ tests/` — clean
- [x] `uv run pyright src/stirling/agents/contradiction/
src/stirling/contracts/contradiction.py` — 0 errors
- [x] `./gradlew :proprietary:test` — green; no Java was touched, but
verified untouched
- [x] Page-traceability tests cover: valid page kept, hallucinated page
dropped, mechanical-reassign on misattribution, anchor-quality verbatim
vs paraphrased
- [x] Review integration: ADD_COMMENTS plan with two paired CommentSpecs
per contradiction; NeedIngestResponse precheck; precedence vs math
intent pinned
- [x] Question integration: all three capabilities wired into
smart-model toolset; `find_contradictions` returns formatted report text
- [x] ChunkedMapper standalone: slicing, multi-chunk ordering, worker
failures, timeouts, cancellation drain, semaphore saturation
- [x] ChunkedReasoner regression: all pre-existing tests pass unchanged
after the internal split

## Relationship to closed #6304

#6304 was closed in favour of this PR. The closed PR predated #6314 and
modelled the agent as a Java-orchestrated two-round examine/deliberate
flow with its own HTTP endpoint and a discriminated-union resume
artifact. With #6314 making full ordered page text available to the
engine via `DocumentService.read_pages`, none of that is needed. Net
effect: drop ~600 lines of Java, drop the two-round handshake, drop the
`ToolReportArtifact` lift, while ending up with a more scalable agent
(chunk-based instead of page-based extraction; tested to
ChunkedReasoner-equivalent scale).
This commit is contained in:
ConnorYoh
2026-05-22 13:23:52 +00:00
committed by GitHub
parent 0a50e765b7
commit 017c8d59fa
27 changed files with 4401 additions and 268 deletions
Download Patch File Download Diff File Expand all files Collapse all files
engine/tests/agents/test_chunked_reasoner.py
+89 -18
View File
@@ -8,11 +8,13 @@ from __future__ import annotations
import asyncio
from dataclasses import dataclass
from typing import Any
from unittest.mock import AsyncMock, patch
import pytest
from pydantic import BaseModel
from stirling.agents.shared.chunked_mapper import _ChunkExtraction
from stirling.agents.shared.chunked_reasoner import ChunkedReasoner, ChunkNotes
from stirling.contracts import WholeDocSliceDone
from stirling.contracts.documents import Page
@@ -83,14 +85,20 @@ class TestReason:
async def test_runs_one_chunk_per_slice_and_synthesises(self, runtime: AppRuntime) -> None:
"""Three small pages with a generous budget produce one chunk and one extractor call;
the synthesis stage receives notes from all chunks and returns the final answer."""
from stirling.agents.shared.chunked_reasoner import _ExtractedNotes
reasoner = ChunkedReasoner(runtime, chars_per_slice=1000)
pages = [_page(1, "alpha"), _page(2, "beta"), _page(3, "gamma")]
canned_notes = ChunkNotes(pages=[1, 2, 3], summary="all three pages", facts=["fact-1"])
canned_extracted = _ExtractedNotes(summary="all three pages", facts=["fact-1"])
canned_answer = _Answer(answer="final answer")
with (
patch.object(reasoner, "_extract_chunk", AsyncMock(return_value=(canned_notes, 0.0))) as chunk_mock,
patch.object(
reasoner._mapper,
"_extract_chunk",
AsyncMock(return_value=_ChunkExtraction(output=canned_extracted, duration_seconds=0.0)),
) as chunk_mock,
patch.object(reasoner, "_synthesise", AsyncMock(return_value=canned_answer)) as synth_mock,
):
result = await reasoner.reason(
@@ -107,20 +115,29 @@ class TestReason:
assert synth_args is not None
# _synthesise(question, notes, answer_prompt, answer_type)
_, notes_arg, _, type_arg = synth_args.args
assert notes_arg == [canned_notes]
assert len(notes_arg) == 1
assert notes_arg[0].pages == [1, 2, 3]
assert notes_arg[0].summary == "all three pages"
assert notes_arg[0].facts == ["fact-1"]
assert type_arg is _Answer
@pytest.mark.anyio
async def test_fans_out_when_pages_exceed_slice_budget(self, runtime: AppRuntime) -> None:
"""Pages that don't fit into a single slice produce one extractor call per slice."""
from stirling.agents.shared.chunked_reasoner import _ExtractedNotes
reasoner = ChunkedReasoner(runtime, chars_per_slice=10)
pages = [_page(i, "x" * 8) for i in range(1, 6)]
canned_notes = ChunkNotes(pages=[0], summary="placeholder")
canned_extracted = _ExtractedNotes(summary="placeholder")
canned_answer = _Answer(answer="ok")
with (
patch.object(reasoner, "_extract_chunk", AsyncMock(return_value=(canned_notes, 0.0))) as chunk_mock,
patch.object(
reasoner._mapper,
"_extract_chunk",
AsyncMock(return_value=_ChunkExtraction(output=canned_extracted, duration_seconds=0.0)),
) as chunk_mock,
patch.object(reasoner, "_synthesise", AsyncMock(return_value=canned_answer)),
):
await reasoner.reason(
@@ -138,22 +155,24 @@ class TestReason:
"""First-round chunks have no fallback notes, so a failure is dropped
rather than preserving anything; the surviving notes still flow into
synthesis."""
from stirling.agents.shared.chunked_reasoner import _ExtractedNotes
reasoner = ChunkedReasoner(runtime, chars_per_slice=10)
pages = [_page(i, "x" * 8) for i in range(1, 4)]
good = ChunkNotes(pages=[1], summary="ok")
async_results = [good, RuntimeError("chunk boom"), good]
good = _ExtractedNotes(summary="ok")
async_results: list[_ExtractedNotes | BaseException] = [good, RuntimeError("chunk boom"), good]
async def _chunk(*_args: object, **_kwargs: object) -> tuple[ChunkNotes, float]:
async def _chunk(*_args: object, **_kwargs: object) -> _ChunkExtraction[_ExtractedNotes]:
value = async_results.pop(0)
if isinstance(value, BaseException):
raise value
return value, 0.0
return _ChunkExtraction(output=value, duration_seconds=0.0)
canned_answer = _Answer(answer="resilient")
with (
patch.object(reasoner, "_extract_chunk", AsyncMock(side_effect=_chunk)),
patch.object(reasoner._mapper, "_extract_chunk", AsyncMock(side_effect=_chunk)),
patch.object(reasoner, "_synthesise", AsyncMock(return_value=canned_answer)) as synth_mock,
):
result = await reasoner.reason(
@@ -175,7 +194,7 @@ class TestReason:
pages = [_page(i, "x" * 8) for i in range(1, 3)]
with (
patch.object(reasoner, "_extract_chunk", AsyncMock(side_effect=RuntimeError("boom"))),
patch.object(reasoner._mapper, "_extract_chunk", AsyncMock(side_effect=RuntimeError("boom"))),
patch.object(reasoner, "_synthesise", AsyncMock()) as synth_mock,
pytest.raises(RuntimeError, match="no notes"),
):
@@ -310,9 +329,13 @@ class TestPromptConstruction:
def test_extraction_prompt_includes_question_and_page_markers(self, runtime: AppRuntime) -> None:
"""A first-round chunk's content carries ``[Page N]`` markers; the
extraction prompt prepends the user question."""
from stirling.agents.shared.chunked_mapper import ChunkedMapper
reasoner = ChunkedReasoner(runtime)
chunk = reasoner._chunk_from_pages([_page(2, "page two body"), _page(3, "page three body")])
prompt = reasoner._build_extraction_prompt(chunk.content, "what is on page two?")
# Render chunk content through the mapper's public helper — the
# first-round chunk shape lives in ChunkedMapper.
content = ChunkedMapper.format_chunk_content([_page(2, "page two body"), _page(3, "page three body")])
prompt = reasoner._build_extraction_prompt(content, "what is on page two?")
assert "what is on page two?" in prompt
assert "[Page 2]" in prompt
@@ -334,10 +357,11 @@ class TestPromptConstruction:
# Hierarchical compression
#
# The compression loop is part of ``_run_chunks`` and isn't exposed
# directly, so these tests drive it end-to-end via ``gather_notes`` with a
# stubbed extractor that controls per-call output (and per-call failure
# patterns) by counting calls.
# The compression loop is part of ``_compress_until_fits`` /
# ``_run_compression_round`` and isn't exposed directly, so these tests
# drive it end-to-end via ``gather_notes`` with a stubbed extractor that
# controls per-call output (and per-call failure patterns) by counting
# calls.
class TestCompression:
@@ -503,9 +527,56 @@ class TestExtractChunk:
"run",
AsyncMock(return_value=_StubAgentResult(output=canned)),
):
note, _ = await reasoner._extract_chunk(chunk, "anything")
extraction = await reasoner._extract_compression_chunk(chunk, "compress these")
note = extraction.output
assert note.pages == [1, 2, 3, 4, 5]
assert note.summary == "merged"
assert note.facts == ["x"]
assert note.relevant_excerpts == ["y"]
assert extraction.duration_seconds >= 0
@pytest.mark.anyio
async def test_compression_rounds_receive_user_question_through_gather_notes(self, runtime: AppRuntime) -> None:
"""Regression — every extractor call (first round AND every
compression round) MUST carry the same user question. The pre-fix
bug passed ``""`` to the compression-round prompt builder, so the
model consolidated notes against different relevance criteria
than it extracted them under. Flagged by Aikido on PR #6369;
pinned end-to-end here by capturing every prompt the extractor
sees while ``gather_notes`` forces a compression round through a
tight notes budget.
"""
from stirling.agents.shared.chunked_reasoner import _ExtractedNotes
# Small notes budget forces a compression round; small slice
# budget produces multiple first-round chunks that overflow it.
reasoner = ChunkedReasoner(runtime, chars_per_slice=200, notes_char_budget=200)
pages = [_page(i, "x" * 150) for i in range(1, 5)]
call_count = 0
async def _stub(*_args: object, **_kwargs: object) -> _StubAgentResult[object]:
nonlocal call_count
call_count += 1
if call_count <= 4:
# Round 1: each note ~60 chars rendered. 4 * 80 = 320 chars,
# over the 200 budget so a compression round must fire.
return _StubAgentResult(output=_ExtractedNotes(summary="x" * 60))
# Round 2: smaller note so the post-round set fits the budget.
return _StubAgentResult(output=_ExtractedNotes(summary="ok"))
seen_prompts: list[str] = []
async def _capture(prompt: str, *_a: Any, **_kw: Any) -> Any:
seen_prompts.append(prompt)
return await _stub()
with patch.object(reasoner._extractor, "run", side_effect=_capture):
await reasoner.gather_notes(pages, "what is the deadline?")
# At least four first-round calls plus the compression-round
# calls — every single one must carry the user question.
assert len(seen_prompts) >= 5
for prompt in seen_prompts:
assert "what is the deadline?" in prompt