mirror of
https://github.com/arsvendg/Stirling-PDF.git
synced 2026-07-16 19:33:11 +02:00
feat(ai): add Contradiction Agent on a new ChunkedMapper primitive (#6369)
## Summary Adds a new AI specialist that finds **textual contradictions** across one or more PDFs — conflicting claims, recommendations, points of view, contested facts — built entirely in Python on top of the new `DocumentService` + `ChunkedReasoner` stack from #6314. Replaces the closed #6304, which was started before #6314 landed and therefore over-engineered (Java orchestrator, two-round handshake, resume artifact, discriminated-union lift). Two commits: 1. **`refactor(engine): extract ChunkedMapper[T] from ChunkedReasoner`** — pure refactor, public API of ChunkedReasoner unchanged. New `ChunkedMapper[T: BaseModel]` is a generic parallel-chunk primitive (slicing, semaphore, time-bounded extraction, cancellation drain, progress events) that's now a peer to ChunkedReasoner rather than locked inside it. The compression loop stays on ChunkedReasoner where it belongs. 2. **`feat(ai): add Contradiction Agent on ChunkedMapper`** — the agent itself, plus integrations into `PdfReviewAgent` and `PdfQuestionAgent`. ## Architecture - **Python-only.** No Java code. No `AgentToolId.CONTRADICTION_AGENT`. No dedicated HTTP endpoint. No resume artifact, no discriminated-union lift in `contracts/common.py`. Detector runs inside the Python engine and the Python engine alone. - **Review path** (`PdfReviewAgent`): a new `ContradictionIntentClassifier` fires on contradiction-flavoured prompts; agent runs detection synchronously and emits a single `EditPlanResponse(steps=[ADD_COMMENTS])`. Single-turn flow — no resume. - **Question path** (`PdfQuestionAgent`): a new `ContradictionCapability` joins `RagCapability` and `WholeDocReaderCapability` in the smart-model toolset, exposing `find_contradictions(query)`. The smart model picks it from the toolset alongside `search_knowledge` and `read_full_document`. ## Inside `ContradictionDetector.detect()` 1. `DocumentService.read_pages(file_id)` → ordered `list[Page]`. 2. `ChunkedMapper[_ExtractedClaims].map_pages(...)` — char-budgeted multi-page slicing; each slice runs the claim-extractor LLM in parallel under a semaphore. 3. Page-traceability: the extractor returns `_ExtractedClaim.page` (which `[Page N]` marker the claim came from). The wrapper validates `page ∈ chunk.pages`; if not, mechanical fallback searches the chunk's page text for the verbatim quote and reassigns. If still no match, drop the claim. 4. `Claim.anchor_quality: Literal[\"verbatim\", \"paraphrased\"]` is set by a substring check against the declared page's text. Verbatim quotes feed `anchor_text` for snap-to-quote add-comments placement; paraphrased ones fall back to margin geometry. 5. Subject canonicalisation: ONE fast-model LLM call collapses synonyms across the document. Fails open to lexical bucketing. 6. Pre-filters: drop identical-quote pairs; drop same-page same-polarity paraphrases. 7. Per-bucket pair detection in parallel (separate semaphore, cap 5). Buckets > 12 claims chunk into windows of 12 with overlap 2; pairs deduped across overlapping windows by frozen `(i, j)` index pair. 8. Summary fast-model call with fallback string on error. ## Prompt-injection hardening Every prompt that interpolates user-supplied or PDF-extracted text wraps content in `<user_message>` / `<verdict>` / `<content>` tags with an explicit SECURITY preamble instructing the model to treat tagged content as data only. ## Limitations - **Combined math + contradiction intent**: when both intent classifiers fire on the same prompt, contradiction takes precedence and the math intent is silently dropped. Documented in the Review module docstring and pinned by `test_review_integration.py::test_contradiction_precedence_over_math`. - **Cross-window contradiction reach**: within a subject bucket, pairs more than ~10 claim indices apart in the same chunked window may be missed by the overlap-2 strategy. Documented in `test_detector.py`. Acceptable for v1. ## Settings (engine/src/stirling/config/settings.py) ```python contradiction_detect_concurrency = 5 # per-bucket detector semaphore contradiction_bucket_chunk_size = 12 # max claims per detector call contradiction_bucket_chunk_overlap = 2 # overlap for >threshold buckets ``` `chars_per_slice` and extraction concurrency are reused from the existing `chunked_reasoner_*` settings. ## Test plan - [x] `uv run pytest tests/ -v` — **245/245 pass** (210 pre-existing + 35 new) - [x] `uv run ruff check src/ tests/` — clean - [x] `uv run pyright src/stirling/agents/contradiction/ src/stirling/contracts/contradiction.py` — 0 errors - [x] `./gradlew :proprietary:test` — green; no Java was touched, but verified untouched - [x] Page-traceability tests cover: valid page kept, hallucinated page dropped, mechanical-reassign on misattribution, anchor-quality verbatim vs paraphrased - [x] Review integration: ADD_COMMENTS plan with two paired CommentSpecs per contradiction; NeedIngestResponse precheck; precedence vs math intent pinned - [x] Question integration: all three capabilities wired into smart-model toolset; `find_contradictions` returns formatted report text - [x] ChunkedMapper standalone: slicing, multi-chunk ordering, worker failures, timeouts, cancellation drain, semaphore saturation - [x] ChunkedReasoner regression: all pre-existing tests pass unchanged after the internal split ## Relationship to closed #6304 #6304 was closed in favour of this PR. The closed PR predated #6314 and modelled the agent as a Java-orchestrated two-round examine/deliberate flow with its own HTTP endpoint and a discriminated-union resume artifact. With #6314 making full ordered page text available to the engine via `DocumentService.read_pages`, none of that is needed. Net effect: drop ~600 lines of Java, drop the two-round handshake, drop the `ToolReportArtifact` lift, while ending up with a more scalable agent (chunk-based instead of page-based extraction; tested to ChunkedReasoner-equivalent scale).
This commit is contained in:
ConnorYoh
@@ -29,6 +29,12 @@ from .common import (
|
||||
format_conversation_history,
|
||||
format_file_names,
|
||||
)
|
||||
from .contradiction import (
|
||||
Claim,
|
||||
Contradiction,
|
||||
ContradictionReport,
|
||||
ContradictionSeverity,
|
||||
)
|
||||
from .documents import (
|
||||
DeleteDocumentResponse,
|
||||
IngestDocumentRequest,
|
||||
@@ -88,6 +94,7 @@ from .pdf_questions import (
|
||||
PdfQuestionResponse,
|
||||
PdfQuestionTerminalResponse,
|
||||
)
|
||||
from .pdf_review import PdfReviewOrchestrateResponse
|
||||
from .pdf_to_markdown import (
|
||||
LayoutFragment,
|
||||
LayoutLine,
|
||||
@@ -122,8 +129,12 @@ __all__ = [
|
||||
"AiToolAgentStep",
|
||||
"ArtifactKind",
|
||||
"CannotContinueExecutionAction",
|
||||
"Claim",
|
||||
"CommentSpec",
|
||||
"CompletedExecutionAction",
|
||||
"Contradiction",
|
||||
"ContradictionReport",
|
||||
"ContradictionSeverity",
|
||||
"ConversationMessage",
|
||||
"DeleteDocumentResponse",
|
||||
"PdfToMarkdownCannotDoResponse",
|
||||
@@ -178,6 +189,7 @@ __all__ = [
|
||||
"PdfQuestionRequest",
|
||||
"PdfQuestionResponse",
|
||||
"PdfQuestionTerminalResponse",
|
||||
"PdfReviewOrchestrateResponse",
|
||||
"PdfTextSelection",
|
||||
"ProgressEvent",
|
||||
"Requisition",
|
||||
|
||||
@@ -0,0 +1,151 @@
|
||||
"""Contradiction Agent — Python-only contract models.
|
||||
|
||||
The contradiction agent runs entirely inside the engine: there is no Java
|
||||
counterpart, no HTTP endpoint, and no discriminated-union resume artifact.
|
||||
These types are consumed by ``PdfReviewAgent`` (which produces sticky-note
|
||||
comment specs) and by ``ContradictionCapability`` (which formats the
|
||||
report as a tool-call payload for the smart model).
|
||||
|
||||
Page numbers are 1-indexed to match :class:`stirling.contracts.documents.Page`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from enum import StrEnum
|
||||
from typing import Literal
|
||||
|
||||
from pydantic import Field
|
||||
|
||||
from stirling.models import ApiModel
|
||||
|
||||
__all__ = [
|
||||
"Claim",
|
||||
"ClaimPolarity",
|
||||
"Contradiction",
|
||||
"ContradictionReport",
|
||||
"ContradictionSeverity",
|
||||
]
|
||||
|
||||
|
||||
# Shared type alias for the polarity field. Spelled out once here so the
|
||||
# detector's internal LLM-output schema and the public Claim contract stay
|
||||
# in sync — adding a new polarity requires touching one place.
|
||||
ClaimPolarity = Literal["assert", "deny", "recommend", "reject", "neutral"]
|
||||
|
||||
|
||||
class ContradictionSeverity(StrEnum):
|
||||
"""Severity of a textual contradiction.
|
||||
|
||||
``ERROR``: definite logical contradiction (the two claims cannot both be true).
|
||||
``WARNING``: plausible tension; possible paraphrase, hedging, or
|
||||
context-dependent reading.
|
||||
"""
|
||||
|
||||
ERROR = "error"
|
||||
WARNING = "warning"
|
||||
|
||||
|
||||
class Claim(ApiModel):
|
||||
"""A single atomic factual claim extracted from a page.
|
||||
|
||||
``page`` is 1-indexed (matches :class:`Page.page_number`). The
|
||||
``anchor_quality`` flag records whether ``quote`` was located
|
||||
verbatim in the declared page's text — verbatim claims can be
|
||||
placed by anchor text; paraphrased claims fall back to deterministic
|
||||
margin geometry in the review-comment builder.
|
||||
"""
|
||||
|
||||
page: int = Field(ge=1, description="1-indexed page number where the claim was found.")
|
||||
subject: str = Field(
|
||||
min_length=1,
|
||||
description="Short noun phrase naming what the claim is about (e.g. 'project deadline').",
|
||||
)
|
||||
polarity: ClaimPolarity = Field(
|
||||
description="Stance the claim takes toward the subject.",
|
||||
)
|
||||
text: str = Field(
|
||||
min_length=1,
|
||||
description="One-sentence paraphrase of the claim in the document's language.",
|
||||
)
|
||||
quote: str = Field(
|
||||
min_length=1,
|
||||
max_length=400,
|
||||
description="Verbatim excerpt from the page (typically <= 400 chars).",
|
||||
)
|
||||
anchor_quality: Literal["verbatim", "paraphrased"] = Field(
|
||||
default="verbatim",
|
||||
description=(
|
||||
"Whether the ``quote`` was located as a substring inside the declared "
|
||||
"page's text. ``verbatim`` claims can be anchored by text search; "
|
||||
"``paraphrased`` claims fall back to margin-geometry placement."
|
||||
),
|
||||
)
|
||||
file_name: str | None = Field(
|
||||
default=None,
|
||||
description=(
|
||||
"Name of the source file this claim was extracted from. Required for "
|
||||
"disambiguating claims when the detector audits multiple PDFs that "
|
||||
"share page numbers; ``None`` is acceptable for single-file audits "
|
||||
"where the answer is unambiguous."
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
class Contradiction(ApiModel):
|
||||
"""Two claims about the same subject that cannot both be true."""
|
||||
|
||||
subject: str = Field(min_length=1, description="Canonical subject shared by both claims.")
|
||||
claim1: Claim
|
||||
claim2: Claim
|
||||
explanation: str = Field(
|
||||
min_length=1,
|
||||
description="One-sentence explanation of why the claims conflict.",
|
||||
)
|
||||
severity: ContradictionSeverity
|
||||
|
||||
@property
|
||||
def page1(self) -> int:
|
||||
"""Lower-numbered page of the pair."""
|
||||
return min(self.claim1.page, self.claim2.page)
|
||||
|
||||
@property
|
||||
def page2(self) -> int:
|
||||
"""Higher-numbered page of the pair."""
|
||||
return max(self.claim1.page, self.claim2.page)
|
||||
|
||||
|
||||
class ContradictionReport(ApiModel):
|
||||
"""Output of :meth:`ContradictionDetector.detect`.
|
||||
|
||||
Lives entirely inside the engine — no Java counterpart. The review
|
||||
agent projects this into sticky-note ``CommentSpec`` pairs; the
|
||||
question agent's capability formats it into notes-style text for
|
||||
the smart model.
|
||||
"""
|
||||
|
||||
contradictions: list[Contradiction] = Field(default_factory=list)
|
||||
pages_examined: list[int] = Field(
|
||||
default_factory=list,
|
||||
description=(
|
||||
"1-indexed pages whose extractor pass ran, regardless of whether "
|
||||
"any claims were produced. Pages whose extraction failed "
|
||||
"(chunk-level timeout or crash) are excluded. Multi-file audits "
|
||||
"may show duplicate page numbers — page 1 from report.pdf and "
|
||||
"page 1 from memo.pdf are distinct pages and both count. Per-file "
|
||||
"attribution lives on each ``Claim.file_name``."
|
||||
),
|
||||
)
|
||||
clean: bool = Field(
|
||||
description="True iff no ERROR-severity contradictions were found.",
|
||||
)
|
||||
summary: str = Field(
|
||||
description="One or two neutral sentences summarising the audit outcome.",
|
||||
)
|
||||
|
||||
@property
|
||||
def error_count(self) -> int:
|
||||
return sum(1 for c in self.contradictions if c.severity == ContradictionSeverity.ERROR)
|
||||
|
||||
@property
|
||||
def warning_count(self) -> int:
|
||||
return sum(1 for c in self.contradictions if c.severity == ContradictionSeverity.WARNING)
|
||||
@@ -0,0 +1,21 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Annotated
|
||||
|
||||
from pydantic import Field
|
||||
|
||||
from .common import NeedIngestResponse
|
||||
from .pdf_edit import EditPlanResponse
|
||||
|
||||
# Mirrors :data:`PdfQuestionOrchestrateResponse` for parity with the
|
||||
# question agent. ``PdfReviewAgent.orchestrate`` either emits the
|
||||
# multi-step plan it wants Java to run (review → add-comments) or asks
|
||||
# Java to ingest the files first via :class:`NeedIngestResponse`.
|
||||
#
|
||||
# The discriminated union on ``outcome`` keeps the wire format honest:
|
||||
# Java sees a single `outcome` field and routes on its value, exactly
|
||||
# as it does for the question delegate.
|
||||
type PdfReviewOrchestrateResponse = Annotated[
|
||||
EditPlanResponse | NeedIngestResponse,
|
||||
Field(discriminator="outcome"),
|
||||
]
|
||||
Reference in New Issue
Block a user