mirror of
https://github.com/arsvendg/Stirling-PDF.git
synced 2026-07-14 18:44:05 +02:00
ConnorYohandGitHub
017c8d59fa
## Summary Adds a new AI specialist that finds **textual contradictions** across one or more PDFs — conflicting claims, recommendations, points of view, contested facts — built entirely in Python on top of the new `DocumentService` + `ChunkedReasoner` stack from #6314. Replaces the closed #6304, which was started before #6314 landed and therefore over-engineered (Java orchestrator, two-round handshake, resume artifact, discriminated-union lift). Two commits: 1. **`refactor(engine): extract ChunkedMapper[T] from ChunkedReasoner`** — pure refactor, public API of ChunkedReasoner unchanged. New `ChunkedMapper[T: BaseModel]` is a generic parallel-chunk primitive (slicing, semaphore, time-bounded extraction, cancellation drain, progress events) that's now a peer to ChunkedReasoner rather than locked inside it. The compression loop stays on ChunkedReasoner where it belongs. 2. **`feat(ai): add Contradiction Agent on ChunkedMapper`** — the agent itself, plus integrations into `PdfReviewAgent` and `PdfQuestionAgent`. ## Architecture - **Python-only.** No Java code. No `AgentToolId.CONTRADICTION_AGENT`. No dedicated HTTP endpoint. No resume artifact, no discriminated-union lift in `contracts/common.py`. Detector runs inside the Python engine and the Python engine alone. - **Review path** (`PdfReviewAgent`): a new `ContradictionIntentClassifier` fires on contradiction-flavoured prompts; agent runs detection synchronously and emits a single `EditPlanResponse(steps=[ADD_COMMENTS])`. Single-turn flow — no resume. - **Question path** (`PdfQuestionAgent`): a new `ContradictionCapability` joins `RagCapability` and `WholeDocReaderCapability` in the smart-model toolset, exposing `find_contradictions(query)`. The smart model picks it from the toolset alongside `search_knowledge` and `read_full_document`. ## Inside `ContradictionDetector.detect()` 1. `DocumentService.read_pages(file_id)` → ordered `list[Page]`. 2. `ChunkedMapper[_ExtractedClaims].map_pages(...)` — char-budgeted multi-page slicing; each slice runs the claim-extractor LLM in parallel under a semaphore. 3. Page-traceability: the extractor returns `_ExtractedClaim.page` (which `[Page N]` marker the claim came from). The wrapper validates `page ∈ chunk.pages`; if not, mechanical fallback searches the chunk's page text for the verbatim quote and reassigns. If still no match, drop the claim. 4. `Claim.anchor_quality: Literal[\"verbatim\", \"paraphrased\"]` is set by a substring check against the declared page's text. Verbatim quotes feed `anchor_text` for snap-to-quote add-comments placement; paraphrased ones fall back to margin geometry. 5. Subject canonicalisation: ONE fast-model LLM call collapses synonyms across the document. Fails open to lexical bucketing. 6. Pre-filters: drop identical-quote pairs; drop same-page same-polarity paraphrases. 7. Per-bucket pair detection in parallel (separate semaphore, cap 5). Buckets > 12 claims chunk into windows of 12 with overlap 2; pairs deduped across overlapping windows by frozen `(i, j)` index pair. 8. Summary fast-model call with fallback string on error. ## Prompt-injection hardening Every prompt that interpolates user-supplied or PDF-extracted text wraps content in `<user_message>` / `<verdict>` / `<content>` tags with an explicit SECURITY preamble instructing the model to treat tagged content as data only. ## Limitations - **Combined math + contradiction intent**: when both intent classifiers fire on the same prompt, contradiction takes precedence and the math intent is silently dropped. Documented in the Review module docstring and pinned by `test_review_integration.py::test_contradiction_precedence_over_math`. - **Cross-window contradiction reach**: within a subject bucket, pairs more than ~10 claim indices apart in the same chunked window may be missed by the overlap-2 strategy. Documented in `test_detector.py`. Acceptable for v1. ## Settings (engine/src/stirling/config/settings.py) ```python contradiction_detect_concurrency = 5 # per-bucket detector semaphore contradiction_bucket_chunk_size = 12 # max claims per detector call contradiction_bucket_chunk_overlap = 2 # overlap for >threshold buckets ``` `chars_per_slice` and extraction concurrency are reused from the existing `chunked_reasoner_*` settings. ## Test plan - [x] `uv run pytest tests/ -v` — **245/245 pass** (210 pre-existing + 35 new) - [x] `uv run ruff check src/ tests/` — clean - [x] `uv run pyright src/stirling/agents/contradiction/ src/stirling/contracts/contradiction.py` — 0 errors - [x] `./gradlew :proprietary:test` — green; no Java was touched, but verified untouched - [x] Page-traceability tests cover: valid page kept, hallucinated page dropped, mechanical-reassign on misattribution, anchor-quality verbatim vs paraphrased - [x] Review integration: ADD_COMMENTS plan with two paired CommentSpecs per contradiction; NeedIngestResponse precheck; precedence vs math intent pinned - [x] Question integration: all three capabilities wired into smart-model toolset; `find_contradictions` returns formatted report text - [x] ChunkedMapper standalone: slicing, multi-chunk ordering, worker failures, timeouts, cancellation drain, semaphore saturation - [x] ChunkedReasoner regression: all pre-existing tests pass unchanged after the internal split ## Relationship to closed #6304 #6304 was closed in favour of this PR. The closed PR predated #6314 and modelled the agent as a Java-orchestrated two-round examine/deliberate flow with its own HTTP endpoint and a discriminated-union resume artifact. With #6314 making full ordered page text available to the engine via `DocumentService.read_pages`, none of that is needed. Net effect: drop ~600 lines of Java, drop the two-round handshake, drop the `ToolReportArtifact` lift, while ending up with a more scalable agent (chunk-based instead of page-based extraction; tested to ChunkedReasoner-equivalent scale).
58 lines
1.6 KiB
Python
58 lines
1.6 KiB
Python
from __future__ import annotations
|
|
|
|
from collections.abc import Iterator
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
from stirling.config import AppSettings, RagBackend, load_settings
|
|
from stirling.services import build_runtime
|
|
from stirling.services.runtime import AppRuntime
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def clear_settings_cache() -> Iterator[None]:
|
|
load_settings.cache_clear()
|
|
yield
|
|
load_settings.cache_clear()
|
|
|
|
|
|
def build_app_settings() -> AppSettings:
|
|
return AppSettings(
|
|
smart_model_name="test",
|
|
fast_model_name="test",
|
|
smart_model_max_tokens=8192,
|
|
fast_model_max_tokens=2048,
|
|
rag_backend=RagBackend.SQLITE,
|
|
rag_embedding_model="voyageai:voyage-4",
|
|
rag_store_path=Path(":memory:"),
|
|
rag_pgvector_dsn="",
|
|
rag_chunk_size=512,
|
|
rag_chunk_overlap=64,
|
|
rag_default_top_k=5,
|
|
rag_max_searches=5,
|
|
chunked_reasoner_chars_per_slice=16_000,
|
|
chunked_reasoner_concurrency=10,
|
|
chunked_reasoner_notes_char_budget=250_000,
|
|
chunked_reasoner_worker_timeout_seconds=60.0,
|
|
contradiction_detect_concurrency=5,
|
|
contradiction_bucket_chunk_size=12,
|
|
contradiction_bucket_chunk_overlap=2,
|
|
contradiction_canonicaliser_batch_size=500,
|
|
max_pages=200,
|
|
max_characters=200_000,
|
|
posthog_enabled=False,
|
|
posthog_api_key="",
|
|
posthog_host="https://eu.i.posthog.com",
|
|
)
|
|
|
|
|
|
@pytest.fixture
|
|
def app_settings() -> AppSettings:
|
|
return build_app_settings()
|
|
|
|
|
|
@pytest.fixture
|
|
def runtime(app_settings: AppSettings) -> AppRuntime:
|
|
return build_runtime(app_settings)
|