Add ability for Stirling engine to reason across large documents (#6314)

# Description of Changes
Adds storage in the database for full document content alongside the RAG
content (and changes the service to `DocumentService` instead of
`RagService`). Then adds a generic capability that should be usable by
any agent (currently just used by the Question Agent) which allows the
agent to pull out the full contents of the doc, chunks it into various
sections that will fit in the context window, and then processes them in
parallel to create an intermediate result, and then processes the
intermediate result into a final answer. It will re-chunk as many times
as necessary to get the content small enough for the actual answer to be
analysed (I've tested on PDFs ~3500 pages long, which is well above the
context limit and requires maybe 3 rounds of compression to get an
answer).

The new full doc analysis stuff is heavier than the RAG lookup so both
remain. The agents should use RAG for targeted info and the chunked
reasoner for info that requires reading the full doc.

engine/.env

@@ -37,6 +37,22 @@ STIRLING_RAG_TOP_K=20
 # rather than chain more searches.
 STIRLING_RAG_MAX_SEARCHES=5
 
+# Chunked reasoner settings: how big each per-worker slice is (in characters),
+# how many workers may run in parallel against the fast model, and how long
+# any single worker is allowed to wait for a response before being abandoned.
+# Worker timeouts protect gather_notes from upstream model stalls (which
+# otherwise hang at the provider's ~10 minute HTTP default); the affected
+# slice is dropped and the rest of the document still answers.
+STIRLING_CHUNKED_REASONER_CHARS_PER_SLICE=16000
+STIRLING_CHUNKED_REASONER_CONCURRENCY=10
+STIRLING_CHUNKED_REASONER_WORKER_TIMEOUT_SECONDS=60
+
+# When the rendered slice notes would exceed this many characters, the
+# reasoner folds them hierarchically with fast-model calls until they fit.
+# This keeps the synthesis prompt under the model's context limit on long
+# documents (a 3000-page novel produces ~900k chars of raw notes).
+STIRLING_CHUNKED_REASONER_NOTES_CHAR_BUDGET=250000
+
 # Upper bounds on PDF page text the engine will request per extraction round.
 STIRLING_MAX_PAGES=200
 STIRLING_MAX_CHARACTERS=200000
@@ -51,3 +67,8 @@ STIRLING_LOG_LEVEL=INFO
 
 # Path to log file. Rolls daily, keeps 1 backup. Leave empty for console only.
 STIRLING_LOG_FILE=
+
+# Set true to log every outgoing httpx / Anthropic SDK request with timing.
+# Use when diagnosing worker stalls: a hung call shows a "Request" line with
+# no matching "Response" line. Noisy; leave off in normal use.
+STIRLING_HTTP_DEBUG=false