Add ability for Stirling engine to reason across large documents (#6314)

# Description of Changes
Adds storage in the database for full document content alongside the RAG
content (and changes the service to `DocumentService` instead of
`RagService`). Then adds a generic capability that should be usable by
any agent (currently just used by the Question Agent) which allows the
agent to pull out the full contents of the doc, chunks it into various
sections that will fit in the context window, and then processes them in
parallel to create an intermediate result, and then processes the
intermediate result into a final answer. It will re-chunk as many times
as necessary to get the content small enough for the actual answer to be
analysed (I've tested on PDFs ~3500 pages long, which is well above the
context limit and requires maybe 3 rounds of compression to get an
answer).

The new full doc analysis stuff is heavier than the RAG lookup so both
remain. The agents should use RAG for targeted info and the chunked
reasoner for info that requires reading the full doc.

engine/src/stirling/documents/README.md

@@ -0,0 +1,102 @@
+# Document Storage
+
+The `documents` package owns all stored content for a document under a single
+`collection` (file id):
+
+* **Vector chunks** — small, embedded chunks for RAG-style retrieval.
+* **Ordered pages** — the original page text retained in document order, used
+  for whole-document reading.
+
+Both representations are populated by a single `ingest()` call and removed
+together by `delete_collection()`.
+
+## Adding RAG to an Agent
+
+```python
+from pydantic_ai import Agent
+
+from stirling.services import AppRuntime
+
+class MyAgent:
+    def __init__(self, runtime: AppRuntime) -> None:
+        rag = runtime.rag_capability
+        self.agent = Agent(
+            model=runtime.smart_model,
+            system_prompt="Your prompt here...",
+            instructions=rag.instructions,
+            toolsets=[rag.toolset],
+        )
+```
+
+That's it. The agent gets a `search_knowledge` tool it can call autonomously.
+
+## Scoping to Specific Collections
+
+Collections are named buckets of indexed documents - think folders. By default
+an agent searches everything in the store. Pass `collections=` to restrict it
+to only the docs indexed under those names.
+
+```python
+from stirling.documents import RagCapability
+
+# Only searches docs indexed under "company-docs"
+scoped = RagCapability(runtime.documents, collections=["company-docs"], top_k=3)
+
+# Searches multiple collections
+multi = RagCapability(runtime.documents, collections=["company-docs", "product-specs"])
+
+# No collections arg = searches all collections in the store
+everything = RagCapability(runtime.documents)
+```
+
+## Config
+
+Non-secret defaults live in the committed `engine/.env`:
+
+```
+STIRLING_RAG_BACKEND=sqlite              # or "pgvector"
+STIRLING_RAG_EMBEDDING_MODEL=voyageai:voyage-4
+STIRLING_RAG_STORE_PATH=data/rag.db      # used when backend=sqlite
+STIRLING_RAG_PGVECTOR_DSN=               # used when backend=pgvector
+STIRLING_RAG_CHUNK_SIZE=512
+STIRLING_RAG_CHUNK_OVERLAP=64
+STIRLING_RAG_TOP_K=5
+```
+
+Provider credentials (and any local overrides) go in the uncommitted
+`engine/.env.local`:
+
+```
+VOYAGE_API_KEY=your-key
+```
+
+## Backends
+
+**`sqlite`** - Embedded sqlite-vec. Single `.db` file, zero ops. Ideal for dev
+and self-hosted deployments.
+
+**`pgvector`** - External PostgreSQL with the `vector` extension. Point
+`STIRLING_RAG_PGVECTOR_DSN` at your Postgres instance.
+
+Both backends implement the same `DocumentStore` interface, so agents and the
+service work identically regardless of which you pick.
+
+For a self-hosted embedding server (e.g. Ollama, TEI, vLLM) set the model
+string accordingly and point at the server via its native env var:
+
+```
+# Ollama running on another machine
+STIRLING_RAG_EMBEDDING_MODEL=ollama:nomic-embed-text
+OLLAMA_HOST=http://192.168.1.50:11434
+
+# Any OpenAI-compatible embedding server
+STIRLING_RAG_EMBEDDING_MODEL=openai:my-model
+OPENAI_BASE_URL=http://192.168.1.50:8080/v1
+```
+
+## API Endpoints
+
+| Method | Endpoint | Purpose |
+|--------|----------|---------|
+| POST | `/api/v1/documents` | Replace-ingest a document's pages |
+| DELETE | `/api/v1/documents/{document_id}` | Delete a document's stored content |