2 Commits

Include renames

Author	SHA1	Message	Date
ConnorYohandGitHub	84aca12055	PR-S4: shadow-mode hardening (review follow-ups) (#6523) ... ## What this PR does Bundles the **low-risk polish items** from the [multi-agent review of #6519](https://github.com/Stirling-Tools/Stirling-PDF/pull/6519). Each change is independent, mechanical, and ships with focused unit-test coverage. The medium-severity items (\`?async=true\` OUTPUT recording, JSON-consumes endpoint coverage, SpringBootTest harness) are tracked separately in [\`notes/PAYG_DESIGN.md\` §7.5 PR-S4](https://github.com/Stirling-Tools/Stirling-PDF/blob/payg-s4-hardening/notes/PAYG_DESIGN.md) — they need design decisions + bigger infrastructure work, so this PR sticks to the mechanical wins. Stacked on #6519. When that merges to main, this rebases cleanly — no code changes. ## Changes | Area | What | Why | |---|---|---| | **\`tool_id\` becomes route pattern** | \`PaygChargeInterceptor.resolveToolId()\` prefers \`HandlerMapping.BEST_MATCHING_PATTERN_ATTRIBUTE\` over \`request.getRequestURI()\`. Truncates to 128 + WARN log + \`payg.filter.errors\` increment when truncation fires. | Audit rollups aggregate by endpoint instead of by every individual request's path-variable / matrix-param variant. Silent truncation now louder. | | **Direct PDF magic-byte check** | \`PaygOutputExtractor.extract()\` magic-checks the body even for direct \`application/pdf\` responses. | Asymmetric with the ZIP-entry path which always magic-checks. A tool that emits \`application/pdf\` for a JSON / HTML payload would otherwise pollute \`job_artifact_hash\`. | | **DESKTOP_APP detection** | \`X-Stirling-Client: desktop\` header → \`JobSource.DESKTOP_APP\`. | The enum value was unreachable from \`determineSource()\`; Tauri shell traffic was mis-classified as WEB. No anti-spoof — V12 step limits are identical for WEB/DESKTOP_APP so the worst-case abuse value is zero today. | | **\`max-bytes\` sensible default** | 500 MiB instead of \`null\` (unbounded). | Covers the largest realistic Stirling responses (full split-to-ZIP on a 1000-page document) while preventing pathological cases from tying up the interceptor for minutes. Set to \`null\` to disable. | | **\`BufferedOutputStream\` for spill** | Wraps the spill \`OutputStream\` in 64 KiB \`BufferedOutputStream\`. | Previously every Tomcat chunk (default 8 KiB) was a separate syscall. Big spilled responses get a syscall-bound speedup. | | **Duration timer per phase** | \`payg.filter.duration\` tagged \`phase=preHandle\` vs \`phase=afterCompletion\`. | Two distinct latency distributions were blended into one histogram; hard to alert on. | ## Tests | Test | What it covers | |---|---| | \`PaygOutputExtractorTest.pdfContentType_butBodyMissingPdfMagic_returnsEmpty\` | New direct-PDF magic-byte gate. | | \`PaygChargeInterceptorTest.preHandle_desktopClientHeader_setsJobSourceDesktopApp\` | New \`X-Stirling-Client: desktop\` → DESKTOP_APP path. | | \`PaygChargeInterceptorTest.preHandle_toolId_prefersBestMatchingPattern\` | Route pattern wins over URI when both are set. | | \`PaygChargeInterceptorTest.preHandle_toolId_truncatesAndCountsWhenLongerThan128\` | Oversized values truncate + increment errors counter. | Full saas suite green (210 tests), coverage targets met. ## What's NOT in this PR (deliberately) - **\`?async=true\` OUTPUT recording.** The JobExecutorService returns a synchronous \`JobResponse{jobId}\` body before the async tool actually runs; \`afterCompletion\` fires too early. Needs a design decision: short-circuit PAYG when \`async=true\` OR hook into \`TaskManager\` completion. Tracked in PR-S4 design doc. - **JSON-consumes endpoint coverage.** The \`MultipartHttpServletRequest\` cast skips endpoints with \`consumes = APPLICATION_JSON_VALUE\` (e.g. \`ConvertPdfJsonController.exportPartialPdf\`). Fix is either extract a request-body hash for JSON or add a CI lint forbidding non-multipart \`@AutoJobPostMapping\`. Design discussion needed. - **SpringBootTest harness for filter + interceptor wiring.** Saas module doesn't have one yet. Separate work — PR-S3 takes a different approach (docker-compose + Behave); a SpringBootTest layer would be additive in-process coverage. These are tracked in \`notes/PAYG_DESIGN.md §7.5\` so they don't slip. ## Tracked in \`notes/PAYG_DESIGN.md\` §7.5 PR-S4.	2026-06-10 09:08:43 +00:00
ConnorYohandGitHub	ff96a80947	PAYG B-3 / S-3: cucumber suite for shadow-mode flows + CI workflow (#6522) ... ## What this PR is End-to-end cucumber coverage for the PAYG shadow charging engine (the filter + interceptor stack from #6519), wired into CI via a new `docker-compose-tests-saas.yml` workflow that runs only on PAYG-touching PRs. Stacked on #6519. ## Automated scenarios (run by `docker-compose-tests-saas.yml`) See [`testing/cucumber/features/payg/shadow_charges.feature`](../tree/payg-s3-cucumber/testing/cucumber/features/payg/shadow_charges.feature): | Scenario | Validates | |---|---| | First tool call writes a CHARGED row | Filter + interceptor fire end-to-end | | Lineage join — second call on output | `JobService.joinOrOpen` matching; no new shadow row | | 4xx leaves the row CHARGED | "Customer paid for the attempt" semantics | | ZIP-returning tool records per-PDF OUTPUT | `PaygOutputExtractor` unpacks + records signatures | | Multi-file input writes a single shadow row | Multi-input group sizing | | `X-Stirling-Automation` sets PIPELINE source | Header → `JobSource` detection | All 6 run locally via `./testing/test-payg.sh` and will run on CI for any PR that touches `app/saas/**`, the PAYG cucumber features, the saas compose stack, or the workflow itself. ## Manual-only scenarios — documented in design doc, not in this suite Two parts of the shadow engine are deliberately not automated; the engine paths are unit-tested in `PaygChargeInterceptorTest.afterCompletion_5xx_opened_*`, and the manual procedures (which require a temporary throw endpoint or a container restart with a flag flipped) live in [`notes/PAYG_DESIGN.md` §7.5.2 "PAYG cucumber: manual-only scenarios"](../tree/payg-s3-cucumber/notes/PAYG_DESIGN.md). - **5xx first-step failure → REFUNDED + CLOSED.** No reliably-5xx-ing endpoint exists; manual procedure adds a throw endpoint, runs, asserts, removes. - **Kill-switch (`PAYG_FILTER_ENABLED=false`).** Needs a container restart mid-suite; manual procedure tears down, flips env, brings up, asserts zero shadow rows. If either gets a hot-reload path (test-only throw endpoint shipped behind a profile gate, or admin endpoint for the kill switch), automate it in a follow-up and drop the manual procedure. ## CI workflow `.github/workflows/docker-compose-tests-saas.yml` (new) — self-contained, not wired into `build.yml`'s `files-changed` matrix so the saas-cucumber job fails and succeeds independently. Triggers only on PAYG-relevant paths. No JaCoCo coverage in v1 (saas compose doesn't have the coverage override; can add later). ## Test infrastructure (recap) - **`testing/compose/docker-compose-saas.yml`** — Stirling-PDF backend with `STIRLING_FLAVOR=saas` + Postgres holding the `stirling_pdf` schema. Supabase JWT auto-config disabled; API-key auth via `SECURITY_CUSTOMGLOBALAPIKEY` is the live path the cucumber tests exercise. - **`testing/compose/payg/saas-init.sql`** + **`saas-seed.sql`** — schema bootstrap + idempotent seed (team / user / wallet_policy). - **`testing/cucumber/features/payg/shadow_charges.feature`** — the 6 scenarios above. - **`testing/cucumber/features/steps/payg_step_definitions.py`** — step defs using `requests` (HTTP) + `psycopg` (direct DB inspection). Direct DB reads are deliberate — we want to see the filter's side effects, not relay them through another API layer. - **`testing/test-payg.sh`** — companion runner to `testing/test.sh`. Brings up the saas compose, waits for health, seeds, runs behave, tears down. - **`behave.ini`** excludes `features/payg` from the default behave run (the saas-cucumber CI job invokes it explicitly). ## Why a separate harness from `testing/test.sh` The existing `test.sh` covers the proprietary-flavour stack (no PAYG tables, no saas profile). Coupling two CI matrices that fail and succeed independently into one script is asking for trouble. Keep the saas-cucumber job focused on its own concerns; once the harness is mature, the wider team can decide whether to merge them. ## Tracked in `notes/PAYG_DESIGN.md` §7.5 (PR-S3) + §7.5.2 (manual scenarios).	2026-06-09 14:47:40 +00:00