arsvendg/Stirling-PDF
1
0
Fork 0
mirror of https://github.com/arsvendg/Stirling-PDF.git synced 2026-07-14 10:34:06 +02:00
Code Issues Actions 4 Packages Projects Releases 1 Wiki Activity
5,217 Commits 1 Branch 1 Tag
1e739b6f6f0273ef877b2ddf84f08815441b8bbc
Commit Graph
468 Commits
Author SHA1 Message Date
Anthony StirlingandGitHub 1e739b6f6f SaaS-aware API landing page (#6585) 
# Description of Changes

OLD  (and still current in selfhosted)
<img width="610" height="869" alt="image"
src="https://github.com/user-attachments/assets/f8019298-b4ee-4a68-b928-a9746b64ac1c"
/>


New (in SaaS mode)

<img width="635" height="876" alt="image"
src="https://github.com/user-attachments/assets/6ee4946f-1d7b-42ec-a6f7-75e85739e348"
/>


---

## Checklist

### General

- [ ] I have read the [Contribution
Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md)
- [ ] I have read the [Stirling-PDF Developer
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md)
(if applicable)
- [ ] I have read the [How to add new languages to
Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md)
(if applicable)
- [ ] I have performed a self-review of my own code
- [ ] My changes generate no new warnings

### Documentation

- [ ] I have updated relevant docs on [Stirling-PDF's doc
repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/)
(if functionality has heavily changed)
- [ ] I have read the section [Add New Translation
Tags](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md#add-new-translation-tags)
(for new translation tags only)

### Translations (if applicable)

- [ ] I ran
[`scripts/counter_translation.py`](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/docs/counter_translation.md)

### UI Changes (if applicable)

- [ ] Screenshots or videos demonstrating the UI changes are attached
(e.g., as comments or direct attachments in the PR)

### Testing (if applicable)

- [ ] I have run `task check` to verify linters, typechecks, and tests
pass
- [ ] I have tested my changes locally. Refer to the [Testing
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md#7-testing)
for more details.
 2026-06-09 14:49:20 +00:00
ConnorYohandGitHub 98967bfa86 PAYG: V14 + V15 — subscription_id, free-tier, RPCs, audit logs (#6532) 
## Summary

Two Flyway migrations + matching JPA entity updates. **Part 1 of 2** in
the Stripe/Supabase wire-up (PR-SB-1 in
`payg-stripe-supabase-plan.html`); the companion SaaS PR carries the
twin Supabase migrations + new edge functions.

### V14 — payg_subscription_state.sql

- `payg_team_extensions.payg_subscription_id` — the single switch that
decides whether a team is billed. NULL = free-tier or block; NOT NULL =
post Stripe meter events.
- `pricing_policy.free_tier_units_per_cycle` — per-policy free allowance
before a card is required. Default 0.
- `payg_link_subscription(team_id, customer_id, sub_id)` RPC —
idempotent.
- `payg_unlink_subscription(team_id, reason)` RPC — called on
`subscription.deleted`.
- AFTER-INSERT trigger on `teams` so every new signup gets a
`payg_team_extensions` sidecar row from creation.
- Backfill for existing teams without a sidecar row.
- RLS: SELECT permissive (any team member), UPDATE restricted to LEADER.
Service-role bypasses (backend reads + day-1 migration writes).

### V15 — payg_audit_logs.sql

- `payg_meter_event_log` — backend audit of every Stripe meter event
POST attempt (idempotency-key UNIQUE; index on unposted rows for nightly
reconcile).
- `payg_subscription_change_log` — written by V14 RPCs on every
link/unlink.

### Entity updates

- `PaygTeamExtensions.paygSubscriptionId` — read-only field; RPC
functions are the only writers.
- `PricingPolicy.freeTierUnitsPerCycle` — read by upcoming
`PaygTeamUsageService` (PR-SB-4).

### Behaviour change

**None yet.** The columns + functions sit unused until PR-SB-4 wires
`PaygMeterReportingService` and the free-tier gate into
`JobChargeService`. This PR is pure schema + JPA wiring.

## Test plan

- [x] `./gradlew :saas:test` — BUILD SUCCESSFUL
- [x] Manual schema review: column types, FK directions, RLS scope
- [ ] Apply against v3-Supabase via `supabase db push` (after companion
SaaS PR merges)
- [ ] Smoke-test the trigger: `INSERT INTO teams(...)` → assert
`payg_team_extensions` row appears
- [ ] Smoke-test RPCs: SQL-only test of `payg_link_subscription` +
`payg_unlink_subscription` produces expected row + audit entries

## References

- `notes/PAYG_DESIGN.md` (revision note 2026-06-03)
- `payg-stripe-supabase-plan.html` §3.1 (RPC functions), §3.5 (RLS),
§3.10 (audit-log tables)
 2026-06-09 14:48:05 +00:00
ConnorYohandGitHub ff96a80947 PAYG B-3 / S-3: cucumber suite for shadow-mode flows + CI workflow (#6522) 
## What this PR is

End-to-end cucumber coverage for the PAYG shadow charging engine (the
filter + interceptor stack from #6519), wired into CI via a new
`docker-compose-tests-saas.yml` workflow that runs only on PAYG-touching
PRs.

Stacked on #6519.

## Automated scenarios (run by `docker-compose-tests-saas.yml`)

See
[`testing/cucumber/features/payg/shadow_charges.feature`](../tree/payg-s3-cucumber/testing/cucumber/features/payg/shadow_charges.feature):

| Scenario | Validates |
|---|---|
| First tool call writes a CHARGED row | Filter + interceptor fire
end-to-end |
| Lineage join — second call on output | `JobService.joinOrOpen`
matching; no new shadow row |
| 4xx leaves the row CHARGED | "Customer paid for the attempt" semantics
|
| ZIP-returning tool records per-PDF OUTPUT | `PaygOutputExtractor`
unpacks + records signatures |
| Multi-file input writes a single shadow row | Multi-input group sizing
|
| `X-Stirling-Automation` sets PIPELINE source | Header → `JobSource`
detection |

All 6 run locally via `./testing/test-payg.sh` and will run on CI for
any PR that touches `app/saas/**`, the PAYG cucumber features, the saas
compose stack, or the workflow itself.

## Manual-only scenarios — documented in design doc, not in this suite

Two parts of the shadow engine are deliberately not automated; the
engine paths are unit-tested in
`PaygChargeInterceptorTest.afterCompletion_5xx_opened_*`, and the manual
procedures (which require a temporary throw endpoint or a container
restart with a flag flipped) live in [`notes/PAYG_DESIGN.md` §7.5.2
"PAYG cucumber: manual-only
scenarios"](../tree/payg-s3-cucumber/notes/PAYG_DESIGN.md).

- **5xx first-step failure → REFUNDED + CLOSED.** No reliably-5xx-ing
endpoint exists; manual procedure adds a throw endpoint, runs, asserts,
removes.
- **Kill-switch (`PAYG_FILTER_ENABLED=false`).** Needs a container
restart mid-suite; manual procedure tears down, flips env, brings up,
asserts zero shadow rows.

If either gets a hot-reload path (test-only throw endpoint shipped
behind a profile gate, or admin endpoint for the kill switch), automate
it in a follow-up and drop the manual procedure.

## CI workflow

`.github/workflows/docker-compose-tests-saas.yml` (new) —
self-contained, not wired into `build.yml`'s `files-changed` matrix so
the saas-cucumber job fails and succeeds independently. Triggers only on
PAYG-relevant paths. No JaCoCo coverage in v1 (saas compose doesn't have
the coverage override; can add later).

## Test infrastructure (recap)

- **`testing/compose/docker-compose-saas.yml`** — Stirling-PDF backend
with `STIRLING_FLAVOR=saas` + Postgres holding the `stirling_pdf`
schema. Supabase JWT auto-config disabled; API-key auth via
`SECURITY_CUSTOMGLOBALAPIKEY` is the live path the cucumber tests
exercise.
- **`testing/compose/payg/saas-init.sql`** + **`saas-seed.sql`** —
schema bootstrap + idempotent seed (team / user / wallet_policy).
- **`testing/cucumber/features/payg/shadow_charges.feature`** — the 6
scenarios above.
- **`testing/cucumber/features/steps/payg_step_definitions.py`** — step
defs using `requests` (HTTP) + `psycopg` (direct DB inspection). Direct
DB reads are deliberate — we want to see the filter's side effects, not
relay them through another API layer.
- **`testing/test-payg.sh`** — companion runner to `testing/test.sh`.
Brings up the saas compose, waits for health, seeds, runs behave, tears
down.
- **`behave.ini`** excludes `features/payg` from the default behave run
(the saas-cucumber CI job invokes it explicitly).

## Why a separate harness from `testing/test.sh`

The existing `test.sh` covers the proprietary-flavour stack (no PAYG
tables, no saas profile). Coupling two CI matrices that fail and succeed
independently into one script is asking for trouble. Keep the
saas-cucumber job focused on its own concerns; once the harness is
mature, the wider team can decide whether to merge them.

## Tracked in

`notes/PAYG_DESIGN.md` §7.5 (PR-S3) + §7.5.2 (manual scenarios).
 2026-06-09 14:47:40 +00:00
Anthony StirlingandGitHub 800a411167 Hide endpoints (#6586) 
# Description of Changes

Hides the /api/v1/credits endpoints from the generated OpenAPI/Swagger
docs. The root GET /api/v1/credits and GET /api/v1/credits/usage now
carry @Hidden (the 8 admin credit endpoints were already hidden), so the
whole Credit Management controller is gone from the docs.

Adds a single global AI tag to the OpenAPI definition.


Why
We don't want the credits or AI endpoints surfaced in the public API
docs yet as they are not ready for public use, but we do want the AI
endpoints pre-grouped under one AI tag so they land cleanly when we
later un-hide them but dont clutter PDF APIs.

---

## Checklist

### General

- [ ] I have read the [Contribution
Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md)
- [ ] I have read the [Stirling-PDF Developer
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md)
(if applicable)
- [ ] I have read the [How to add new languages to
Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md)
(if applicable)
- [ ] I have performed a self-review of my own code
- [ ] My changes generate no new warnings

### Documentation

- [ ] I have updated relevant docs on [Stirling-PDF's doc
repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/)
(if functionality has heavily changed)
- [ ] I have read the section [Add New Translation
Tags](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md#add-new-translation-tags)
(for new translation tags only)

### Translations (if applicable)

- [ ] I ran
[`scripts/counter_translation.py`](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/docs/counter_translation.md)

### UI Changes (if applicable)

- [ ] Screenshots or videos demonstrating the UI changes are attached
(e.g., as comments or direct attachments in the PR)

### Testing (if applicable)

- [ ] I have run `task check` to verify linters, typechecks, and tests
pass
- [ ] I have tested my changes locally. Refer to the [Testing
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md#7-testing)
for more details.
 2026-06-09 13:18:36 +00:00
James BruntonandGitHub 51478e5051 Policies backend (#6527) 
# Description of Changes
Add a backend for running any multi-step PDF operations. This is
designed to be used for the upcoming Policies feature, along with
anything else that will require automated running of PDF operations,
like the Automate tool or Processing Folders.

The implementation is not complete. I've tried to get all the
infrastructure in there so that we can add in whichever triggers we need
in the future (like cron triggers or watching folders on disk) but
currently it just supports manual triggering of the policy.

The basis of this work was the operation running from the Stirling
Engine, which this PR removes in favour of this new system. The only
currently accessible frontend way to test this work is to ask the AI
chat to execute multiple operations on a PDF, but I've also extensively
tested with direct API calls to make sure that the policies work and
persist properly.
 2026-06-08 10:50:55 +00:00
Anthony StirlingandGitHub 69e62d8949 exclude unused Redis auto-config (#6547) 
fix(health): exclude unused Redis auto-config so default
/actuator/health stays UP

---

## Checklist

### General

- [ ] I have read the [Contribution
Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md)
- [ ] I have read the [Stirling-PDF Developer
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md)
(if applicable)
- [ ] I have read the [How to add new languages to
Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md)
(if applicable)
- [ ] I have performed a self-review of my own code
- [ ] My changes generate no new warnings

### Documentation

- [ ] I have updated relevant docs on [Stirling-PDF's doc
repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/)
(if functionality has heavily changed)
- [ ] I have read the section [Add New Translation
Tags](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md#add-new-translation-tags)
(for new translation tags only)

### Translations (if applicable)

- [ ] I ran
[`scripts/counter_translation.py`](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/docs/counter_translation.md)

### UI Changes (if applicable)

- [ ] Screenshots or videos demonstrating the UI changes are attached
(e.g., as comments or direct attachments in the PR)

### Testing (if applicable)

- [ ] I have run `task check` to verify linters, typechecks, and tests
pass
- [ ] I have tested my changes locally. Refer to the [Testing
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md#7-testing)
for more details.
 2026-06-08 10:41:38 +00:00
Anthony StirlingandGitHub d202c9c32f Minor: sanitize SVG (#6572) 
# Description of Changes

Sanitize SVG

---

## Checklist

### General

- [ ] I have read the [Contribution
Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md)
- [ ] I have read the [Stirling-PDF Developer
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md)
(if applicable)
- [ ] I have read the [How to add new languages to
Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md)
(if applicable)
- [ ] I have performed a self-review of my own code
- [ ] My changes generate no new warnings

### Documentation

- [ ] I have updated relevant docs on [Stirling-PDF's doc
repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/)
(if functionality has heavily changed)
- [ ] I have read the section [Add New Translation
Tags](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md#add-new-translation-tags)
(for new translation tags only)

### Translations (if applicable)

- [ ] I ran
[`scripts/counter_translation.py`](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/docs/counter_translation.md)

### UI Changes (if applicable)

- [ ] Screenshots or videos demonstrating the UI changes are attached
(e.g., as comments or direct attachments in the PR)

### Testing (if applicable)

- [ ] I have run `task check` to verify linters, typechecks, and tests
pass
- [ ] I have tested my changes locally. Refer to the [Testing
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md#7-testing)
for more details.
 2026-06-08 10:31:41 +00:00
briosandGitHub 0dff192281 perf(compression): add vite-plugin-compression for gzip and Brotli support (#6279) 
Signed-off-by: Balázs Szücs <[email protected]>
 2026-06-05 18:49:09 +01:00
EthanHealy01andGitHub 1698769928 Improvements to agent chat markdown rendering. (#6507) 
### To test

- Ask the agent to “list all the things you can do and put them in a
markdown table”. I know we’re explicitly asking it for markdown, but I
don’t want to update the system prompt to ask it to make tables when
necessary because it’ll probably turn everything into a table, not sure
though, we can test in future.
     -  Notice how the loading is different
- Notice how the user chat is in a bubble but the agent chat is flat
(super standard design practice in AI tools, and looks much better when
the agent outputs mardown, expecially tables and needs room to do so)
- Ask it to do something different, then close the chat, and see that
the agent is marked as running and has a green outline and a green dot.
     - Play around with resizing the chat to make it bigger/smaller    
    
Open to any and all criticisms on any of the design choices, and of
course the usual, code etc.


Resizing
<img width="1572" height="812" alt="Screenshot 2026-06-01 at 2 47 53 PM"
src="https://github.com/user-attachments/assets/ec0ac1d0-01da-4025-bf7e-eea4eb544181"
/>

Loading (cool animation not visible through screenshot obviously)
<img width="559" height="141" alt="Screenshot 2026-06-01 at 2 53 41 PM"
src="https://github.com/user-attachments/assets/99f0b1f5-1719-4d78-8947-21b142293052"
/>

Removed bubbles for agent chat (maybe controversial, let me know) and
markdown now renders properly again
<img width="654" height="1060" alt="Screenshot 2026-06-01 at 2 55 01 PM"
src="https://github.com/user-attachments/assets/445f0889-a632-4751-9a16-f80ae388c632"
/>
 2026-06-04 18:26:19 +00:00
ConnorYohandGitHub 22dacbed01 PAYG B-2: shadow-mode filter + interceptor (engine activation) (#6519) 
## What this PR does

Wires the **B-1 shadow charging engine** into real HTTP request flow.
After this lands, flipping an internal team to ``PAYG_SHADOW`` via SQL
begins populating ``payg_shadow_charge`` automatically — with **zero
impact** on the legacy credit deduction path.

**This is the load-bearing PR for shadow mode.** Without it, B-1's
engine sits idle — nothing in the codebase calls
``JobChargeService.openProcess()`` from a real HTTP request.

Stacks on top of #6477 (PR B-1).

## Components

| Class | Role |
|---|---|
| ``PaygResponseBodyWrapperFilter`` | Servlet filter, installs tee'ing
response wrapper. Defers wrapper close to ``AsyncListener`` for
``DeferredResult`` / ``CompletableFuture`` controllers so the lifetime
spans the async window. |
| ``PaygResponseBodyWrapper`` | ``HttpServletResponseWrapper`` —
in-memory ``ByteArrayOutputStream`` up to 10 MiB; spills to ``TempFile``
above. ``materialisedPath()`` always returns a uniform ``Path``
interface. |
| ``PaygChargeInterceptor`` | ``AsyncHandlerInterceptor`` mirroring
``UnifiedCreditInterceptor`` shape. ``preHandle`` gates on
``@AutoJobPostMapping``, materialises multipart inputs, calls
``JobChargeService.openProcess``. ``afterCompletion`` branches on HTTP
status. |
| ``PaygOutputExtractor`` | Pulls PDFs out of the response body. Direct
``application/pdf`` returns body verbatim; ``application/zip`` iterates
entries and keeps each ``.pdf`` entry whose first bytes match the
``%PDF-`` magic. |
| ``PaygWebMvcConfig`` | Registers filter at end of Spring filter chain
(after security); interceptor after ``UnifiedCreditInterceptor``. |
| ``PaygFilterProperties`` | ``payg.filter.enabled`` master switch +
in-memory threshold + optional max-bytes ceiling. |

## Status branching in afterCompletion

| HTTP status | Action |
|---|---|
| **2xx** | Append OK step; extract PDFs from response;
``JobService.recordOutput`` per PDF |
| **4xx** | Append FAILED step with ``errorCode``. No refund — customer
paid for the attempt. No OUTPUT recording. |
| **5xx + OPENED** (first-step) |
``JobChargeService.markFirstStepFailed`` → shadow row flipped to
``REFUNDED``, process CLOSED. Refund counter incremented. |
| **5xx + JOINED** (mid-chain) | ``JobChargeService.decrementStepCount``
— step slot returned without resetting ``lastStepAt`` (workflow window
stays active for retry). |

## New ``JobChargeService`` methods

- **``markFirstStepFailed(jobId, reason)``** — flips shadow row to
``REFUNDED`` with ``refundedAt`` + ``refundReason``, closes the process.
Idempotent. Mimics the eventual Stripe
``meter_event_adjustment(cancel)`` flow that real-mode will invoke at
the same callsite. **Refund implies close** so a same-input retry can't
lineage-join into a refunded chain for free work.
- **``decrementStepCount(jobId)``** — defensive floor at 1; never drives
count negative.

## Schema

- Backend: ``V13__payg_shadow_charge_status.sql`` adds ``status``
(``CHARGED`` | ``REFUNDED``) + ``refunded_at`` + ``refund_reason``.
``DEFAULT 'CHARGED'`` so existing B-1 rows stay correct without
backfill.
- Supabase: matching migration in
[Stirling-PDF-SaaS#payg-shadow-charge-status](https://github.com/Stirling-Tools/Stirling-PDF-SaaS/tree/payg-shadow-charge-status)

## Fail-open semantics in shadow

Any ``RuntimeException`` in ``preHandle`` / ``afterCompletion`` is
logged at WARN, increments ``payg.filter.errors``, and lets the
customer's tool call proceed unbilled. This **reverses to fail-closed**
when ``wallet_policy.engine = PAYG`` (real charging) — that reversal
lives inside ``JobChargeService`` and ships with the cap evaluator PR
(PR-C1 in PAYG_DESIGN.md).

## Observability

Micrometer metrics:
- ``payg.filter.errors`` Counter — internal failures (preHandle +
afterCompletion). Alert source.
- ``payg.filter.calls`` Counter, tagged ``disposition`` (``OPENED`` |
``JOINED`` | ``SHORT_CIRCUIT``)
- ``payg.filter.refunds`` Counter — first-step 5xx refunds
- ``payg.filter.duration`` Timer — preHandle + afterCompletion
wall-clock per request

## Test coverage (38 tests across 4 classes)

- **PaygResponseBodyWrapperTest** (12 tests) — in-memory, spill,
threshold crossing mid-chunk, writer vs outputStream exclusivity,
``resetBuffer`` with and without spill, close idempotency, single-byte
writes across threshold.
- **PaygOutputExtractorTest** (7 tests) — direct PDF, parametrised
content type, ZIP with mixed entries + magic-byte gate, corrupt ZIP
fail-open, empty ZIP.
- **PaygChargeInterceptorTest** (13 tests) — all preHandle
short-circuits, OPENED disposition stash, fail-open on chargeService
exception, 2xx recordOutputs path, 5xx OPENED → markFirstStepFailed, 5xx
JOINED → decrementStepCount, 4xx FAILED step append, max-bytes ceiling
skip, PIPELINE header detection.
- **JobChargeServiceTest extended** (+6 tests) — markFirstStepFailed
happy path, idempotency, missing-shadow-row case, long-reason trim;
decrementStepCount happy path, floor-at-1 defence, missing-job no-op.

## What's NOT in this PR (deliberate)

- **No SpringBootTest layer.** The saas module doesn't have bootstrap
test infrastructure (Supabase JWT config + H2 schema harness).
Integration confidence comes from the manual staging deploy + SQL-flip
of an internal team. Bootstrap-test infra is a focused follow-up if
needed.
- **No saas-mode Behave / docker-compose.** Per design §17 — deferred.
Existing ``testing/cucumber/`` infrastructure doesn't yet have a
saas-profile compose target; that's its own PR when warranted.
- **No CreditService wire-in** (per design §13 decision). Per-row
comparison data moves to the reconciliation report PR (PR-S2).
``legacy_credits_charged`` + ``diff_pct`` columns stay at 0 in shadow
rows.
- **No reconciliation report endpoint.** Direct SQL queries against
``payg_shadow_charge`` cover the data-access need until patterns emerge.

## Rollback levers

| Symptom | Lever |
|---|---|
| Some / all tool calls breaking due to filter |
``payg.filter.enabled=false`` + restart (~20s) |
| Shadow rows look wrong for a specific team | ``UPDATE wallet_policy
SET engine = 'LEGACY' WHERE team_id = ?`` |
| Mass shadow weirdness | ``UPDATE wallet_policy SET engine = 'LEGACY'``
|
| Memory exhaustion from response tee | Lower
``payg.filter.response.in-memory-threshold-bytes`` |

## Test plan

- [ ] CI green (build + tests)
- [ ] Aikido / Snyk / SonarCloud clean
- [ ] Manual: deploy to staging
- [ ] Manual: flip one internal team via ``UPDATE wallet_policy SET
engine = 'PAYG_SHADOW' WHERE team_id = ?``
- [ ] Manual: hit ``/api/v1/security/add-password`` with that team's
JWT; verify a ``payg_shadow_charge`` row appears with
``status='CHARGED'``
- [ ] Manual: trigger a 503 (e.g. via temporary backend kill
mid-request); verify the resulting row is ``status='REFUNDED'`` + the
process is ``CLOSED``
- [ ] Manual: hit ``/api/v1/general/split`` with a multi-page PDF;
verify one OUTPUT signature per inner PDF appears in
``job_artifact_hash``
- [ ] Manual: chain ``add-password`` → ``compress`` on the output;
verify the second call JOINS the first process (no new shadow row) and
the inner output OUTPUT signature is what drove the lineage join

## Stacks on / references

- Stacks on: #6477 (B-1 — shadow charging engine)
- Schema mirror: Stirling-PDF-SaaS#payg-shadow-charge-status branch
- Design doc: ``notes/PAYG_FILTER_DESIGN.md`` (all 19 decisions DECIDED)
 2026-06-04 15:07:58 +00:00
ConnorYohandGitHub 3807cdfbc6 PAYG: process tracking + shadow charging engine (PR B-1) (#6477) 
> 📌 **Stacked on
[#6464](https://github.com/Stirling-Tools/Stirling-PDF/pull/6464)**
(lineage primitives, still in review). #6469 has merged so its commits
are no longer in this PR's diff. Once #6464 merges, a final rebase
collapses the lineage-primitives commits out of this diff too — leaving
only the B-1 work.

## What this is

Process tracking + shadow charging engine. Bundles PR-I7 service half
with the non-filter piece of PR-I7a so the pieces ship together — none
of them is useful in isolation.

**Review focus:** the new files in:
- \`app/saas/src/main/java/stirling/software/saas/payg/job/\`
(\`JobService\`, \`JobContext\`, \`JoinOrOpenResult\`,
\`StaleJobCloser\`)
- \`app/saas/src/main/java/stirling/software/saas/payg/charge/\`
(\`JobChargeService\`, \`ChargeContext\`, \`ChargeOutcome\`,
\`JobInput\`)
-
\`app/saas/src/main/java/stirling/software/saas/payg/lineage/LineagePruneScheduler.java\`
- their tests

The 8 files inherited from #6464 (lineage primitives) are unchanged from
there — they ride along in this diff until #6464 lands.

The remaining work for shadow-in-staging is the ingress/egress filter
that wires controllers into this engine — that's PR B-2.

## Scope

### \`JobService\` — persistence + lineage policy

- **\`joinOrOpen\`** — the multi-input "any-match-joins, newest wins"
rule. Hash every input via the lineage detector; if any matches an open
process in the workflow window, attach to the one with the freshest
\`lastStepAt\`. Step-limit overflow on the matched job spawns a fresh
process; the new job's input signatures are still recorded so
\`mostRecentMatchWins\` routes future calls forward.
- **\`recordOutput\`** — post-tool-success path. Records OUTPUT
signatures so the next call that takes this file as input
lineage-matches into the same process.
- **\`appendStep\`** — audit-trail step row written after a tool
completes.
- **\`close\`** — idempotent; safe to call from multiple paths
(explicit, FE on-unload, scheduler). Returns the same row on re-close,
no state mutation.
- **\`findStale\` / \`closeStale\`** — workflow-window-based stale
closure used by the scheduler.

### \`JobChargeService\` — the orchestrator (shadow variant)

\`openProcess\` resolves the effective policy via
\`PricingPolicyService\` (now in main via #6469), derives the step-limit
for the current \`JobSource\` (with a defensive fallback if the policy
is missing an entry), delegates to \`JobService.joinOrOpen\`, and on
OPENED runs the \`DocumentClassifier\` + writes a \`payg_shadow_charge\`
row. Applies the policy-level \`minChargeUnits\` floor per design § 3.4.

Shadow variant only — never debits the ledger, never posts a Stripe
meter event. The real-charging follow-up reuses the same orchestration
and swaps the side-effect. \`legacyCreditsCharged\` on the shadow row
stays \`0\` until the legacy \`CreditService\` is wired in (PR B-2),
where the comparison becomes meaningful.

### Schedulers (both plain \`@Scheduled\`)

- **\`StaleJobCloser\`** — fixed-rate 60 s. Closes \`OPEN\` jobs idle
past the workflow window. API users never have to call close explicitly
— this is the safety net.
- **\`LineagePruneScheduler\`** — hourly cron, retention 1 h. Deletes
\`job_artifact_hash\` rows older than the retention window.
- **No \`@SchedulerLock\` / no \`shedlock\` table** — consistent with
the 5 existing unguarded \`@Scheduled\` tasks in \`:saas\`
(\`CreditResetScheduler\` and friends, none of which are guarded today).
Cluster-correctness across all 7 saas schedulers is tracked in design §
9 as a separate focused cleanup. Underlying operations are idempotent —
duplicate firings on multi-pod would be wasted DB load, not data
corruption.

### Records (call-shape glue for PR B-2's filter)

- \`JobContext\` / \`JoinOrOpenResult\` — input/output for
\`JobService\`.
- \`ChargeContext\` / \`ChargeOutcome\` — input/output for
\`JobChargeService\`.
- \`JobInput\` — paired \`(MultipartFile, materialised Path)\` so the
upcoming ingress filter can pass both views without re-materialising.

## Tests

**26 new, all green.**

- 14 × \`JobServiceTest\` — no-match → opens new, single-match → joins
existing, multi-input any-match-joins, multi-match newest-wins (older
job never even looked up), step-limit hit spawns fresh job (and original
\`stepCount\` is NOT mutated), empty inputs reject, stale-signature
handling, recordOutput delegation, close idempotency, closeStale,
appendStep persistence.
- 7 × \`JobChargeServiceTest\` — JOINED skips classifier + shadow write
entirely, OPENED writes shadow row + classifies (single + multi file
paths), \`minChargeUnits\` floor applied, step-limit resolved
per-\`JobSource\` from policy, missing source entry falls back to
conservative default of 10.
- 2 × \`StaleJobCloserTest\`, 3 × \`LineagePruneSchedulerTest\` —
scheduler-wiring smoke + constructor-validation tests.

\`ENABLE_SAAS=true ./gradlew :saas:test\` — BUILD SUCCESSFUL.

## What's not in this PR (lands in PR B-2)

- **Tool ingress/egress servlet filter.** The highest-risk piece —
materialises the request body into a \`JobInput\`, calls
\`JobChargeService.openProcess\` from every \`@AutoJobPostMapping\`,
records OUTPUT after success. Edge cases to validate: multipart parts,
async controllers, streaming responses, errored 5xx paths, very large
files. Design decisions for the filter are being worked through in
\`notes/PAYG_FILTER_DESIGN.md\` before any code is written.
- **Wire shadow path into legacy \`CreditService\`.** Every legacy debit
writes a comparison row carrying both the PAYG would-be units and the
legacy actual credits, populating \`diffPct\`.

## Design doc

\`notes/PAYG_DESIGN.md\` — PR-I7 + PR-I7a status updated to reflect this
bundle. § 9 carries the cluster-correctness deferral note alongside the
existing LISTEN/NOTIFY trade-off note. § 7.5.1 readiness summary shows
the path now needs just **1 more PR** (the filter half + CreditService
wire-in, both bundled into PR B-2).
 2026-06-04 11:09:25 +00:00
EthanHealy01GitHubJames Brunton
 		35a712a278 smart redaction (#6195) 
Co-authored-by: James Brunton <[email protected]>
 2026-06-03 16:16:33 +00:00
stirlingbot[bot]GitHubAnthony Stirling
 		895dcbbafd Update Backend 3rd Party Licenses (#6407) 
Co-authored-by: Anthony Stirling <[email protected]>
Signed-off-by: stirlingbot[bot] <stirlingbot[bot]@users.noreply.github.com>
 2026-06-03 15:52:28 +01:00
James BruntonandGitHub b705c5b84c Switch to use JPDFium v1.0.2, which signs the Mac binaries (#6521) 
# Description of Changes
Currently, it's not possible to develop the backend on Mac without
manually signing the JPDFium binaries yourself since macOS will reject
running the unsigned binaries. [We've now updated JPDFium to sign the
Mac binaries in
v1.0.2](https://github.com/Stirling-Tools/JPDFium/releases/tag/v1.0.2),
so update to use that version.
 2026-06-03 11:55:38 +00:00
James BruntonandGitHub 1264f4cfed Set up document management for Stirling Engine (#6476) 
# Description of Changes
Change Stirling Engine to support deleting documents automatically. This
happens both on user logout and after an amount of time specified by the
Java when ingesting a document (allowing for personal documents to have
short lifetimes but org documents to be left in the db with no expiry
date). Also sets up an [ACL
policy](https://en.wikipedia.org/wiki/Access-control_list) for the
documents so the database knows which users have access to which
documents. This is not fully implemented in the Java, so currently all
docs are treated as having a single owner, the uploader, but
theoretically when we need to support org storage, we shouldn't need to
change the db schema.
 2026-06-03 11:52:11 +00:00
ConnorYohandGitHub e6974d52f7 PAYG: hash-lineage detection primitives (modular extractor / store / detector) (#6464) 
## What this is

Three orthogonal interfaces — each with one production impl — for
detecting whether an incoming tool call should join an existing process
via content-hash lineage. Groundwork for PR-I7a: nothing in this PR
calls the detector yet; the ingress/egress filter that wires it into
every controller lands separately.

Built to be modular along three axes. Swapping any of them should not
require changes elsewhere:

| Axis | Interface | V1 impl | Plausible future impl |
|---|---|---|---|
| Hash algorithm | `LineageSignatureExtractor` |
`ByteHashSignatureExtractor` (SHA-256) | `PdfMetadataSignatureExtractor`
(PDF `/ID`, content-stream hash) |
| Storage backend | `JobLineageStore` | `JpaJobLineageStore` |
`RedisJobLineageStore` |
| Matching policy | `HashLineageDetector` | `DefaultHashLineageDetector`
| strategy-driven variant (any-match-joins for multi-input — lives in
JobService) |

## Interfaces

### `LineageSignatureExtractor` — what counts as a fingerprint

```java
public interface LineageSignatureExtractor {
    Set<LineageSignature> extract(Path file) throws IOException;
    String name();
}
```

File-based (not stream-based) so a future PDF-aware extractor can open
the same file via jpdfium / PDFBox and pull `/ID[0]` or a content-stream
hash. Multiple extractors compose at the detector layer — Spring
auto-wires all `LineageSignatureExtractor` beans, the detector unions
their results.

Production impl: **`ByteHashSignatureExtractor`** — SHA-256 over the
file via 64 KiB-buffered `DigestInputStream`. Hardware-accelerated by
the JVM (Intel SHA-NI, ARM SHA).

### `JobLineageStore` — where signatures live

```java
public interface JobLineageStore {
    void record(UUID jobId, Set<LineageSignature> signatures, ArtifactKind kind);
    Optional<LineageMatch> findOpenJobForSignatures(Long userId, Set<LineageSignature> candidates, Duration window);
    int pruneOlderThan(Instant cutoff);
}
```

Knows nothing about storage technology. Production impl
**`JpaJobLineageStore`** runs a single joined query against
`job_artifact_hash` ⋈ `processing_job` — status + window filtering
happen at the database. The query is bounded by `Limit.of(1)` on the hot
path so a job set sharing a popular signature doesn't materialise
unwanted rows. A future `RedisJobLineageStore` (or write-through hybrid)
is a drop-in.

### `HashLineageDetector` — the high-level API

```java
public interface HashLineageDetector {
    Optional<LineageMatch> detect(Long userId, Path inputFile) throws IOException;
    void record(UUID jobId, Path file, ArtifactKind kind) throws IOException;
}
```

**`DefaultHashLineageDetector`** delegates extraction to every
registered `LineageSignatureExtractor`, storage to the configured
`JobLineageStore`, and reads `payg.lineage.workflow-window` (default
`PT5M`) from config. When a single extractor throws (e.g. a future
PDF-aware extractor against a malformed PDF), the other extractors still
contribute — failures don't block the byte-hash from landing.

## Profile gating

All three `@Component` beans (`JpaJobLineageStore`,
`ByteHashSignatureExtractor`, `DefaultHashLineageDetector`) are
`@Profile("saas")` — consistent with every other `:saas` bean. Without
this guard the JPA store would fail to wire against its profile-gated
repository in non-saas profiles that pull `:saas` onto the classpath.

## Tests

Run entirely in-memory; no database required.

- **`LineageSignatureTest`** — storage-key encoding round-trips, rejects
malformed `"type:value"` keys.
- **`ByteHashSignatureExtractorTest`** — identical bytes → identical
sigs; empty file hashes to the well-known SHA-256-of-empty constant; 10
MiB file streams without OOM.
- **`DefaultHashLineageDetectorTest`** — same-user / within-window /
status=OPEN filtering, multi-signature matching (one extractor sees
`pdf-id` and matches even when bytes differ), most-recent-job-wins,
record+detect round-trip, extractor-throwing-doesn't-break-others.

**`InMemoryJobLineageStore`** (in test sources) implements the same
`JobLineageStore` interface as the JPA impl, plus a `registerJob` hook
for tests to model job state. Same contract — proves the abstraction is
portable. When the Redis impl lands it gets the same contract tests.

## What's not in this PR (deliberate)

- The tool ingress/egress filter that wires the detector into every
controller — separate, focused review.
- `JobChargeService.openProcess()` — uses the detector, part of the
charging machinery, separate PR.
- Prune scheduler that calls `pruneOlderThan` — small follow-up
alongside the `shedlock` foundational table.
- PDF-aware extractor (`PdfMetadataSignatureExtractor`) — to be added
when we measure how often byte-hash-only misses real workflows.
- Multi-input "any-match-joins" lineage policy — that's a `JobService`
decision (PR-I7), not a primitive.

## Self-review pass applied

An independent code-review on this PR caught:

- **HIGH:** Missing `@Profile("saas")` on the three `@Component` beans →
fixed.
- **MEDIUM:** `pruneOlderThan` missing `@Transactional` (its
`@Modifying` query would have thrown
`InvalidDataAccessApiUsageException`) → fixed.
- **MEDIUM:** `findOpenJobsForSignatures` fetching the whole match set
just to `get(0)` → now takes `Limit`, JPA store passes `Limit.of(1)` on
the hot path.
- **LOW:** `InMemoryJobLineageStore` used both `synchronized` methods
and `ConcurrentHashMap` → dropped the redundant `ConcurrentHashMap`.

Deferred: project-wide UTC unification (`LocalDateTime.now()`
system-zone is the established convention; flipping one file mid-stack
caused a real test failure — proper fix needs its own audit).

## Rollback

Straight `git revert`. No callers yet; deleting these classes wouldn't
break anything.

---

## Checklist

- [x] Tests pass: `ENABLE_SAAS=true ./gradlew :saas:test`
- [x] No new warnings
- [x] Self-review performed (HIGH + MEDIUM findings addressed)
 2026-06-03 11:00:08 +00:00
Anthony StirlingandGitHub b355ccec9e Add Valkey cluster backplane and sticky-410 ownership (clusters) (#6472) 2026-06-02 14:59:20 +01:00
briosandGitHub 2c0ebc28a7 perf(api): optimize static asset caching, enable ETag support, and expand response compression mime types. (#6273) 2026-06-01 16:37:56 +01:00
Anthony StirlingandGitHub 30e782e29c Disable Save-to-server when storage off, fix QR port 0 (#6473) 2026-05-29 19:37:42 +01:00
ConnorYohandGitHub 28b81828b5 PAYG: PricingPolicyService + admin REST + 30s read cache (#6469) 
## What this is

PR-I1 service half from `notes/PAYG_DESIGN.md`. Built on top of the data
model from #6460 — answers "what pricing policy applies to this team
right now?" with a fast cache and an admin write surface.

## Scope

| Piece | Where |
|---|---|
| `PricingPolicyService` — `getEffectivePolicy(teamId)` with 30s
Caffeine cache + admin write paths |
`app/saas/.../payg/policy/PricingPolicyService.java` |
| `PolicyChangedEvent` — published after admin writes for in-process
cache invalidation | `app/saas/.../payg/policy/PolicyChangedEvent.java`
|
| Admin REST — list / get / create / set-default / set team override /
get effective |
`app/saas/.../payg/policy/admin/PricingPolicyAdminController.java` +
DTOs |
| `PricingPolicyRepository.clearDefaultFlag()` — atomic clear for
set-default | repository update |
| `SaasJpaConfigScanTest` — drift guard against the JPA scan paths going
stale (carried over from the #6460 review concern) | new test |
| V12 default-policy seed (`v1-initial`, 25 pages/unit, 5 MiB/unit,
per-`JobSource` step limits) | `V12__seed_default_payg_policy.sql` |

## Lookup precedence

1. `PaygTeamExtensions.pricingPolicyId` set → return that policy
2. Else return the `pricing_policy` row with `is_default = TRUE`
3. Override row points at a deleted policy → log warn, fall back to
default (safety net for racing deletes)
4. No default → `IllegalStateException` (V12 seed guarantees one exists)

## Cache behaviour

- 30s `expireAfterWrite` Caffeine, max 10k entries, keyed by `teamId`.
- **Single correctness model: the TTL.** Cross-instance propagation is
at-most-30-seconds. The writer instance sees its own change immediately
via the `PolicyChangedEvent` after-commit publish. Other instances pick
it up on the next TTL expiry.
- Admin reads use `getEffectivePolicyUncached` so admins always see
their own write straight back.

**Why no LISTEN/NOTIFY runner.** An earlier cut of this PR included a
Postgres `LISTEN policy_changed` runner so cross-instance propagation
was instant. Dropped — admin policy changes are events-per-week and the
30s TTL is already the correctness floor; the listener was ~250 lines of
nontrivial code (raw JDBC outside HikariCP, daemon thread, reconnect
loop, lock-protected connection lifecycle) for a use case that isn't on
the hot path. Trade-off is documented in `notes/PAYG_DESIGN.md` §9 with
three concrete triggers that would justify reintroducing it (aggressive
cap enforcement, Redis landing for other reasons, real-time admin UI).

## Writes — transactional, fire `PolicyChangedEvent` after commit

- `create(draft)` — rejects pre-set `policy_id` or `is_default=true`
(promotion must go through `setDefault` so the partial unique idx is
freed first).
- `setDefault(id)` — atomically clears the existing default via
`clearDefaultFlag()` then flips the new row. Idempotent: silent no-op if
the row is already default.
- `setTeamOverride(teamId, policyId | null)` — validates the policy
exists before save; `null` clears the override.

`publishOnCommit` uses `TransactionSynchronizationManager.afterCommit`
so listeners never see pre-commit state. Outside a transaction (test
paths) falls through to immediate publish.

## Admin REST surface — `/api/v1/admin/payg/...`

All endpoints `@PreAuthorize("hasRole('ADMIN')")`:

- `GET  /policies` — list all
- `GET  /policies/{id}` — read one
- `POST /policies` — create new (non-default)
- `POST /policies/{id}/set-default` — atomic promote
- `PUT /teams/{teamId}/policy-override` — set or clear per-team override
- `GET  /teams/{teamId}/effective-policy` — cache-bypassing live read

Validation errors → 400, unknown rows → 404.

## Counterpart Supabase PR


[`Stirling-PDF-SaaS#298`](https://github.com/Stirling-Tools/Stirling-PDF-SaaS/pull/298)
— seeds the same V1 default policy on the Supabase side via
`20260528000002_payg_seed_default_policy.sql`.

## Tests

- 17 × `PricingPolicyServiceTest` — lookup precedence, cache hit/miss,
invalidation on event, mutation paths publishing event, error cases.
- 14 × `PricingPolicyAdminControllerTest` — every endpoint's happy path
+ error mapping, DTO defensive-copy invariant.
- 2 × `SaasJpaConfigScanTest` — reflection-based guard that
`payg.repository` is in `@EnableJpaRepositories` and `payg` is in
`@EntityScan`. Without this, new sub-packages can silently fail to wire
at runtime — same class of bug that the #6460 review caught.

Full `:saas:test` BUILD SUCCESSFUL.

## Design doc

`notes/PAYG_DESIGN.md` §7.4 PR-I1 — completes the service half (the
schema half landed in #6460). §9 carries the 30s-TTL trade-off note.
 2026-05-29 16:27:01 +00:00
ConnorYohandGitHub 83ea07ed6a saas: DocumentClassifier + PAYG data model (#6460) 
# Description of Changes

Two layers — the `DocumentClassifier` utility plus the full data model
for the new billing engine. Nothing wires the entities into application
behaviour yet; services and controllers land in follow-up PRs.

**Companion PR:**
[Stirling-PDF-SaaS#296](https://github.com/Stirling-Tools/Stirling-PDF-SaaS/pull/296)
— Supabase migration for the v3 dev branch, schema-equivalent to the
Flyway migration in this PR.

## 1. DocumentClassifier (under `payg.docs`)

`DocumentClassifier` computes the doc-unit cost of an uploaded file (or
multi-file input) under a `PricingPolicy`. PDFs read page count via
`stirling.software.jpdfium.PdfDocument`; non-PDFs are bytes-only.
Formula: `max(ceil(pages / docPagesPerUnit), ceil(bytes /
docBytesPerUnit))` clamped to `[1, fileUnitCap]`. Multi-file is the sum
of raw per-file units capped at `fileUnitCap × file_count`.

Two floors, by design: the classifier returns `docUnits` with an
absolute `1` floor for non-empty input; the policy-level
`minChargeUnits` is intentionally applied later, at process-open time in
`JobChargeService`, per design § 3.4 (`unitsForProcess =
max(policy.min_charge_units, docUnits)`). Documented in the interface +
impl javadoc.

Upload bytes are materialised through
`TempFileManager.createManagedTempFile` so jpdfium gets a `Path`; the
temp file auto-deletes on close.

Twelve tests, all in-memory fixtures generated with PDFBox at test time
— no committed binary blobs.

## 2. PAYG data model (under `payg.*`)

JPA entities, repositories, and a Flyway migration covering the full
schema in §6 of the design.

**Enums** (`payg.model`):

`JobSource`, `ProcessType`, `JobStatus`, `JobStepStatus`,
`ArtifactKind`, `LedgerEntryType`, `LedgerBucket`, `ReferenceType`,
`EntitlementState`, `FeatureSet`, `FeatureGate`, `WalletEngine`,
`CapPeriod`, `AutoGroupStrategy`.

**Entities + repositories:**

| Entity | Table | Notes |
|---|---|---|
| `PricingPolicy` | `pricing_policy` | Promoted from a record.
`stepLimits` is `Map<JobSource, Integer>` persisted via normalised child
table `pricing_policy_step_limit`. `stripePriceIds` is `Set<String>`
persisted via `pricing_policy_stripe_price` — currency comes from
`stripe.prices` via Sync Engine, not stored locally. |
| `ProcessingJob` | `processing_job` | UUID PK. Tracks lineage window
via `step_count` and `last_step_at`. |
| `ProcessingJobStep` | `processing_job_step` | Per-tool-call audit. |
| `JobArtifactHash` | `job_artifact_hash` | Composite key `(job_id,
content_hash, kind)`. `content_hash VARCHAR(128)` so multiple signature
schemes coexist as `"type:value"` storage keys. Lineage detector queries
this. |
| `WalletLedgerEntry` | `wallet_ledger` | Append-only, signed
`amount_units`. Two unique indexes kill double-posting. |
| `WalletPolicy` | `wallet_policy` | Per-team engine + cap + degradation
rules + lineage strategy. No `@Version` — admin-only writes (documented
in javadoc). |
| `WalletEntitlementSnapshot` | `wallet_entitlement_snapshot` |
Composite key `(team_id, user_id)`; `user_id = 0` is the team-wide
sentinel. No `@Version` — full-row recompute via
`EntitlementService.recompute` (documented in javadoc). |
| `PaygShadowCharge` | `payg_shadow_charge` | Per-job diff while in
`PAYG_SHADOW` engine mode. |
| `PaygTeamExtensions` | `payg_team_extensions` | Sidecar 1:1 with
`teams` carrying `pricing_policy_id` (per-team override) +
`stripe_customer_id`. Sidecar pattern (mirrors `saas_team_extensions`)
so OSS Hibernate ddl-auto never sees PAYG columns on `teams`. |

**Column adds:**

- `team_memberships.cap_units` (optional per-member sub-cap)

**Width split (intentional, documented in V11):** per-row deltas
(`wallet_ledger.amount_units`, `processing_job.charged_units`) are
`INTEGER` because no single charge realistically approaches 2B units.
Cap and period-rollup columns (`team_memberships.cap_units`,
`wallet_policy.cap_units`,
`wallet_entitlement_snapshot.period_spend_units / period_cap_units`) are
`BIGINT` because they accumulate across a billing period and admins may
legitimately set headroom-cap values into the millions.

**JPA wiring:** `SaasJpaConfig` was updated to include
`stirling.software.saas.payg.repository` in
`@EnableJpaRepositories.basePackages` and `stirling.software.saas.payg`
in `@EntityScan` (covers `payg.policy` / `payg.job` / `payg.wallet` /
`payg.entitlement` / `payg.shadow` recursively). New
`SaasJpaConfigScanTest` reads the annotations reflectively and asserts
every expected package is wired — catches the next time someone adds a
new sub-package without updating the scan paths.

**Migration:** `V11__saas_payg_model.sql` (purely additive).
Schema-equivalent to the Supabase migration in the companion PR —
including the `VARCHAR(128) content_hash` width that's needed for the
multi-signature-scheme storage encoding the lineage layer uses.

## 3. Smoke tests

`PaygEntitiesSmokeTest` exercises each entity via the no-arg ctor JPA
requires, plus getter/setter round-trips and composite-key equality —
catches Lombok/annotation regressions without needing a database.
Real-DB integration coverage lands alongside the services that consume
each entity.

## Why this is safe to land now

- All schema changes are additive — no existing rows modified, no
columns dropped.
- The entities are not yet referenced from any production code path;
they exist for the next PRs to build on.
- The v3 Supabase dev branch picks up the schema via the companion PR;
the main repo's Flyway migration applies the same shape when an instance
boots against a freshly-migrated v3 database.

## Open decisions made

- **Step-limits keyed by `JobSource`** rather than by `ProcessType`.
Captures the "self-hosted gets a different knob" framing in earlier
feedback. Trivially overridable per pricing policy version.
- **Step limits + Stripe price IDs normalised into child tables** rather
than JSONB on `pricing_policy` (per Connor's review on #296). Typed
columns, queryable directly, no JSON parsing.
- **Currency dropped from `pricing_policy_stripe_price`** — it lives on
`stripe.prices.currency` and is resolved via Sync Engine. App is
currency-blind.

## Rollback

Straight `git revert` on this PR. The Supabase migration in #296 is
additive and can be left in place safely — the running app ignores
tables it doesn't reference.

---

## Checklist

- [x] I have read the [Contribution
Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md)
- [x] I have read the [Stirling-PDF Developer
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md)
- [x] I have performed a self-review of my own code
- [x] My changes generate no new warnings
- [x] I have run `task check` (via `./gradlew :saas:test` with
`ENABLE_SAAS=true`) — passes
 2026-05-29 12:03:01 +00:00
EthanHealy01andGitHub 763595a5a3 feat: add Agents UI to proprietary right sidebar (#6454) 
Update UI to include agents

Run `task dev:all` to test
 2026-05-28 17:26:23 +00:00
Anthony StirlingandGitHub 398617391b Fix SSO auto-login and custom metadata settings not persisting on restart (#6468) 2026-05-28 17:39:05 +01:00
ConnorYohandGitHub a0e0e88f07 saas: harden CreditService Stripe ordering + lint @AutoJobPostMapping weights (#6458) 
# Description of Changes

Two narrowly-scoped hardening changes to the credits engine.

## 1. CreditService — move Stripe meter call to `afterCommit`

The Stripe metered-usage call sits inside the surrounding
`@Transactional`, holding the `user_credits` row lock for the duration
of an HTTP round-trip to Supabase. Under load this starves concurrent
debits; a transient Stripe blip rolls back a (correct) free-credit
consumption and forces the caller to retry.

The Stripe call now runs in a `TransactionSynchronization.afterCommit`
hook — DB commits first, Stripe fires immediately after. If Stripe fails
after commit, we log + increment a new `credits.stripe_report.failures`
counter; the idempotency key is stable, so a manual replay recovers
without double-charging.

Applied to both `consumeCreditBySupabaseId` and
`consumeCreditWithWaterfall`.

**Dead-code removed:**
- Unreachable UUID fallback for MDC `requestId` — `CorrelationIdFilter`
already guarantees the key on every request.
- The `"Unable to report usage to Stripe"` `RuntimeException` and its
catch block — the afterCommit refactor eliminates the throw path.
- `StripeRollbackOnFailureTest` — pinned the rollback-on-Stripe-fail
behaviour this refactor replaces.

## 2. `@AutoJobPostMapping` — build-time lint for `resourceWeight`

`UnifiedCreditInterceptor` multiplies `resourceWeight` into the per-call
charge. An endpoint that falls through to the annotation default
produces a charge derived from a value nobody chose.

- Annotation default flipped from `1` to `Integer.MIN_VALUE` (sentinel).
Both runtime readers (`UnifiedCreditInterceptor`, `AutoJobAspect`)
already clamp into `[1, 100]` so behaviour is unchanged.
- New `AutoJobPostMappingWeightTest` scans the classpath and fails the
build if any method leaves the sentinel.
- Initial run caught 11 endpoints relying on the default. Explicit
weights now declared, chosen by comparing to peer endpoints:
  - `EditTextController` — LARGE
  - `EmailController#sendEmailWithAttachment` — SMALL
  - `ConvertPDFToMarkdown` — MEDIUM
  - `AttachmentController` (extract/list/rename/delete) — SMALL × 4
  - `ConvertImgPDFController` (cbr/cbz ↔ pdf) — MEDIUM × 2, LARGE × 2

## Tests

- `StripeUsageIdempotencyKeyTest` — pins the `(supabaseId, overage,
requestId)` idempotency key shape so Stripe always dedupes a retry.
- `StripeAfterCommitOrderingTest` — pins that `afterCommit` fires after
commit and NOT on rollback.
- `AutoJobPostMappingWeightTest` — the lint itself, plus a self-check
that the classpath scan finds at least 10 `@AutoJobPostMapping` methods
(guards against the lint passing vacuously).

Build verified: `ENABLE_SAAS=true ./gradlew :stirling-pdf:test
:saas:test`.

---

## Checklist

### General

- [x] I have read the [Contribution
Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md)
- [x] I have read the [Stirling-PDF Developer
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md)
(if applicable)
- [ ] I have read the [How to add new languages to
Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md)
(if applicable) — no translation changes
- [x] I have performed a self-review of my own code
- [x] My changes generate no new warnings

### Documentation

- [ ] I have updated relevant docs on [Stirling-PDF's doc
repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/)
— internal-billing change, no public docs impact
- [ ] I have read the section [Add New Translation
Tags](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md#add-new-translation-tags)
— N/A

### Translations (if applicable)

- [ ] Not applicable

### UI Changes (if applicable)

- [ ] Not applicable

### Testing (if applicable)

- [x] I have run `task check` (via `./gradlew :stirling-pdf:test
:saas:test` with `ENABLE_SAAS=true`) — passes
- [x] I have tested my changes locally
 2026-05-28 14:57:59 +00:00
Anthony StirlingandGitHub c80a5db5f5 folder and file fixes (#6461) 2026-05-28 15:57:35 +01:00
Anthony StirlingGitHubConnorYoh
 		8bd78d2624 Add landscape page size options (#6248) 
# Description of Changes

Adds orientation (portrait/landscape) to the Adjust Page Scale tool.

- Orientation as a separate parameter (per review), sent through to the
backend
- ScalePagesController simplified; PDFWithPageSize gains the orientation
field
- Regenerated tool_models.py; frontend + backend tests added

---

## Checklist

### General

- [ ] I have read the [Contribution
Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md)
- [ ] I have read the [Stirling-PDF Developer
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md)
(if applicable)
- [ ] I have read the [How to add new languages to
Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md)
(if applicable)
- [ ] I have performed a self-review of my own code
- [ ] My changes generate no new warnings

### Documentation

- [ ] I have updated relevant docs on [Stirling-PDF's doc
repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/)
(if functionality has heavily changed)
- [ ] I have read the section [Add New Translation
Tags](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md#add-new-translation-tags)
(for new translation tags only)

### Translations (if applicable)

- [ ] I ran
[`scripts/counter_translation.py`](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/docs/counter_translation.md)

### UI Changes (if applicable)

- [ ] Screenshots or videos demonstrating the UI changes are attached
(e.g., as comments or direct attachments in the PR)

### Testing (if applicable)

- [ ] I have run `task check` to verify linters, typechecks, and tests
pass
- [ ] I have tested my changes locally. Refer to the [Testing
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md#7-testing)
for more details.

---------

Co-authored-by: ConnorYoh <[email protected]>
 2026-05-28 14:16:15 +00:00
Anthony StirlingandGitHub b3c4b8b463 Add S3 storage and cluster artifact backend (#6457) 2026-05-28 13:06:27 +01:00
James BruntonandGitHub 44fbf8c587 Various bug fixes found while testing SaaS build (#6459) 
# Description of Changes
Various fixes and improvements I made while testing the SaaS code:
- Changes the new `.env.saas` file to live in `app/` and match the
semantics of the other `.env` files
- Adds top-level `task dev:saas` command to spawn SaaS frontend &
backend
- Deletes dead SaaS code and improves some overriding logic
- Fixes refreshing issue when coming back to the tab
- Fix the Compare tool's selection logic
- Make Compare handle error cases properly
- Fixes the location of the "Dismiss All Errors" button (was rendering
on top of the top-bar with a transparent background previously so it
looked rubbish)
- Fixes file selection in PDF Editor
 2026-05-28 11:05:30 +00:00
Anthony StirlingandGitHub 76840d8a57 Add CI DB migration smoke test against v2.0/v2.5/v2.10 updates (#6453) 2026-05-28 11:36:07 +01:00
ConnorYohandGitHub 43b67d213d feat(oauth2): opt-in claim-dump diagnostics for OIDC login failures (#6456) 
# Description of Changes

## What & why

Customers using ADFS (or any generic OIDC provider that doesn't emit
`email`) hit `Attribute value for 'email' cannot be null` during OAuth2
login with no visibility into what claims the provider actually sent.
The only available remedy was guessing at
`security.oauth2.useAsUsername` until something worked.

This PR adds a new opt-in `security.oauth2.debugLogging` flag (default
`false`). When enabled, `CustomOAuth2UserService` logs:

- All ID token claims (sorted, with values)
- All UserInfo endpoint claims (if any)
- The merged attribute key set Spring exposes to `getAttribute()`
- The value the configured `useAsUsername` actually resolved to
- A **`Hint:`** line listing the claim keys present in the token that
map to a valid `UsernameAttribute` enum value — i.e. exactly what the
operator could put in `useAsUsername` to make login work

Logged at `INFO` on the success path and `ERROR` on failure (inside the
existing `catch (IllegalArgumentException)` block that throws
`OAuth2AuthenticationException`). The block is wrapped with a `[OAUTH2
DEBUG] ... [/OAUTH2 DEBUG]` banner and ends with a PII warning so
operators don't leave it on in production.

Default off → zero observable change for anyone not actively
troubleshooting.

## Files changed

| File | Why |
|---|---|
| `app/common/.../ApplicationProperties.java` | New `debugLogging` field
on the `OAUTH2` config class with javadoc warning about PII |
| `app/core/src/main/resources/settings.yml.template` | Documents
`oauth2.debugLogging` so it appears on next startup |
| `app/proprietary/.../security/service/CustomOAuth2UserService.java` |
Emits the claim dump + suggestion hint when the flag is on |
|
`app/proprietary/.../security/service/CustomOAuth2UserServiceDebugLoggingTest.java`
(new) | Unit test: mocks the OIDC delegate, asserts off-path is silent
and on-path emits the dump with the right Hint contents |

## End-to-end verification

Ran the bundled `testing/compose/docker-compose-keycloak-oauth.yml`
Keycloak realm, configured `security.oauth2.useAsUsername: mail`
(Keycloak emits `email`, not `mail`) and `provider: demarest` (matches
the original customer bug report). Triggered the OAuth flow at
`http://localhost:8080/oauth2/authorization/demarest` and confirmed:

- The ERROR-level dump fires with the full 19-claim ID token decoded
- `-- Value at 'mail' : <NULL — this is why login fails>` correctly
identifies the missing claim
- `-- Hint:` correctly suggests `[email, family_name, given_name,
preferred_username]` (the four keys present that map to valid
`UsernameAttribute` values)
- Auth still fails with the original `OAuth2AuthenticationException` —
no change to control flow, just added diagnostic logging

Unit test (`CustomOAuth2UserServiceDebugLoggingTest`) covers both
branches.

## Reviewer notes

- **No new public APIs.** The flag is config-only; no servlet endpoints
exposed.
- **PII is logged when the flag is on.** This is the whole point —
operators need to see the claims to fix their config — but it's gated,
defaults off, and the dump self-documents with a `WARNING: ... Set
security.oauth2.debugLogging=false once troubleshooting is complete.`
footer.
- **Why log everything, not just sub/email?** Because the operator
doesn't know in advance which claim they actually want. ADFS uses `upn`
in some configs and `preferred_username` in others; Azure AD uses `oid`;
the customer here had neither. Dumping the full set is the only way to
make the diagnostic self-service.
- **Out of scope for this PR (follow-ups):**
- The `UsernameAttribute` enum doesn't include `upn` / `unique_name`
(common ADFS claims). If the customer's token only has `upn`, the Hint
will be empty even though the operator can see `upn` in the dump. Worth
a separate PR to extend the enum.
- The known-provider validator in `Provider.java` (rejects e.g.
`useAsUsername: mail` for `provider: keycloak` at startup) bypasses our
diagnostic for those provider names. ADFS customers using `provider:
<name>` fall into the `default` branch so are not affected — but it's a
sharp edge worth documenting.

---

## Checklist

### General

- [x] I have read the [Contribution
Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md)
- [x] I have read the [Stirling-PDF Developer
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md)
(if applicable)
- [ ] I have read the [How to add new languages to
Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md)
(if applicable) — N/A, backend-only change
- [x] I have performed a self-review of my own code
- [x] My changes generate no new warnings

### Documentation

- [ ] Doc-repo update (if functionality has heavily changed) —
diagnostic flag is self-documenting via the `settings.yml.template`
comment and the in-log warning; happy to add a doc-repo entry if
reviewers want one
- [ ] Translation tags — N/A

### UI Changes (if applicable)

- [ ] N/A — backend-only

### Testing (if applicable)

- [x] Unit test added (`CustomOAuth2UserServiceDebugLoggingTest`)
covering on/off paths and Hint correctness
- [x] End-to-end verified locally against bundled Keycloak compose with
intentionally misconfigured `useAsUsername`
- [x] Full `:proprietary:test` suite passes
 2026-05-27 13:01:51 +00:00
Anthony StirlingandGitHub d42b779644 Add server-side folders and files page UI (#6383) 2026-05-27 12:52:46 +01:00
Anthony StirlingandGitHub 4564ed5bec Add cluster backplane abstraction and interfaces (#6449) 2026-05-27 11:54:59 +01:00
Matt Van HornGitHubMatt Van Horn
 		7743720f0a chore: fix "Endoints" typo in health route comment (#6446) 
`app/common/src/main/java/stirling/software/common/util/RequestUriUtils.java:175`
section comment said "Health Endoints" → "Health Endpoints".
Comment-only.

Co-authored-by: Matt Van Horn <[email protected]>
 2026-05-26 23:30:57 +01:00
Anthony StirlingandGitHub 56ff1e5092 impl migration to pdfium for split (#6410) 
## Summary

Migrates `SplitPDFController`, `SplitPdfByChaptersController`,
`SplitPdfBySizeController` from PDFBox to JPDFium.
`SplitPdfBySectionsController` and `AutoSplitPdfController` are
intentionally left on PDFBox (require JPDFium 1.0.2 features that don't
exist yet).

## Benchmark (audited on `audit/jpdfium-split`, file
`app/core/src/test/java/stirling/software/SPDF/bench/SplitBenchmark.java`)

| Workload | PDFBox heap | JPDFium heap | PDFBox wall | JPDFium wall |
|---|---|---|---|---|
| 100 pp, chunk 10 | +21-26 MB | **+0.02 MB** | 80-106 ms | **25 ms** |
| 300 pp, chunk 10 | +59 MB | **+1.0 MB** | 232 ms | **76 ms** |

**98-99.9% heap reduction. 3-4.2x faster wall.**

## Hybrid

- AcroForm-bearing splits keep PDFBox
`FormUtils.pruneOrphanedFormFields` post-pass (FPDF_ImportPagesByIndex
drops AcroForm dict). Sub-bench shows +1.0 MB / +27 ms - tightly
bounded.
- Metadata extraction stays on PDFBox.
- `SplitPdfBySectionsController` - JPDFium `PdfPageSplitter` only does
2-up halving, not arbitrary MxN.
- `AutoSplitPdfController` - needs PDFRenderer + zxing for QR markers.

## Test plan

- [ ] 30/30 unit tests pass with PDFBox `Loader.loadPDF` as the oracle
(assert page counts + document totals)
- [ ] Existing cucumber feature `split.feature` continues to pass
- [ ] AcroForm-bearing PDF round-trips without orphaned widgets (covered
by existing FormUtils tests)
 2026-05-26 17:50:13 +01:00
Reece BrowneandGitHub 4047d02086 chore: address restructure PR review feedback (#6423) 2026-05-26 14:12:05 +01:00
Reece BrowneGitHubClaude Opus 4.7
 		0a50e765b7 Restructure/frontend editor (#6404) 
## Move editor under `frontend/editor/`

Pure restructure: `frontend/` becomes the workspace, `frontend/editor/`
holds
  the PDF editor. 1775 file renames + 40 wiring edits. No logic changes.

  ### Why

`frontend/` is currently the editor — its `src/`, `public/`,
`src-tauri/`,
  config files all sit at the root. Promoting `frontend/` to a
workspace and putting the editor in a sibling folder leaves room for
future
apps to drop in alongside it, sharing one `package.json` /
`node_modules` /
  lint config / Storybook.

  ### What moves

  frontend/
  ├── editor/                ← NEW: everything editor-specific
  │   ├── src/               ← was frontend/src/
  │   ├── public/            ← was frontend/public/
  │   ├── src-tauri/         ← was frontend/src-tauri/
│ ├── index.html, vite.config.ts, vitest.config.ts, playwright.config.ts
  │   ├── tsconfig*.json, tailwind.config.js, postcss.config.js
  │   ├── scripts/
  │   ├── .env, .env.desktop, .env.saas
  │   └── DeveloperGuide.md
├── package.json, package-lock.json, node_modules/ ← workspace install
  ├── eslint.config.mjs, .prettierrc, .prettierignore ← shared tooling
  ├── .gitignore
  └── README.md

  ### Wiring edits (40 files)

  - `.taskfiles/frontend.yml`, `desktop.yml`, `e2e.yml`
  - `build.gradle`, `app/core/build.gradle`
- `eslint.config.mjs`, `frontend/package.json`, `.gitignore`,
`.prettierignore`
  - `docker/frontend/Dockerfile`
  - 8 `.github/workflows/*.yml`, plus `.github/dependabot.yml`,
    `.github/config/.files.yaml`, `.github/labeler-config-srvaroa.yml`
  - `scripts/translations/**`
- Docs: `AGENTS.md`, `CLAUDE.md`, `ADDING_TOOLS.md`,
`DeveloperGuide.md`,
`WINDOWS_SIGNING.md`, `devGuide/HowToAddNewLanguage.md`,
`frontend/README.md`,
    `frontend/editor/DeveloperGuide.md`

Plus 3 renamed + edited: `editor/vite.config.ts` (env path +
node_modules
  walk-up), `editor/scripts/setup-env.mts` (renamed from `.ts` for
`import.meta.url`), `editor/scripts/build-provisioner.mjs` (resolve
src-tauri
  relative to script).

  ### Verification

  | Check | Result |
  |---|---|
  | `task frontend:typecheck:all` (6 variants) | exit 0 |
  | `task frontend:lint` (eslint + dpdm) | exit 0 |
  | `task frontend:format:check` | exit 0 |
  | `task frontend:test` | 657 tests pass, 50 files |
| `task frontend:build:{core,proprietary,saas,desktop,prototypes}` | all
green |
| `task desktop:build` | full Tauri pipeline →
`Stirling-PDF_2.11.0_x64_en-US.msi` |
  | `playwright test --list --project=stubbed` | 172 tests discovered |

`task desktop:build` exercises the heaviest path — Rust + WiX + MSI
bundle
against the moved `editor/src-tauri/`. If anything in the restructure
was
  wrong it wouldn't have built.

  ### Test plan

  - [ ] `frontend-validation.yml` green
  - [ ] `e2e-stubbed.yml` green
  - [ ] `tauri-build.yml` green on at least one platform
  - [ ] `check_toml.yml` runs on a translation-touching PR

---------

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
 2026-05-22 13:40:34 +01:00
Anthony StirlingandGitHub 48027ee9d6 JDK25 - Integrate Stirling-Tools JPDFium for PDF operations (#6387) 2026-05-22 13:19:46 +01:00
Anthony StirlingGitHubConnorYoh
 		9d081d1792 SaaS Consolidation (#6384) 
Co-authored-by: ConnorYoh <[email protected]>
 2026-05-21 16:05:35 +01:00
dependabot[bot]andGitHub ea91a35518 build(deps): bump org.postgresql:postgresql from 42.7.10 to 42.7.11 (#6375) 
Signed-off-by: dependabot[bot] <[email protected]>
 2026-05-20 14:17:36 +01:00
Anthony StirlingGitHubConnorYoh
 		2399da6893 migrate exportUpdatedPages from bytes to stream (#6201) 
Co-authored-by: ConnorYoh <[email protected]>
 2026-05-19 23:03:44 +01:00
Reece BrowneGitHubConnorYoh
 		6b9567cf38 Split and delete forms (#6277) 
Delete orphaned forms when removing pages and maintain forms correctly
when splitting

---------

Co-authored-by: ConnorYoh <[email protected]>
 2026-05-18 11:14:55 +00:00
James BruntonandGitHub beb99e273b Improve edit agent's knowledge of tools (#6356) 
# Description of Changes
Give Edit Agent access to descriptions of the request from the Java API.
This opens the door to us better documenting our Java APIs to give the
stirling engine better knowledge of what the various tools are and how
to use them.

Also improves the tool selection sub-agent to get the tool parameters
and descriptions so it can more intelligently decide which operations
should be used to fulfil the user's request. Also provides it more
encouragement to string together multiple operations if necessary.
 2026-05-14 18:30:39 +00:00
EthanHealy01GitHubJames Brunton
 		ece1bb6865 Feature/pdf to markdown agent (#6271) 
Co-authored-by: James Brunton <[email protected]>
 2026-05-14 16:20:45 +00:00
James BruntonandGitHub 672e81d286 Add ability for Stirling engine to reason across large documents (#6314) 
# Description of Changes
Adds storage in the database for full document content alongside the RAG
content (and changes the service to `DocumentService` instead of
`RagService`). Then adds a generic capability that should be usable by
any agent (currently just used by the Question Agent) which allows the
agent to pull out the full contents of the doc, chunks it into various
sections that will fit in the context window, and then processes them in
parallel to create an intermediate result, and then processes the
intermediate result into a final answer. It will re-chunk as many times
as necessary to get the content small enough for the actual answer to be
analysed (I've tested on PDFs ~3500 pages long, which is well above the
context limit and requires maybe 3 rounds of compression to get an
answer).

The new full doc analysis stuff is heavier than the RAG lookup so both
remain. The agents should use RAG for targeted info and the chunked
reasoner for info that requires reading the full doc.
 2026-05-14 13:19:38 +00:00
stirlingbot[bot]GitHubstirlingbot[bot] <195170888+stirlingbot[bot]@users.noreply.github.com>Anthony Stirling
 		8a59c10f42 Update Backend 3rd Party Licenses (#6312) 
Auto-generated by stirlingbot[bot]

This PR updates the backend license report based on dependency changes.

---------

Signed-off-by: stirlingbot[bot] <stirlingbot[bot]@users.noreply.github.com>
Co-authored-by: stirlingbot[bot] <195170888+stirlingbot[bot]@users.noreply.github.com>
Co-authored-by: Anthony Stirling <[email protected]>
 2026-05-13 16:30:58 +01:00
Anthony StirlingandGitHub d62f2ad3ed unoserver docker (#6328) 
# Description of Changes

<!--
Please provide a summary of the changes, including:

- What was changed
- Why the change was made
- Any challenges encountered

Closes #(issue_number)
-->

---

## Checklist

### General

- [ ] I have read the [Contribution
Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md)
- [ ] I have read the [Stirling-PDF Developer
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md)
(if applicable)
- [ ] I have read the [How to add new languages to
Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md)
(if applicable)
- [ ] I have performed a self-review of my own code
- [ ] My changes generate no new warnings

### Documentation

- [ ] I have updated relevant docs on [Stirling-PDF's doc
repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/)
(if functionality has heavily changed)
- [ ] I have read the section [Add New Translation
Tags](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md#add-new-translation-tags)
(for new translation tags only)

### Translations (if applicable)

- [ ] I ran
[`scripts/counter_translation.py`](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/docs/counter_translation.md)

### UI Changes (if applicable)

- [ ] Screenshots or videos demonstrating the UI changes are attached
(e.g., as comments or direct attachments in the PR)

### Testing (if applicable)

- [ ] I have run `task check` to verify linters, typechecks, and tests
pass
- [ ] I have tested my changes locally. Refer to the [Testing
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md#7-testing)
for more details.
 2026-05-12 13:22:15 +01:00
Anthony StirlingGitHubJames Brunton
 		f60a075443 Add Playwright/bootRun/test.sh tasks (#6244) 
## Description

Consolidates Playwright running under cohesive Task namespaces, isolates
Playwright state from the developer's local working tree, and swaps CI's
frontend webserver from `vite` dev to `vite preview` against a pre-built
`dist/`.

### `e2e:*` namespace

Renames `.taskfiles/testing.yml` to `.taskfiles/e2e.yml` and
consolidates everything Playwright-related under one `e2e:` namespace:

- `e2e:stubbed` / `e2e:live` / `e2e:enterprise` / `e2e:cross-browser`:
project-specific runners
- `e2e:check` (no-Docker subset) and `e2e:check:all` (full)
- `e2e:oauth:up` / `:down`, `e2e:saml:up` / `:down`: symmetric lifecycle
for the keycloak compose stacks
- `e2e:install`: Playwright browser install
- `docker:test`: full Docker integration suite

The redundant `frontend:test:e2e:*` project shortcuts are removed. CI
workflows (`e2e-stubbed.yml`, `e2e-live.yml`, `build-enterprise.yml`,
`nightly.yml`) are updated to call the new task names.

### Isolated Playwright state

New `STIRLING_BASE_PATH` (and `-Dstirling.base-path=`) override in
`InstallationPathConfig` redirects the entire state tree (configs,
backups, customFiles, pipeline, logs) at startup. `task e2e:live` points
it at `.test-state/playwright/` (purged on every invocation) so the
suite never touches the developer's local DB, settings.yml or backups.

`task e2e:live` auto-spawns gradle, waits for `/api/v1/info/status` to
come up, runs Playwright, then tears down the whole backend process
tree.

### CI runs Playwright against `vite preview`

Builds the frontend up-front with `VITE_BUILD_FOR_PREVIEW=1` (forces
absolute base so deep SPA routes resolve `/assets/...`) and the
playwright `webServer` now uses `vite preview --port 5173 --strictPort`
in CI. Avoids the per-page on-demand transform cost that was blowing the
30s navigation timeout under `--workers=3` on
`all-tool-pages-load.spec.ts`. Local dev keeps `vite` dev for HMR.

### OAuth/SAML compose helpers

`start-oauth-test.sh` and `start-saml-test.sh` gain a `--license-key
<KEY>` (`-k`) flag so CI and scripted runs can skip the interactive
license prompt. `start-oauth-test.sh` also moves from `for arg in "$@"`
to a `while`-with-`shift` arg loop to support multi-arg flags
consistently with the SAML script.

### Backend gradlew unification

Drops the per-platform `cmd /c gradlew.bat` branches from `backend.yml`
and routes every gradle invocation through `bash gradlew`. Works
uniformly on Linux/macOS and Windows-with-Git-Bash.

### Compare.tsx flake fix (re-land of
[#6316](https://github.com/Stirling-Tools/Stirling-PDF/pull/6316))

Piggybacks Anthony's never-merged fix from #6316. Without it,
`e2e:stubbed` continues to flake under `--workers=3` on
`compare.spec.ts`'s second-upload case via a React "Maximum update depth
exceeded" infinite loop in the Compare auto-fill effect. CI traces from
recent failed runs match exactly; 10 local runs of `compare.spec.ts`
with `CI=1 --workers=3` pass cleanly with the fix applied.

---------

Co-authored-by: James Brunton <[email protected]>
 2026-05-11 14:50:07 +00:00
James BruntonGitHubEthanHealy01
 		575684ee4b Add edit text support to stirling engine (#6245) 
# Description of Changes
Hooks up the (alpha) PDF Editor backend to the AI engine Edit Agent via
an intermediary API which is easier for the agent to call. It suffers
from all the same issues that the PDF Editor does in actually editing
the text, but should also benefit from any fixes to that.

It also adds protection against the underlying tools misbehaving by
hanging, and fixes a hanging bug in the PDF Editor.

---------

Co-authored-by: EthanHealy01 <[email protected]>
 2026-05-11 09:57:41 +00:00
LudyandGitHub f61121953e fix: replace deprecated payload too large status (#6336) 
# Description of Changes

- Replaced the deprecated `HttpStatus.PAYLOAD_TOO_LARGE` usage with
`HttpStatus.CONTENT_TOO_LARGE` in `GlobalExceptionHandler`
- Updated the upload-size exception response creation and response
status to use the newer HTTP 413 enum
- Updated related unit tests to assert `HttpStatus.CONTENT_TOO_LARGE`
- The change was made to avoid using the deprecated Spring HTTP status
enum while keeping the returned status code behavior unchanged


---

## Checklist

### General

- [ ] I have read the [Contribution
Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md)
- [ ] I have read the [Stirling-PDF Developer
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md)
(if applicable)
- [ ] I have read the [How to add new languages to
Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md)
(if applicable)
- [ ] I have performed a self-review of my own code
- [ ] My changes generate no new warnings

### Documentation

- [ ] I have updated relevant docs on [Stirling-PDF's doc
repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/)
(if functionality has heavily changed)
- [ ] I have read the section [Add New Translation
Tags](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md#add-new-translation-tags)
(for new translation tags only)

### Translations (if applicable)

- [ ] I ran
[`scripts/counter_translation.py`](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/docs/counter_translation.md)

### UI Changes (if applicable)

- [ ] Screenshots or videos demonstrating the UI changes are attached
(e.g., as comments or direct attachments in the PR)

### Testing (if applicable)

- [ ] I have run `task check` to verify linters, typechecks, and tests
pass
- [ ] I have tested my changes locally. Refer to the [Testing
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md#7-testing)
for more details.
 2026-05-11 10:38:19 +01:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 		bd50e2e59d build(deps): bump commons-io:commons-io from 2.21.0 to 2.22.0 in /app/core (#6344) 
Bumps commons-io:commons-io from 2.21.0 to 2.22.0.


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=commons-io:commons-io&package-manager=gradle&previous-version=2.21.0&new-version=2.22.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-05-11 10:37:09 +01:00