mirror of https://github.com/arsvendg/Stirling-PDF.git synced 2026-07-14 10:34:06 +02:00

Files

T

ConnorYohandGitHub ff96a80947 PAYG B-3 / S-3: cucumber suite for shadow-mode flows + CI workflow (#6522 )

## What this PR is

End-to-end cucumber coverage for the PAYG shadow charging engine (the
filter + interceptor stack from #6519), wired into CI via a new
`docker-compose-tests-saas.yml` workflow that runs only on PAYG-touching
PRs.

Stacked on #6519.

## Automated scenarios (run by `docker-compose-tests-saas.yml`)

See
[`testing/cucumber/features/payg/shadow_charges.feature`](../tree/payg-s3-cucumber/testing/cucumber/features/payg/shadow_charges.feature):

| Scenario | Validates |
|---|---|
| First tool call writes a CHARGED row | Filter + interceptor fire
end-to-end |
| Lineage join — second call on output | `JobService.joinOrOpen`
matching; no new shadow row |
| 4xx leaves the row CHARGED | "Customer paid for the attempt" semantics
|
| ZIP-returning tool records per-PDF OUTPUT | `PaygOutputExtractor`
unpacks + records signatures |
| Multi-file input writes a single shadow row | Multi-input group sizing
|
| `X-Stirling-Automation` sets PIPELINE source | Header → `JobSource`
detection |

All 6 run locally via `./testing/test-payg.sh` and will run on CI for
any PR that touches `app/saas/**`, the PAYG cucumber features, the saas
compose stack, or the workflow itself.

## Manual-only scenarios — documented in design doc, not in this suite

Two parts of the shadow engine are deliberately not automated; the
engine paths are unit-tested in
`PaygChargeInterceptorTest.afterCompletion_5xx_opened_*`, and the manual
procedures (which require a temporary throw endpoint or a container
restart with a flag flipped) live in [`notes/PAYG_DESIGN.md` §7.5.2
"PAYG cucumber: manual-only
scenarios"](../tree/payg-s3-cucumber/notes/PAYG_DESIGN.md).

- **5xx first-step failure → REFUNDED + CLOSED.** No reliably-5xx-ing
endpoint exists; manual procedure adds a throw endpoint, runs, asserts,
removes.
- **Kill-switch (`PAYG_FILTER_ENABLED=false`).** Needs a container
restart mid-suite; manual procedure tears down, flips env, brings up,
asserts zero shadow rows.

If either gets a hot-reload path (test-only throw endpoint shipped
behind a profile gate, or admin endpoint for the kill switch), automate
it in a follow-up and drop the manual procedure.

## CI workflow

`.github/workflows/docker-compose-tests-saas.yml` (new) —
self-contained, not wired into `build.yml`'s `files-changed` matrix so
the saas-cucumber job fails and succeeds independently. Triggers only on
PAYG-relevant paths. No JaCoCo coverage in v1 (saas compose doesn't have
the coverage override; can add later).

## Test infrastructure (recap)

- **`testing/compose/docker-compose-saas.yml`** — Stirling-PDF backend
with `STIRLING_FLAVOR=saas` + Postgres holding the `stirling_pdf`
schema. Supabase JWT auto-config disabled; API-key auth via
`SECURITY_CUSTOMGLOBALAPIKEY` is the live path the cucumber tests
exercise.
- **`testing/compose/payg/saas-init.sql`** + **`saas-seed.sql`** —
schema bootstrap + idempotent seed (team / user / wallet_policy).
- **`testing/cucumber/features/payg/shadow_charges.feature`** — the 6
scenarios above.
- **`testing/cucumber/features/steps/payg_step_definitions.py`** — step
defs using `requests` (HTTP) + `psycopg` (direct DB inspection). Direct
DB reads are deliberate — we want to see the filter's side effects, not
relay them through another API layer.
- **`testing/test-payg.sh`** — companion runner to `testing/test.sh`.
Brings up the saas compose, waits for health, seeds, runs behave, tears
down.
- **`behave.ini`** excludes `features/payg` from the default behave run
(the saas-cucumber CI job invokes it explicitly).

## Why a separate harness from `testing/test.sh`

The existing `test.sh` covers the proprietary-flavour stack (no PAYG
tables, no saas profile). Coupling two CI matrices that fail and succeed
independently into one script is asking for trouble. Keep the
saas-cucumber job focused on its own concerns; once the harness is
mature, the wider team can decide whether to merge them.

## Tracked in

`notes/PAYG_DESIGN.md` §7.5 (PR-S3) + §7.5.2 (manual scenarios).

2026-06-09 14:47:40 +00:00

2.5 KiB

Raw Blame History

PAYG cucumber scenarios

End-to-end coverage for the PAYG shadow charging engine (PR #6519 / PR-S3 in notes/PAYG_DESIGN.md §7.5).

Running locally

./testing/test-payg.sh

That script:

Boots testing/compose/docker-compose-saas.yml (Stirling-PDF with STIRLING_FLAVOR=saas + a Postgres holding the stirling_pdf schema)
Waits for backend health
Pipes testing/compose/payg/saas-seed.sql into the test postgres, creating a payg-cucumber-team flipped to wallet_policy.engine = 'PAYG_SHADOW' and a test user with API key payg-cucumber-key
Runs python -m behave features/payg
Tears the stack down

What's covered (automated, run by `docker-compose-tests-saas.yml` on CI)

Scenario	Validates
First tool call writes a CHARGED row	Filter + interceptor fire end-to-end; shadow row written
Lineage join — second call on output	`JobService.joinOrOpen` lineage matching; no new shadow row
First-step 5xx refunds + closes the process	`markFirstStepFailed` flips row to REFUNDED, job to CLOSED
4xx leaves the row CHARGED	"customer paid for the attempt" semantics
ZIP-returning tool records per-PDF OUTPUT	`PaygOutputExtractor` unpacks + records signatures
Multi-file input writes a single shadow row	Multi-input group sizing
`X-Stirling-Automation` sets PIPELINE source	Header → `JobSource` detection

The 5xx scenario drives the refund path through PaygCucumberThrowController — a @Profile("payg-cucumber") stub in app/saas/.../payg/test/ that always throws. The profile is activated by docker-compose-saas.yml's SPRING_PROFILES_ACTIVE=saas,payg-cucumber and is never set in production.

Manual-only scenarios

One part of the shadow engine can't reasonably be driven from this suite and is verified by hand each time its code path changes. The procedure lives in notes/PAYG_DESIGN.md §7.5.2 "PAYG cucumber: manual-only scenarios".

Kill-switch (PAYG_FILTER_ENABLED=false). Needs a docker-container restart mid-suite; orchestrating that in behave is more harness fragility than it's worth for a flag that's only flipped during incident response.

Fixtures

The scenarios reuse testing/cucumber/exampleFiles/:

ghost1.pdf — single-page reference PDF
tables.pdf — multi-page input for split / ZIP scenarios

If those filenames change in the main cucumber harness, update features/steps/payg_step_definitions.py SINGLE_PAGE_PDF / THREE_PAGE_PDF constants to match.

2.5 KiB Raw Blame History