From ff96a80947bca821fa264063560a164c3af66688 Mon Sep 17 00:00:00 2001 From: ConnorYoh <40631091+ConnorYoh@users.noreply.github.com> Date: Tue, 9 Jun 2026 15:47:40 +0100 Subject: [PATCH] PAYG B-3 / S-3: cucumber suite for shadow-mode flows + CI workflow (#6522) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## What this PR is End-to-end cucumber coverage for the PAYG shadow charging engine (the filter + interceptor stack from #6519), wired into CI via a new `docker-compose-tests-saas.yml` workflow that runs only on PAYG-touching PRs. Stacked on #6519. ## Automated scenarios (run by `docker-compose-tests-saas.yml`) See [`testing/cucumber/features/payg/shadow_charges.feature`](../tree/payg-s3-cucumber/testing/cucumber/features/payg/shadow_charges.feature): | Scenario | Validates | |---|---| | First tool call writes a CHARGED row | Filter + interceptor fire end-to-end | | Lineage join — second call on output | `JobService.joinOrOpen` matching; no new shadow row | | 4xx leaves the row CHARGED | "Customer paid for the attempt" semantics | | ZIP-returning tool records per-PDF OUTPUT | `PaygOutputExtractor` unpacks + records signatures | | Multi-file input writes a single shadow row | Multi-input group sizing | | `X-Stirling-Automation` sets PIPELINE source | Header → `JobSource` detection | All 6 run locally via `./testing/test-payg.sh` and will run on CI for any PR that touches `app/saas/**`, the PAYG cucumber features, the saas compose stack, or the workflow itself. ## Manual-only scenarios — documented in design doc, not in this suite Two parts of the shadow engine are deliberately not automated; the engine paths are unit-tested in `PaygChargeInterceptorTest.afterCompletion_5xx_opened_*`, and the manual procedures (which require a temporary throw endpoint or a container restart with a flag flipped) live in [`notes/PAYG_DESIGN.md` §7.5.2 "PAYG cucumber: manual-only scenarios"](../tree/payg-s3-cucumber/notes/PAYG_DESIGN.md). - **5xx first-step failure → REFUNDED + CLOSED.** No reliably-5xx-ing endpoint exists; manual procedure adds a throw endpoint, runs, asserts, removes. - **Kill-switch (`PAYG_FILTER_ENABLED=false`).** Needs a container restart mid-suite; manual procedure tears down, flips env, brings up, asserts zero shadow rows. If either gets a hot-reload path (test-only throw endpoint shipped behind a profile gate, or admin endpoint for the kill switch), automate it in a follow-up and drop the manual procedure. ## CI workflow `.github/workflows/docker-compose-tests-saas.yml` (new) — self-contained, not wired into `build.yml`'s `files-changed` matrix so the saas-cucumber job fails and succeeds independently. Triggers only on PAYG-relevant paths. No JaCoCo coverage in v1 (saas compose doesn't have the coverage override; can add later). ## Test infrastructure (recap) - **`testing/compose/docker-compose-saas.yml`** — Stirling-PDF backend with `STIRLING_FLAVOR=saas` + Postgres holding the `stirling_pdf` schema. Supabase JWT auto-config disabled; API-key auth via `SECURITY_CUSTOMGLOBALAPIKEY` is the live path the cucumber tests exercise. - **`testing/compose/payg/saas-init.sql`** + **`saas-seed.sql`** — schema bootstrap + idempotent seed (team / user / wallet_policy). - **`testing/cucumber/features/payg/shadow_charges.feature`** — the 6 scenarios above. - **`testing/cucumber/features/steps/payg_step_definitions.py`** — step defs using `requests` (HTTP) + `psycopg` (direct DB inspection). Direct DB reads are deliberate — we want to see the filter's side effects, not relay them through another API layer. - **`testing/test-payg.sh`** — companion runner to `testing/test.sh`. Brings up the saas compose, waits for health, seeds, runs behave, tears down. - **`behave.ini`** excludes `features/payg` from the default behave run (the saas-cucumber CI job invokes it explicitly). ## Why a separate harness from `testing/test.sh` The existing `test.sh` covers the proprietary-flavour stack (no PAYG tables, no saas profile). Coupling two CI matrices that fail and succeed independently into one script is asking for trouble. Keep the saas-cucumber job focused on its own concerns; once the harness is mature, the wider team can decide whether to merge them. ## Tracked in `notes/PAYG_DESIGN.md` §7.5 (PR-S3) + §7.5.2 (manual scenarios). --- .../workflows/docker-compose-tests-saas.yml | 136 +++++ .../saas/payg/filter/PaygOutputExtractor.java | 43 +- .../test/PaygCucumberThrowController.java | 78 +++ .../payg/filter/PaygOutputExtractorTest.java | 55 ++ testing/compose/docker-compose-saas.yml | 104 ++++ testing/compose/payg/saas-init.sql | 25 + testing/compose/payg/saas-seed.sql | 118 +++++ testing/cucumber/behave.ini | 10 +- testing/cucumber/features/payg/README.md | 58 +++ .../features/payg/shadow_charges.feature | 105 ++++ .../features/steps/payg_step_definitions.py | 473 ++++++++++++++++++ testing/cucumber/requirements.in | 9 + testing/cucumber/requirements.txt | 91 +++- testing/test-payg.sh | 89 ++++ 14 files changed, 1376 insertions(+), 18 deletions(-) create mode 100644 .github/workflows/docker-compose-tests-saas.yml create mode 100644 app/saas/src/main/java/stirling/software/saas/payg/test/PaygCucumberThrowController.java create mode 100644 testing/compose/docker-compose-saas.yml create mode 100644 testing/compose/payg/saas-init.sql create mode 100644 testing/compose/payg/saas-seed.sql create mode 100644 testing/cucumber/features/payg/README.md create mode 100644 testing/cucumber/features/payg/shadow_charges.feature create mode 100644 testing/cucumber/features/steps/payg_step_definitions.py create mode 100644 testing/test-payg.sh diff --git a/.github/workflows/docker-compose-tests-saas.yml b/.github/workflows/docker-compose-tests-saas.yml new file mode 100644 index 000000000..c4e2a0834 --- /dev/null +++ b/.github/workflows/docker-compose-tests-saas.yml @@ -0,0 +1,136 @@ +name: Docker Compose Cucumber tests (saas / PAYG) + +# Self-contained CI job for the PAYG shadow-mode cucumber scenarios. +# Triggers only on PAYG-relevant paths so we don't add CI minutes to every PR +# that doesn't touch the saas flavour. +# +# Companion to `docker-compose-tests.yml` (which runs against the +# proprietary-flavour stack and skips features/payg via behave.ini's +# exclude_re). Kept as a separate workflow so the saas matrix can fail and +# succeed independently without touching the main cucumber harness. + +on: + pull_request: + paths: + - "app/saas/**" + - "testing/cucumber/features/payg/**" + - "testing/cucumber/features/steps/payg_step_definitions.py" + - "testing/cucumber/requirements.txt" + - "testing/compose/docker-compose-saas.yml" + - "testing/compose/payg/**" + - "testing/test-payg.sh" + - ".github/workflows/docker-compose-tests-saas.yml" + push: + branches: [main] + paths: + - "app/saas/**" + - "testing/cucumber/features/payg/**" + - "testing/cucumber/features/steps/payg_step_definitions.py" + - "testing/cucumber/requirements.txt" + - "testing/compose/docker-compose-saas.yml" + - "testing/compose/payg/**" + - "testing/test-payg.sh" + - ".github/workflows/docker-compose-tests-saas.yml" + +permissions: + contents: read + +jobs: + pick: + uses: ./.github/workflows/_runner-pick.yml + + docker-compose-tests-saas: + needs: pick + runs-on: ${{ needs.pick.outputs.is_fork == 'true' && 'ubuntu-latest' || 'depot-ubuntu-24.04-4' }} + permissions: + actions: write + contents: read + checks: write + env: + DEPOT_TOKEN: ${{ secrets.DEPOT_TOKEN }} + + steps: + - name: Harden Runner + uses: step-security/harden-runner@ab7a9404c0f3da075243ca237b5fac12c98deaa5 # v2.19.3 + with: + egress-policy: audit + + - name: Checkout Repository + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + + - name: Set up JDK 25 + uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654 # v5.2.0 + with: + java-version: "25" + distribution: "temurin" + + - name: Cache Gradle dependency artifacts + uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5 + with: + path: | + ~/.gradle/wrapper + ~/.gradle/caches/modules-2/files-2.1 + ~/.gradle/caches/modules-2/metadata-2.* + key: gradle-deps-saas-${{ runner.os }}-jdk-25-${{ hashFiles('**/gradle/wrapper/gradle-wrapper.properties', '**/*.gradle', '**/*.gradle.kts', 'settings.gradle', 'settings.gradle.kts', 'gradle/libs.versions.toml') }} + + - name: Setup Gradle + uses: gradle/actions/setup-gradle@50e97c2cd7a37755bbfafc9c5b7cafaece252f6e # v6.1.0 + with: + gradle-version: 9.3.1 + cache-disabled: true + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0 + + - name: Expose GitHub runtime for Buildx cache + uses: crazy-max/ghaction-github-runtime@04d248b84655b509d8c44dc1d6f990c879747487 # v4.0.0 + + # No "Install Docker Compose" step: Ubuntu runners ship with `docker compose` + # v2 (built into the Docker CLI). test-payg.sh uses the v2 form throughout + # (`docker compose …`, no hyphen), so the legacy v1 `docker-compose` binary + # isn't needed. Avoids a `curl | sudo install` without checksum verification + # (Aikido flagged this when copy-pasted from docker-compose-tests.yml). + + - name: Set up Python + uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0 + with: + python-version: "3.12" + cache: "pip" + cache-dependency-path: ./testing/cucumber/requirements.txt + + - name: Pip requirements + run: | + pip install --require-hashes --only-binary=:all: -r ./testing/cucumber/requirements.txt + + - name: Run PAYG Cucumber Tests + env: + MAVEN_USER: ${{ secrets.MAVEN_USER }} + MAVEN_PASSWORD: ${{ secrets.MAVEN_PASSWORD }} + MAVEN_PUBLIC_URL: ${{ secrets.MAVEN_PUBLIC_URL }} + run: | + chmod +x ./testing/test-payg.sh + ./testing/test-payg.sh + + - name: Dump saas container logs on failure + if: failure() + run: | + docker compose -f testing/compose/docker-compose-saas.yml logs --tail 500 stirling-pdf-saas || true + docker compose -f testing/compose/docker-compose-saas.yml logs --tail 200 postgres-saas || true + + - name: Upload PAYG Cucumber Report + if: always() + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: payg-cucumber-report + path: testing/cucumber/report-payg.html + retention-days: 7 + if-no-files-found: warn + + - name: PAYG Cucumber Test Report + if: always() + uses: dorny/test-reporter@a43b3a5f7366b97d083190328d2c652e1a8b6aa2 # v3.0.0 + with: + name: PAYG Cucumber Tests + path: testing/cucumber/junit-payg/*.xml + reporter: java-junit + fail-on-error: false diff --git a/app/saas/src/main/java/stirling/software/saas/payg/filter/PaygOutputExtractor.java b/app/saas/src/main/java/stirling/software/saas/payg/filter/PaygOutputExtractor.java index a19a78e0a..86b5cb602 100644 --- a/app/saas/src/main/java/stirling/software/saas/payg/filter/PaygOutputExtractor.java +++ b/app/saas/src/main/java/stirling/software/saas/payg/filter/PaygOutputExtractor.java @@ -49,8 +49,12 @@ public class PaygOutputExtractor { /** {@code %PDF-} in ASCII. */ private static final byte[] PDF_MAGIC = {0x25, 0x50, 0x44, 0x46, 0x2D}; + /** {@code PK\x03\x04} — local file header for any non-empty ZIP. */ + private static final byte[] ZIP_MAGIC = {0x50, 0x4B, 0x03, 0x04}; + private static final String ZIP_CONTENT_TYPE = "application/zip"; private static final String PDF_CONTENT_TYPE = "application/pdf"; + private static final String OCTET_STREAM_CONTENT_TYPE = "application/octet-stream"; private final TempFileManager tempFileManager; @@ -74,11 +78,31 @@ public class PaygOutputExtractor { return List.of(); } String mediaType = stripParameters(contentType); - if (PDF_CONTENT_TYPE.equalsIgnoreCase(mediaType)) { + + // Stirling-PDF tool endpoints sometimes set Content-Type to + // application/octet-stream (or no header at all) even when the body + // is a real PDF or ZIP — Spring's default StreamingResponseBody path + // doesn't always negotiate content type. When the declared + // Content-Type is missing or generic, sniff magic bytes to recover + // lineage capture. + boolean missingOrGeneric = + mediaType == null || OCTET_STREAM_CONTENT_TYPE.equalsIgnoreCase(mediaType); + boolean isPdf = PDF_CONTENT_TYPE.equalsIgnoreCase(mediaType); + boolean isZip = ZIP_CONTENT_TYPE.equalsIgnoreCase(mediaType); + + if (missingOrGeneric) { + if (isMagic(bodyPath, PDF_MAGIC)) { + isPdf = true; + } else if (isMagic(bodyPath, ZIP_MAGIC)) { + isZip = true; + } + } + + if (isPdf) { // Wrapper-owned path. Don't claim ownership. return List.of(new ExtractedPdf(bodyPath, null)); } - if (ZIP_CONTENT_TYPE.equalsIgnoreCase(mediaType)) { + if (isZip) { return extractZip(bodyPath); } return List.of(); @@ -126,20 +150,25 @@ public class PaygOutputExtractor { } private boolean isPdfMagic(Path path) { - byte[] head = new byte[PDF_MAGIC.length]; + return isMagic(path, PDF_MAGIC); + } + + /** Returns true if the first {@code magic.length} bytes of {@code path} equal {@code magic}. */ + private boolean isMagic(Path path, byte[] magic) { + byte[] head = new byte[magic.length]; try (InputStream in = Files.newInputStream(path)) { int read = in.read(head); - if (read != PDF_MAGIC.length) { + if (read != magic.length) { return false; } - for (int i = 0; i < PDF_MAGIC.length; i++) { - if (head[i] != PDF_MAGIC[i]) { + for (int i = 0; i < magic.length; i++) { + if (head[i] != magic[i]) { return false; } } return true; } catch (IOException e) { - log.debug("PDF magic-byte check failed for {}", path, e); + log.debug("Magic-byte check failed for {}", path, e); return false; } } diff --git a/app/saas/src/main/java/stirling/software/saas/payg/test/PaygCucumberThrowController.java b/app/saas/src/main/java/stirling/software/saas/payg/test/PaygCucumberThrowController.java new file mode 100644 index 000000000..8a9082e2c --- /dev/null +++ b/app/saas/src/main/java/stirling/software/saas/payg/test/PaygCucumberThrowController.java @@ -0,0 +1,78 @@ +package stirling.software.saas.payg.test; + +import org.springframework.context.annotation.Profile; +import org.springframework.http.MediaType; +import org.springframework.http.ResponseEntity; +import org.springframework.web.bind.annotation.RequestMapping; +import org.springframework.web.bind.annotation.RequestParam; +import org.springframework.web.bind.annotation.RestController; +import org.springframework.web.multipart.MultipartFile; + +import io.swagger.v3.oas.annotations.Hidden; + +import lombok.extern.slf4j.Slf4j; + +import stirling.software.common.annotations.AutoJobPostMapping; +import stirling.software.common.enumeration.ResourceWeight; + +/** + * Cucumber-only force-5xx endpoint. Gated behind the {@code payg-cucumber} Spring profile so the + * bean never registers in production — only the PAYG cucumber compose stack activates the profile + * (via {@code SPRING_PROFILES_ACTIVE=saas,payg-cucumber} in {@code docker-compose-saas.yml}). + * + *

Purpose: drive the PAYG filter+interceptor's 5xx-first-step branch end-to-end. No reliably- + * 5xx-ing real tool endpoint exists in current Stirling — every malformed input is caught as 4xx by + * {@code GlobalExceptionHandler}. Without this stub the only way to exercise the refund path was a + * manual procedure (a temporary throw endpoint added, run, removed) documented in {@code + * notes/PAYG_DESIGN.md} §7.5.2 M1. This controller replaces that procedure with a profile- gated + * automated scenario. + * + *

{@link AutoJobPostMapping} consumes {@code multipart/form-data} so the filter's {@code + * MultipartHttpServletRequest} cast runs and the input lineage hash is computed before the + * controller throws. The PAYG filter chain therefore sees: preHandle → openProcess (CHARGED row + * written) → controller throws → afterCompletion observes status 500 → markFirstStepFailed (row → + * REFUNDED, job → CLOSED). + * + *

The thrown {@link IllegalStateException} is unwrapped by {@code GlobalExceptionHandler}'s + * RuntimeException handler — it has no IOException / IllegalArgument / BaseAppException cause, so + * it falls through to the "Unexpected RuntimeException" branch which sets HTTP 500. Don't use + * {@code ResponseStatusException} here — its dedicated handler is reached before the unwrapping + * branch but the upstream test still passes; {@link IllegalStateException} is the more honest mimic + * of a real bug-driven 500. + * + *

{@code @Hidden} keeps this endpoint out of OpenAPI / Swagger output even when the profile is + * active, so no docs leak to anyone running the cucumber stack interactively. + * + *

Return type matters: the method declares {@link ResponseEntity}{@code }, not + * {@code void}. Spring's {@code InvocableHandlerMethod} picks a return-value handler based on the + * declared return type; {@code void} matches the "no body, status 200" handler regardless of + * what the {@code @Around} advice on {@link AutoJobPostMapping} actually returns at runtime. The + * first version of this stub declared {@code void} and the 500 ResponseEntity from {@code + * JobExecutorService}'s catch block was silently discarded — the client saw a 200 with an empty + * body. Declaring {@code ResponseEntity} matches the real runtime type and lets the advice's + * 500 reach the wire. + */ +@Slf4j +@RestController +@Profile("payg-cucumber") +@RequestMapping("/api/v1/payg-cucumber") +@Hidden +public class PaygCucumberThrowController { + + @AutoJobPostMapping( + value = "/throw-500", + consumes = MediaType.MULTIPART_FORM_DATA_VALUE, + resourceWeight = ResourceWeight.SMALL_WEIGHT) + public ResponseEntity throw500( + @RequestParam(value = "fileInput", required = false) MultipartFile fileInput) { + // The file is read by the PAYG filter via getMultiFileMap() before we get here; the + // controller param is just to keep Spring's multipart binding happy. We don't touch it. + log.warn( + "PAYG cucumber forced 500 (fileInput name='{}' size={} bytes)", + fileInput != null ? fileInput.getOriginalFilename() : null, + fileInput != null ? fileInput.getSize() : 0); + throw new IllegalStateException("PAYG cucumber forced 500"); + // unreachable — kept as a type signature so AutoJobAspect's @Around return value (the + // 500 ResponseEntity from JobExecutorService) actually reaches the wire. + } +} diff --git a/app/saas/src/test/java/stirling/software/saas/payg/filter/PaygOutputExtractorTest.java b/app/saas/src/test/java/stirling/software/saas/payg/filter/PaygOutputExtractorTest.java index 201a0d68e..688377ad3 100644 --- a/app/saas/src/test/java/stirling/software/saas/payg/filter/PaygOutputExtractorTest.java +++ b/app/saas/src/test/java/stirling/software/saas/payg/filter/PaygOutputExtractorTest.java @@ -102,6 +102,61 @@ class PaygOutputExtractorTest { assertThat(extractor.extract("application/zip", zip)).isEmpty(); } + @Test + void octetStreamWithPdfMagic_treatedAsPdf(@TempDir Path tmp) throws IOException { + // Stirling tool endpoints sometimes set Content-Type to + // application/octet-stream for streamed responses even when the body + // is a real PDF. The extractor must sniff magic bytes when the + // declared Content-Type is generic. + Path body = tmp.resolve("body.bin"); + Files.write(body, pdfBytes("real-pdf-content")); + List out = + extractor.extract("application/octet-stream", body); + assertThat(out).hasSize(1); + assertThat(out.get(0).path()).isEqualTo(body); + } + + @Test + void octetStreamWithZipMagic_unpackedAsZip(@TempDir Path tmp) throws IOException { + Path zip = tmp.resolve("body.bin"); + try (ZipOutputStream zos = new ZipOutputStream(Files.newOutputStream(zip))) { + writeEntry(zos, "page1.pdf", pdfBytes("a")); + writeEntry(zos, "page2.pdf", pdfBytes("b")); + } + List out = + extractor.extract("application/octet-stream", zip); + try { + assertThat(out).hasSize(2); + } finally { + for (PaygOutputExtractor.ExtractedPdf p : out) { + p.close(); + } + } + } + + @Test + void nullContentTypeWithZipMagic_unpackedAsZip(@TempDir Path tmp) throws IOException { + Path zip = tmp.resolve("body.bin"); + try (ZipOutputStream zos = new ZipOutputStream(Files.newOutputStream(zip))) { + writeEntry(zos, "page1.pdf", pdfBytes("a")); + } + List out = extractor.extract(null, zip); + try { + assertThat(out).hasSize(1); + } finally { + for (PaygOutputExtractor.ExtractedPdf p : out) { + p.close(); + } + } + } + + @Test + void octetStreamWithoutMagic_returnsEmpty(@TempDir Path tmp) throws IOException { + Path body = tmp.resolve("body.bin"); + Files.write(body, "neither pdf nor zip".getBytes(StandardCharsets.UTF_8)); + assertThat(extractor.extract("application/octet-stream", body)).isEmpty(); + } + private static void writeEntry(ZipOutputStream zos, String name, byte[] data) throws IOException { zos.putNextEntry(new ZipEntry(name)); diff --git a/testing/compose/docker-compose-saas.yml b/testing/compose/docker-compose-saas.yml new file mode 100644 index 000000000..880518b42 --- /dev/null +++ b/testing/compose/docker-compose-saas.yml @@ -0,0 +1,104 @@ +services: + # --------------------------------------------------------------------- + # Postgres holding the stirling_pdf schema. Flyway migrates V1-V13 on + # backend startup against this DB. Exposed on host port 5433 so the + # cucumber harness can connect via psycopg from features/steps/payg_*. + # --------------------------------------------------------------------- + postgres-saas: + image: postgres:17-alpine + container_name: payg-cucumber-postgres + environment: + POSTGRES_PASSWORD: postgres + POSTGRES_DB: postgres + ports: + - "5433:5432" + volumes: + # Bootstrap stirling_pdf schema + seed test team / api key. + # Runs once when the container is first created. + - ./payg/saas-init.sql:/docker-entrypoint-initdb.d/00-init.sql:ro + healthcheck: + test: ["CMD-SHELL", "pg_isready -U postgres -d postgres"] + interval: 3s + timeout: 5s + retries: 20 + networks: + - payg-cucumber-network + + # --------------------------------------------------------------------- + # Stirling-PDF backend with STIRLING_FLAVOR=saas. Authenticates via + # API key (SECURITY_CUSTOMGLOBALAPIKEY) — Supabase JWT enforcement is + # disabled for the test profile via SAAS_DB_PROJECT_REF=disabled so the + # OAuth resource-server filter no-ops. The PaygChargeInterceptor's + # resolveUser() ApiKey-token path is what the cucumber tests exercise. + # --------------------------------------------------------------------- + stirling-pdf-saas: + build: + context: ../.. + dockerfile: docker/embedded/Dockerfile + args: + STIRLING_FLAVOR: saas + container_name: payg-cucumber-stirling + depends_on: + postgres-saas: + condition: service_healthy + restart: "no" + deploy: + resources: + limits: + memory: 4G + healthcheck: + test: ["CMD-SHELL", "curl -f http://localhost:8080/api/v1/info/status | grep -q 'UP'"] + interval: 5s + timeout: 10s + retries: 24 + ports: + - "8080:8080" + environment: + # Activate the saas profile + bring the saas subproject onto the classpath. + # `payg-cucumber` registers the test-only throw-500 controller in + # `app/saas/.../payg/test/PaygCucumberThrowController.java` — that bean is + # @Profile("payg-cucumber") so it never registers in production. + SPRING_PROFILES_ACTIVE: "saas,payg-cucumber" + STIRLING_FLAVOR: "saas" + ENABLE_SAAS: "true" + DISABLE_ADDITIONAL_FEATURES: "false" + + # Datasource — point Flyway + JPA at the test postgres. + SAAS_DB_URL: "jdbc:postgresql://postgres-saas:5432/postgres" + SAAS_DB_USERNAME: "postgres" + SAAS_DB_PASSWORD: "postgres" + SAAS_DB_PROJECT_REF: "disabled" + + # The saas Flyway migrations assume `users` and `teams` already exist + # (V2 ALTERs `users`; V5 references `teams(id)`) because in production + # those tables are provisioned by Supabase before Stirling starts. On + # a clean test postgres they don't exist, so we disable Flyway and let + # Hibernate's `ddl-auto=create-drop` build the full schema from the + # entity graph. The default-pricing-policy row that V12 seeds in + # production is re-seeded by testing/compose/payg/saas-seed.sql. + SPRING_FLYWAY_ENABLED: "false" + SPRING_JPA_HIBERNATE_DDL_AUTO: "create-drop" + + # Disable Supabase JWT enforcement for the test profile. The + # PaygChargeInterceptor resolves via ApiKey-token authority — that + # path is what we cover here. + SPRING_AUTOCONFIGURE_EXCLUDE: "org.springframework.boot.autoconfigure.security.oauth2.resource.servlet.OAuth2ResourceServerAutoConfiguration" + + # Test API key matching the user seeded by saas-init.sql. + SECURITY_CUSTOMGLOBALAPIKEY: "payg-cucumber-key" + SECURITY_ENABLELOGIN: "false" + + # Conservative defaults. The cucumber runs send small fixtures so + # the in-memory threshold never bites; the kill-switch is needed by + # one scenario but is toggled via a separate compose override. + PAYG_FILTER_ENABLED: "true" + PAYG_FILTER_RESPONSE_IN_MEMORY_THRESHOLD_BYTES: "10485760" + + SYSTEM_DEFAULTLOCALE: "en-US" + SYSTEM_MAXFILESIZE: "100" + networks: + - payg-cucumber-network + +networks: + payg-cucumber-network: + driver: bridge diff --git a/testing/compose/payg/saas-init.sql b/testing/compose/payg/saas-init.sql new file mode 100644 index 000000000..1a40cdde6 --- /dev/null +++ b/testing/compose/payg/saas-init.sql @@ -0,0 +1,25 @@ +-- Bootstrap minimal stirling_pdf state for PAYG cucumber tests. +-- +-- Runs once when the postgres-saas container is first created (loaded via +-- /docker-entrypoint-initdb.d on the official postgres image). Flyway then +-- migrates V1-V13 on backend startup, which fills in the schema; this file +-- only seeds the data rows the cucumber scenarios reference by name. +-- +-- Idempotent — if the container is re-created, this whole file runs again +-- but every INSERT is guarded against duplicates. + +CREATE SCHEMA IF NOT EXISTS stirling_pdf; + +-- Note: the actual seed of teams / users / payg_team_extensions / wallet_policy +-- happens AFTER Flyway migrations have applied. We can't insert here because +-- the tables don't exist yet at container-init time. Instead we register a +-- helper function the backend invokes once at startup (or the cucumber +-- harness invokes via psql before running scenarios). +-- +-- For "make a start", the simplest approach is to keep this file as the +-- schema bootstrap, and put the seed inserts in a separate sidecar SQL that +-- the cucumber test.sh harness pipes through psql AFTER waiting for backend +-- health. See testing/compose/payg/saas-seed.sql below. + +-- Reserved for future container-init-time schema bootstrap if we ever need it. +SELECT 1; diff --git a/testing/compose/payg/saas-seed.sql b/testing/compose/payg/saas-seed.sql new file mode 100644 index 000000000..612cbc9d1 --- /dev/null +++ b/testing/compose/payg/saas-seed.sql @@ -0,0 +1,118 @@ +-- Seed test team / payg policy / wallet_policy in PAYG_SHADOW mode for the +-- cucumber harness. Piped through psql AFTER Hibernate has built the schema +-- on backend startup (compose disables Flyway — see docker-compose-saas.yml +-- for the rationale: saas Flyway migrations assume `users` + `teams` exist +-- because Supabase normally provisions them). +-- +-- Idempotent — guarded against duplicate keys so re-running on the same +-- container is safe between scenarios. + +-- --------------------------------------------------------------------------- +-- 0. Default pricing policy + per-source step limits (V12 in production). +-- Required because PricingPolicyService.getEffectivePolicy() throws if +-- no row has is_default = TRUE. Explicit timestamps because Hibernate's +-- @CreationTimestamp is application-side; direct INSERTs bypass it. +-- --------------------------------------------------------------------------- +INSERT INTO stirling_pdf.pricing_policy ( + version, effective_from, doc_pages_per_unit, doc_bytes_per_unit, + min_charge_units, file_unit_cap, is_default, notes, created_by, + created_at +) +SELECT + 'v1-cucumber', CURRENT_TIMESTAMP, 25, 5242880, + 1, 1000, TRUE, + 'Cucumber test default policy', 'system', + CURRENT_TIMESTAMP +WHERE NOT EXISTS ( + SELECT 1 FROM stirling_pdf.pricing_policy WHERE is_default = TRUE +); + +INSERT INTO stirling_pdf.pricing_policy_step_limit (policy_id, job_source, step_limit) +SELECT p.policy_id, src.job_source, src.step_limit +FROM stirling_pdf.pricing_policy p +CROSS JOIN ( + VALUES + ('WEB', 10), + ('API', 10), + ('PIPELINE', 20), + ('DESKTOP_APP', 10) +) AS src(job_source, step_limit) +WHERE p.is_default = TRUE + AND NOT EXISTS ( + SELECT 1 FROM stirling_pdf.pricing_policy_step_limit s + WHERE s.policy_id = p.policy_id AND s.job_source = src.job_source + ); + +-- --------------------------------------------------------------------------- +-- 1. Test team. The Stirling-PDF backend auto-creates `Default` and +-- `Internal` teams at boot; we add a third one specifically for the +-- PAYG scenarios so we can isolate state and assert per-team. +-- --------------------------------------------------------------------------- +INSERT INTO stirling_pdf.teams (name) +SELECT 'payg-cucumber-team' +WHERE NOT EXISTS ( + SELECT 1 FROM stirling_pdf.teams WHERE name = 'payg-cucumber-team' +); + +-- --------------------------------------------------------------------------- +-- 2. payg_team_extensions sidecar — uses the default pricing policy. +-- --------------------------------------------------------------------------- +INSERT INTO stirling_pdf.payg_team_extensions (team_id, pricing_policy_id, stripe_customer_id) +SELECT t.team_id, NULL, NULL +FROM stirling_pdf.teams t +WHERE t.name = 'payg-cucumber-team' + AND NOT EXISTS ( + SELECT 1 FROM stirling_pdf.payg_team_extensions ext WHERE ext.team_id = t.team_id + ); + +-- --------------------------------------------------------------------------- +-- 3. Bind the auto-created CUSTOM_API_USER to our cucumber team. The user +-- is created by the backend's SECURITY_CUSTOMGLOBALAPIKEY handling — we +-- don't seed our own user (would collide on the unique api_key). +-- Update their primary `team_id` so JobChargeService picks up +-- payg-cucumber-team as the owner team on requests. +-- --------------------------------------------------------------------------- +UPDATE stirling_pdf.users u +SET team_id = (SELECT team_id FROM stirling_pdf.teams WHERE name = 'payg-cucumber-team') +WHERE u.username = 'CUSTOM_API_USER'; + +-- --------------------------------------------------------------------------- +-- 4. Team membership row so /teams/* admin paths recognise the user. +-- All timestamp columns set explicitly because Hibernate-generated DDL +-- doesn't carry the Flyway-migration DEFAULT CURRENT_TIMESTAMP. +-- --------------------------------------------------------------------------- +INSERT INTO stirling_pdf.team_memberships ( + team_id, user_id, role, invited_at, created_at, updated_at +) +SELECT t.team_id, u.user_id, 'LEADER', + CURRENT_TIMESTAMP, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP +FROM stirling_pdf.teams t, stirling_pdf.users u +WHERE t.name = 'payg-cucumber-team' + AND u.username = 'CUSTOM_API_USER' + AND NOT EXISTS ( + SELECT 1 FROM stirling_pdf.team_memberships m + WHERE m.team_id = t.team_id AND m.user_id = u.user_id + ); + +-- --------------------------------------------------------------------------- +-- 5. wallet_policy in PAYG_SHADOW mode. +-- auto_group_strategy is the dead column tracked for drop in PR-R11 +-- (PAYG_DESIGN.md §7.7) — until that ships, the NOT NULL constraint +-- means we have to populate it. 'AUTO' is the prod default. +-- --------------------------------------------------------------------------- +INSERT INTO stirling_pdf.wallet_policy ( + team_id, engine, cap_period, warn_at_pct, degrade_at_pct, + degraded_feature_set, auto_group_strategy, notification_emails, + updated_at +) +SELECT t.team_id, 'PAYG_SHADOW', 'CALENDAR_MONTH', 80, 100, + 'MINIMAL', 'AUTO', '[]'::jsonb, CURRENT_TIMESTAMP +FROM stirling_pdf.teams t +WHERE t.name = 'payg-cucumber-team' + AND NOT EXISTS ( + SELECT 1 FROM stirling_pdf.wallet_policy wp WHERE wp.team_id = t.team_id + ); + +UPDATE stirling_pdf.wallet_policy +SET engine = 'PAYG_SHADOW', updated_at = CURRENT_TIMESTAMP +WHERE team_id = (SELECT team_id FROM stirling_pdf.teams WHERE name = 'payg-cucumber-team'); diff --git a/testing/cucumber/behave.ini b/testing/cucumber/behave.ini index 202b26576..b398282b2 100644 --- a/testing/cucumber/behave.ini +++ b/testing/cucumber/behave.ini @@ -5,4 +5,12 @@ # # python -m behave features/enterprise # -exclude_re = features/enterprise +# PAYG shadow-mode features live in features/payg/. They need the saas +# profile + Postgres + seeded test team (testing/compose/docker-compose-saas.yml). +# The default run excludes them; the saas-cucumber CI job invokes +# behave with `features/payg` explicitly. The PAYG harness in +# testing/test-payg.sh swaps a temp behave.ini that drops the payg +# exclusion + tags @manual to skip restart-hook scenarios. +# +exclude_re = features/(enterprise|payg) +tags = ~@manual diff --git a/testing/cucumber/features/payg/README.md b/testing/cucumber/features/payg/README.md new file mode 100644 index 000000000..584ef446a --- /dev/null +++ b/testing/cucumber/features/payg/README.md @@ -0,0 +1,58 @@ +# PAYG cucumber scenarios + +End-to-end coverage for the PAYG shadow charging engine (PR #6519 / PR-S3 +in `notes/PAYG_DESIGN.md` §7.5). + +## Running locally + +```bash +./testing/test-payg.sh +``` + +That script: +1. Boots `testing/compose/docker-compose-saas.yml` (Stirling-PDF with + `STIRLING_FLAVOR=saas` + a Postgres holding the `stirling_pdf` schema) +2. Waits for backend health +3. Pipes `testing/compose/payg/saas-seed.sql` into the test postgres, + creating a `payg-cucumber-team` flipped to `wallet_policy.engine = + 'PAYG_SHADOW'` and a test user with API key `payg-cucumber-key` +4. Runs `python -m behave features/payg` +5. Tears the stack down + +## What's covered (automated, run by `docker-compose-tests-saas.yml` on CI) + +| Scenario | Validates | +|---|---| +| First tool call writes a CHARGED row | Filter + interceptor fire end-to-end; shadow row written | +| Lineage join — second call on output | `JobService.joinOrOpen` lineage matching; no new shadow row | +| First-step 5xx refunds + closes the process | `markFirstStepFailed` flips row to REFUNDED, job to CLOSED | +| 4xx leaves the row CHARGED | "customer paid for the attempt" semantics | +| ZIP-returning tool records per-PDF OUTPUT | `PaygOutputExtractor` unpacks + records signatures | +| Multi-file input writes a single shadow row | Multi-input group sizing | +| `X-Stirling-Automation` sets PIPELINE source | Header → `JobSource` detection | + +The 5xx scenario drives the refund path through `PaygCucumberThrowController` +— a `@Profile("payg-cucumber")` stub in `app/saas/.../payg/test/` that always +throws. The profile is activated by `docker-compose-saas.yml`'s +`SPRING_PROFILES_ACTIVE=saas,payg-cucumber` and is never set in production. + +## Manual-only scenarios + +One part of the shadow engine can't reasonably be driven from this suite +and is verified by hand each time its code path changes. The procedure +lives in `notes/PAYG_DESIGN.md` §7.5.2 "PAYG cucumber: manual-only +scenarios". + +- **Kill-switch (`PAYG_FILTER_ENABLED=false`).** Needs a docker-container + restart mid-suite; orchestrating that in behave is more harness fragility + than it's worth for a flag that's only flipped during incident response. + +## Fixtures + +The scenarios reuse `testing/cucumber/exampleFiles/`: +- `ghost1.pdf` — single-page reference PDF +- `tables.pdf` — multi-page input for split / ZIP scenarios + +If those filenames change in the main cucumber harness, update +`features/steps/payg_step_definitions.py` SINGLE_PAGE_PDF / THREE_PAGE_PDF +constants to match. diff --git a/testing/cucumber/features/payg/shadow_charges.feature b/testing/cucumber/features/payg/shadow_charges.feature new file mode 100644 index 000000000..2141b66b0 --- /dev/null +++ b/testing/cucumber/features/payg/shadow_charges.feature @@ -0,0 +1,105 @@ +Feature: PAYG shadow-mode charging + # End-to-end coverage for the PAYG shadow charging engine via the filter + + # interceptor stack landed in PR #6519. Runs against the saas-profile + # docker-compose target (testing/compose/docker-compose-saas.yml). + # + # Each scenario sets up a test team in PAYG_SHADOW mode, hits a real tool + # endpoint, then asserts the shape of the rows written to payg_shadow_charge + # and processing_job / processing_job_step. + # + # Lives under features/payg so it's only loaded when the saas-cucumber job + # explicitly includes this directory (separate from the main behave run + # which boots the proprietary-flavor stack and has no PAYG tables). + + Background: + Given the SaaS stack is running with PAYG enabled + And team "payg-cucumber-team" exists with wallet_policy.engine = "PAYG_SHADOW" + And I am authenticated as a member of team "payg-cucumber-team" + + Scenario: First tool call writes a CHARGED shadow row + Given there are no existing shadow charges for team "payg-cucumber-team" + When I POST a single-page PDF to "/api/v1/security/add-password" + Then the response status is 200 + And exactly 1 shadow charge row exists for team "payg-cucumber-team" + And the latest shadow charge row has status "CHARGED" + And the latest shadow charge row has payg_units >= 1 + And the latest shadow charge row's job is OPEN + And the latest job has 1 step recorded with status "OK" + + Scenario: Lineage join — second call on the same output joins the first process + Given there are no existing shadow charges for team "payg-cucumber-team" + When I POST a single-page PDF to "/api/v1/security/add-password" + And I take the response body as "step1-output" + And I POST "step1-output" to "/api/v1/security/sanitize-pdf" + Then exactly 1 shadow charge row exists for team "payg-cucumber-team" + # The second call joined the first process — no new shadow row + And the latest job has step_count = 2 + And the latest job is OPEN + + Scenario: First-step 5xx refunds the shadow row + closes the process + # No reliably-5xx-ing real tool endpoint exists (every malformed input is + # caught as 4xx by GlobalExceptionHandler), so we drive the refund path + # through PaygCucumberThrowController — a @Profile("payg-cucumber") stub + # that always throws IllegalStateException → 500. The profile is active + # only inside docker-compose-saas.yml, never in production. + Given there are no existing shadow charges for team "payg-cucumber-team" + When I POST a single-page PDF to "/api/v1/payg-cucumber/throw-500" + Then the response status is 500 + And exactly 1 shadow charge row exists for team "payg-cucumber-team" + And the latest shadow charge row has status "REFUNDED" + And the latest shadow charge row's refunded_at is not null + And the latest shadow charge row's refund_reason starts with "first-step-5xx" + And the latest job is CLOSED + And the latest job has 1 step recorded with status "FAILED" + + # ────────────────────────────────────────────────────────────────────────── + # NOTE: The kill-switch scenario (PAYG_FILTER_ENABLED=false) is NOT + # automated — it requires restarting the stirling-pdf-saas container with + # a different env var mid-run, which couples test setup to compose state + # more than it's worth for a flag that only flips during incident response. + # See notes/PAYG_DESIGN.md §7.5.2 "PAYG cucumber: manual-only scenarios". + # ────────────────────────────────────────────────────────────────────────── + + Scenario: 4xx leaves the shadow row CHARGED (customer pays for the attempt) + # /sanitize-pdf on an encrypted PDF without the password reliably 400s + # via GlobalExceptionHandler's PdfPasswordException → ProblemDetail path. + # We chain: first call encrypts a PDF (CHARGED), second call tries to + # sanitize WITHOUT the password and 400s. The 4xx assertion is on the + # SECOND call's behaviour. + Given there are no existing shadow charges for team "payg-cucumber-team" + When I POST a single-page PDF to "/api/v1/security/add-password" + And I take the response body as "encrypted" + And I POST "encrypted" to "/api/v1/security/sanitize-pdf" + Then the response status is >= 400 and < 500 + # Note: shadow_charges count is 1 because the second call lineage-joins + # the first (its input matches the first's output). The 4xx therefore + # appears as a FAILED step on the existing process, not a new shadow row. + And exactly 1 shadow charge row exists for team "payg-cucumber-team" + And the latest shadow charge row has status "CHARGED" + And the latest job has step_count = 2 + And the latest step's error_code matches the response status + + Scenario: ZIP-returning tool records OUTPUT signatures per inner PDF + # /split-pages with multiple page numbers returns a ZIP. Stirling sends + # application/octet-stream rather than application/zip — the extractor + # sniffs the PK\x03\x04 magic so the ZIP unpack path still fires. + Given there are no existing shadow charges for team "payg-cucumber-team" + When I POST a 3-page PDF to "/api/v1/general/split-pages" with form fields: + | pageNumbers | 1,2 | + Then the response status is 200 + And exactly 1 shadow charge row exists for team "payg-cucumber-team" + And the latest job has at least 2 OUTPUT artifact hashes recorded + + Scenario: Multi-file input writes a single shadow row sized by the group + Given there are no existing shadow charges for team "payg-cucumber-team" + When I POST two single-page PDFs as a multi-file payload to "/api/v1/general/merge-pdfs" + Then the response status is 200 + And exactly 1 shadow charge row exists for team "payg-cucumber-team" + And the latest shadow charge row has payg_units >= 1 + + Scenario: PIPELINE header sets the job source + Given there are no existing shadow charges for team "payg-cucumber-team" + When I POST a single-page PDF with header "X-Stirling-Automation: true" to "/api/v1/security/add-password" + Then the response status is 200 + And exactly 1 shadow charge row exists for team "payg-cucumber-team" + And the latest job's source is "PIPELINE" diff --git a/testing/cucumber/features/steps/payg_step_definitions.py b/testing/cucumber/features/steps/payg_step_definitions.py new file mode 100644 index 000000000..5069f9a00 --- /dev/null +++ b/testing/cucumber/features/steps/payg_step_definitions.py @@ -0,0 +1,473 @@ +""" +Step definitions for PAYG shadow-mode end-to-end tests. + +Runs against the saas-profile stack defined in +testing/compose/docker-compose-saas.yml — the backend with STIRLING_FLAVOR=saas +plus a Postgres container holding the stirling_pdf schema. + +The test harness talks to the backend over HTTP and inspects the resulting +rows in `payg_shadow_charge`, `processing_job`, `processing_job_step`, and +`job_artifact_hash` via a direct psycopg connection. Direct DB inspection is +deliberate — we want to verify the *side effects* of the filter, not relay +them through another API layer that itself might be wrong. + +Auth model: the saas profile expects Supabase JWTs. For cucumber we +configure the stack with a test user whose API key is recognised via the +X-API-KEY header, which the PaygChargeInterceptor.resolveUser() path +handles natively. The companion docker-compose-saas.yml seeds the user + +team rows via saas-init.sql so each scenario starts from a known state. +""" + +import os +import time + +import psycopg +import requests +from behave import given, then, when + +BASE_URL = os.environ.get("PAYG_BASE_URL", "http://localhost:8080") +API_KEY = os.environ.get("PAYG_API_KEY", "payg-cucumber-key") + +DB_HOST = os.environ.get("PAYG_DB_HOST", "localhost") +DB_PORT = int(os.environ.get("PAYG_DB_PORT", "5433")) +DB_NAME = os.environ.get("PAYG_DB_NAME", "postgres") +DB_USER = os.environ.get("PAYG_DB_USER", "postgres") +DB_PASSWORD = os.environ.get("PAYG_DB_PASSWORD", "postgres") +DB_SCHEMA = os.environ.get("PAYG_DB_SCHEMA", "stirling_pdf") + +# Fixture paths — small PDFs that ship with the cucumber harness. +FIXTURE_DIR = os.path.join( + os.path.dirname(os.path.dirname(os.path.dirname(__file__))), + "exampleFiles", +) +# Existing cucumber fixtures (verified page counts via pypdf): +# tables.pdf = 1 page → SINGLE_PAGE_PDF +# ghost1.pdf = 3 pages → THREE_PAGE_PDF +# images.pdf = 5 pages (available if a scenario needs more) +# If you add more PAYG scenarios that need a different shape, drop the +# new fixture in exampleFiles/ and reference it here. +SINGLE_PAGE_PDF = os.path.join(FIXTURE_DIR, "tables.pdf") +THREE_PAGE_PDF = os.path.join(FIXTURE_DIR, "ghost1.pdf") + + +# --------------------------------------------------------------------------- +# DB helpers +# --------------------------------------------------------------------------- + + +def _db(): + """Open a fresh connection for each step — keeps things simple.""" + return psycopg.connect( + host=DB_HOST, + port=DB_PORT, + dbname=DB_NAME, + user=DB_USER, + password=DB_PASSWORD, + autocommit=True, + ) + + +def _team_id_for(team_name, conn): + """Look up our test team's id (seeded by saas-init.sql).""" + with conn.cursor() as cur: + cur.execute( + f"SELECT team_id FROM {DB_SCHEMA}.teams WHERE name = %s LIMIT 1", + (team_name,), + ) + row = cur.fetchone() + assert row is not None, f"No team named '{team_name}' in {DB_SCHEMA}.teams" + return row[0] + + +def _shadow_rows_for(team_name): + with _db() as conn: + team_id = _team_id_for(team_name, conn) + with conn.cursor() as cur: + cur.execute( + f"""SELECT shadow_id, payg_units, status, refunded_at, refund_reason, job_id + FROM {DB_SCHEMA}.payg_shadow_charge + WHERE team_id = %s + ORDER BY occurred_at DESC""", + (team_id,), + ) + return cur.fetchall() + + +def _latest_job_for(team_name): + """Return the most recently opened job for the team as a dict.""" + with _db() as conn: + team_id = _team_id_for(team_name, conn) + with conn.cursor() as cur: + cur.execute( + f"""SELECT job_id, status, source, step_count, started_at, closed_at + FROM {DB_SCHEMA}.processing_job + WHERE owner_team_id = %s + ORDER BY started_at DESC + LIMIT 1""", + (team_id,), + ) + row = cur.fetchone() + assert row is not None, f"No processing_job rows for team '{team_name}'" + return { + "job_id": row[0], + "status": row[1], + "source": row[2], + "step_count": row[3], + "started_at": row[4], + "closed_at": row[5], + } + + +def _steps_for_job(job_id): + with _db() as conn: + with conn.cursor() as cur: + cur.execute( + f"""SELECT step_id, tool_id, status, error_code + FROM {DB_SCHEMA}.processing_job_step + WHERE job_id = %s + ORDER BY started_at ASC""", + (job_id,), + ) + return cur.fetchall() + + +def _output_artifact_count(job_id): + with _db() as conn: + with conn.cursor() as cur: + cur.execute( + f"""SELECT COUNT(*) FROM {DB_SCHEMA}.job_artifact_hash + WHERE job_id = %s AND kind = 'OUTPUT'""", + (job_id,), + ) + return cur.fetchone()[0] + + +# --------------------------------------------------------------------------- +# HTTP helpers +# --------------------------------------------------------------------------- + + +def _api_headers(extra=None): + headers = {"X-API-KEY": API_KEY} + if extra: + headers.update(extra) + return headers + + +def _wait_for_health(timeout_seconds=60): + """Block until /api/v1/info/status returns 2xx, or timeout.""" + deadline = time.time() + timeout_seconds + while time.time() < deadline: + try: + r = requests.get(f"{BASE_URL}/api/v1/info/status", timeout=5) + if 200 <= r.status_code < 300: + return + except requests.RequestException: + pass + time.sleep(2) + raise AssertionError( + f"SaaS stack did not become healthy within {timeout_seconds}s at {BASE_URL}" + ) + + +# --------------------------------------------------------------------------- +# GIVEN — environment + test fixtures +# --------------------------------------------------------------------------- + + +@given("the SaaS stack is running with PAYG enabled") +def step_saas_stack_running(context): + _wait_for_health() + + +@given('team "{team_name}" exists with wallet_policy.engine = "{engine}"') +def step_team_exists_with_engine(context, team_name, engine): + """Verify the seed migration created the team + flipped its engine.""" + with _db() as conn: + team_id = _team_id_for(team_name, conn) + with conn.cursor() as cur: + cur.execute( + f"SELECT engine FROM {DB_SCHEMA}.wallet_policy WHERE team_id = %s", + (team_id,), + ) + row = cur.fetchone() + assert row is not None, f"No wallet_policy row for team '{team_name}'" + assert row[0] == engine, ( + f"Team '{team_name}' engine is '{row[0]}', expected '{engine}'. " + "Check saas-init.sql seeded the row correctly." + ) + context.team_name = team_name + + +@given('I am authenticated as a member of team "{team_name}"') +def step_authenticated_as_team_member(context, team_name): + # The seeded API key is bound to a member of this team via saas-init.sql. + context.team_name = team_name + + +@given('there are no existing shadow charges for team "{team_name}"') +def step_clear_shadow_charges(context, team_name): + with _db() as conn: + team_id = _team_id_for(team_name, conn) + with conn.cursor() as cur: + cur.execute( + f"DELETE FROM {DB_SCHEMA}.payg_shadow_charge WHERE team_id = %s", + (team_id,), + ) + cur.execute( + f"""DELETE FROM {DB_SCHEMA}.job_artifact_hash + WHERE job_id IN ( + SELECT job_id FROM {DB_SCHEMA}.processing_job + WHERE owner_team_id = %s + )""", + (team_id,), + ) + cur.execute( + f"""DELETE FROM {DB_SCHEMA}.processing_job_step + WHERE job_id IN ( + SELECT job_id FROM {DB_SCHEMA}.processing_job + WHERE owner_team_id = %s + )""", + (team_id,), + ) + cur.execute( + f"DELETE FROM {DB_SCHEMA}.processing_job WHERE owner_team_id = %s", + (team_id,), + ) + + +# NOTE: The kill-switch scenario (PAYG_FILTER_ENABLED=false) is verified +# manually — see notes/PAYG_DESIGN.md §7.5 "PAYG cucumber: manual-only +# scenarios" for the procedure. No automated step def is provided because +# toggling the env var requires restarting the docker container, and +# orchestrating that mid-suite would couple test setup to compose state +# in ways that have historically been fragile. + + +# --------------------------------------------------------------------------- +# WHEN — invoke tool endpoints +# --------------------------------------------------------------------------- + + +@when('I POST a single-page PDF to "{endpoint}"') +def step_post_single_pdf(context, endpoint): + _post_pdf(context, endpoint, SINGLE_PAGE_PDF) + + +@when('I POST a 3-page PDF to "{endpoint}"') +def step_post_three_page_pdf(context, endpoint): + _post_pdf(context, endpoint, THREE_PAGE_PDF) + + +@when('I POST a single-page PDF to "{endpoint}" with form fields:') +def step_post_with_form_fields(context, endpoint): + """The Gherkin step takes a `table` of key/value pairs that become + multipart form fields. Use this when the default `password=...` field + from _post_pdf() isn't what we want for the scenario.""" + data = {row[0]: row[1] for row in context.table.rows} if context.table else {} + _post_pdf(context, endpoint, SINGLE_PAGE_PDF, form_data=data) + + +@when('I POST a 3-page PDF to "{endpoint}" with form fields:') +def step_post_three_page_with_form_fields(context, endpoint): + data = {row[0]: row[1] for row in context.table.rows} if context.table else {} + _post_pdf(context, endpoint, THREE_PAGE_PDF, form_data=data) + + +@when('I POST a single-page PDF with header "{header_name}: {header_value}" to "{endpoint}"') +def step_post_with_header(context, header_name, header_value, endpoint): + _post_pdf(context, endpoint, SINGLE_PAGE_PDF, extra_headers={header_name: header_value}) + + +@when('I POST two single-page PDFs as a multi-file payload to "{endpoint}"') +def step_post_two_pdfs(context, endpoint): + with open(SINGLE_PAGE_PDF, "rb") as a, open(SINGLE_PAGE_PDF, "rb") as b: + files = [ + ("fileInput", ("a.pdf", a.read(), "application/pdf")), + ("fileInput", ("b.pdf", b.read(), "application/pdf")), + ] + context.response = requests.post( + f"{BASE_URL}{endpoint}", + files=files, + headers=_api_headers(), + timeout=30, + ) + + +@when('I take the response body as "{name}"') +def step_capture_response_body(context, name): + assert context.response is not None, "No response on context" + if not hasattr(context, "captured_bodies"): + context.captured_bodies = {} + context.captured_bodies[name] = context.response.content + + +@when('I POST "{captured_name}" to "{endpoint}"') +def step_post_captured(context, captured_name, endpoint): + body = context.captured_bodies[captured_name] + files = {"fileInput": (f"{captured_name}.pdf", body, "application/pdf")} + # Most tools need at least one extra form field; for the sanitize endpoint + # the defaults are sufficient. + context.response = requests.post( + f"{BASE_URL}{endpoint}", + files=files, + headers=_api_headers(), + timeout=30, + ) + + +def _post_pdf(context, endpoint, fixture_path, extra_headers=None, form_data=None): + """Default form data carries `password=cucumber-test-password` because + /add-password (our most-used scenario endpoint) requires it. Pass + {@code form_data} to override (e.g. to send the wrong password to + /remove-password for the 4xx scenario).""" + with open(fixture_path, "rb") as f: + files = {"fileInput": (os.path.basename(fixture_path), f, "application/pdf")} + data = form_data if form_data is not None else {"password": "cucumber-test-password"} + context.response = requests.post( + f"{BASE_URL}{endpoint}", + files=files, + data=data, + headers=_api_headers(extra_headers), + timeout=60, + ) + + +# --------------------------------------------------------------------------- +# THEN — assertions +# --------------------------------------------------------------------------- + + +@then("the response status is {status:d}") +def step_assert_status(context, status): + assert context.response.status_code == status, _status_mismatch_message( + context.response, status + ) + + +def _status_mismatch_message(response, expected_status): + """Verbose diagnostic for unexpected statuses — dumps URL, headers, body + so we don't have to round-trip via container logs to figure out what came + back. Kept as a helper so future status-range / status-min steps can call + it too.""" + body_preview = response.text[:1000] if response.text else "" + headers_brief = { + k: v + for k, v in response.headers.items() + if k.lower() + in ("content-type", "content-length", "x-content-type-options", "location") + } + return ( + f"Expected {expected_status}, got {response.status_code}\n" + f" URL: {response.request.method} {response.request.url}\n" + f" Headers: {headers_brief}\n" + f" Body: {body_preview}" + ) + + +@then("the response status is >= {minimum:d}") +def step_assert_status_min(context, minimum): + assert context.response.status_code >= minimum, ( + f"Expected >= {minimum}, got {context.response.status_code}" + ) + + +@then("the response status is >= {minimum:d} and < {maximum:d}") +def step_assert_status_range(context, minimum, maximum): + code = context.response.status_code + assert minimum <= code < maximum, f"Expected [{minimum}, {maximum}), got {code}" + + +@then('the response Content-Type is "{expected}"') +def step_assert_content_type(context, expected): + actual = (context.response.headers.get("Content-Type") or "").split(";")[0].strip() + assert actual == expected, f"Expected {expected}, got {actual}" + + +@then('exactly {n:d} shadow charge row exists for team "{team_name}"') +@then('exactly {n:d} shadow charge rows exist for team "{team_name}"') +def step_assert_shadow_count(context, n, team_name): + rows = _shadow_rows_for(team_name) + assert len(rows) == n, f"Expected {n} shadow rows for '{team_name}', found {len(rows)}: {rows}" + + +@then('the latest shadow charge row has status "{status}"') +def step_assert_latest_shadow_status(context, status): + rows = _shadow_rows_for(context.team_name) + assert rows, "No shadow rows for team" + assert rows[0][2] == status, f"Expected status '{status}', got '{rows[0][2]}'" + + +@then("the latest shadow charge row has payg_units >= {minimum:d}") +def step_assert_payg_units_min(context, minimum): + rows = _shadow_rows_for(context.team_name) + assert rows[0][1] >= minimum, f"Expected payg_units >= {minimum}, got {rows[0][1]}" + + +@then("the latest shadow charge row's refunded_at is not null") +def step_assert_refunded_at_set(context): + rows = _shadow_rows_for(context.team_name) + assert rows[0][3] is not None, "refunded_at is null on the latest shadow row" + + +@then('the latest shadow charge row\'s refund_reason starts with "{prefix}"') +def step_assert_refund_reason(context, prefix): + rows = _shadow_rows_for(context.team_name) + reason = rows[0][4] or "" + assert reason.startswith(prefix), f"refund_reason '{reason}' does not start with '{prefix}'" + + +@then("the latest shadow charge row's job is {status}") +def step_assert_latest_jobstatus_via_shadow(context, status): + job = _latest_job_for(context.team_name) + assert job["status"] == status, f"Latest job status is '{job['status']}', expected '{status}'" + + +@then("the latest job is {status}") +def step_assert_latest_job_status(context, status): + job = _latest_job_for(context.team_name) + assert job["status"] == status, f"Latest job status is '{job['status']}', expected '{status}'" + + +@then("the latest job has step_count = {expected:d}") +def step_assert_step_count(context, expected): + job = _latest_job_for(context.team_name) + assert job["step_count"] == expected, ( + f"Latest job step_count is {job['step_count']}, expected {expected}" + ) + + +@then('the latest job\'s source is "{source}"') +def step_assert_job_source(context, source): + job = _latest_job_for(context.team_name) + assert job["source"] == source, f"Latest job source is '{job['source']}', expected '{source}'" + + +@then('the latest job has {n:d} step recorded with status "{status}"') +@then('the latest job has {n:d} steps recorded with status "{status}"') +def step_assert_step_count_with_status(context, n, status): + job = _latest_job_for(context.team_name) + steps = _steps_for_job(job["job_id"]) + matching = [s for s in steps if s[2] == status] + assert len(matching) == n, ( + f"Expected {n} steps with status '{status}', got {len(matching)}: {steps}" + ) + + +@then("the latest step's error_code matches the response status") +def step_assert_step_error_code(context): + job = _latest_job_for(context.team_name) + steps = _steps_for_job(job["job_id"]) + assert steps, "No steps recorded" + latest_step = steps[-1] + expected = str(context.response.status_code) + assert latest_step[3] == expected, ( + f"Latest step error_code is '{latest_step[3]}', expected '{expected}'" + ) + + +@then("the latest job has at least {n:d} OUTPUT artifact hashes recorded") +def step_assert_output_artifact_count(context, n): + job = _latest_job_for(context.team_name) + count = _output_artifact_count(job["job_id"]) + assert count >= n, f"Latest job has {count} OUTPUT artifacts, expected >= {n}" diff --git a/testing/cucumber/requirements.in b/testing/cucumber/requirements.in index 8b1f01d1f..ae968c9e3 100644 --- a/testing/cucumber/requirements.in +++ b/testing/cucumber/requirements.in @@ -5,3 +5,12 @@ pypdf reportlab PyCryptodome qrcode[pil] +# PAYG cucumber scenarios inspect shadow rows directly against the +# saas-profile Postgres container — see testing/cucumber/features/steps/payg_step_definitions.py. +# Loaded by the saas-cucumber CI job only; main behave run doesn't touch it. +psycopg[binary] +# psycopg declares `typing-extensions>=4.6` under a Python-version marker that +# pip-compile leaves unpinned, which trips `--require-hashes`. Listing it here +# forces pip-compile to lock it to a concrete version with a hash. Without this, +# both saas-cucumber and the main docker-compose-tests fail on `pip install`. +typing-extensions diff --git a/testing/cucumber/requirements.txt b/testing/cucumber/requirements.txt index 66d26c2c3..dedcfe43b 100644 --- a/testing/cucumber/requirements.txt +++ b/testing/cucumber/requirements.txt @@ -1,19 +1,19 @@ # -# This file is autogenerated by pip-compile with Python 3.12 +# This file is autogenerated by pip-compile with Python 3.13 # by the following command: # -# pip-compile --generate-hashes --output-file='testing\cucumber\requirements.txt' --strip-extras 'testing\cucumber\requirements.in' +# pip-compile --generate-hashes --output-file=testing/cucumber/requirements.txt --strip-extras testing/cucumber/requirements.in # behave==1.3.3 \ --hash=sha256:2b8f4b64ed2ea756a5a2a73e23defc1c4631e9e724c499e46661778453ebaf51 \ --hash=sha256:89bdb62af8fb9f147ce245736a5de69f025e5edfb66f1fbe16c5007493f842c0 # via - # -r requirements.in + # -r testing/cucumber/requirements.in # behave-html-formatter behave-html-formatter==0.9.10 \ --hash=sha256:c5a9ad3edcac7be5766b14aacce46794885c749c4741fc93f8fc3a4bf2a891aa \ --hash=sha256:fff7ac2118463701423645ad5a12636845bfd6c8a2dd52097b524b4f290aa7c8 - # via -r requirements.in + # via -r testing/cucumber/requirements.in certifi==2026.2.25 \ --hash=sha256:027692e4402ad994f1c42e52a4997a9763c646b73e4096e4d5d6db8af1d6f0fa \ --hash=sha256:e887ab5cee78ea814d3472169153c2d12cd43b14bd03329a39a9c6e2e80bfba7 @@ -154,7 +154,9 @@ charset-normalizer==3.4.7 \ colorama==0.4.6 \ --hash=sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44 \ --hash=sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6 - # via behave + # via + # behave + # qrcode cucumber-expressions==19.0.0 \ --hash=sha256:8eb5ae46dd03dd37fec1163ace1510529501d7d1868ff372c1ab2cd5aa4543a8 \ --hash=sha256:f452e6c73258c1677043ad67ad5f538c87284d6b502004720510fb6b7452d9c5 @@ -272,6 +274,67 @@ pillow==12.2.0 \ # via # qrcode # reportlab +psycopg==3.3.4 \ + --hash=sha256:b6bbc25ccf05c8fad3b061d9db2ef0909a555171b84b07f29458a447253d679a \ + --hash=sha256:e21207764952cff81b6b8bdacad9a3939f2793367fdac2987b3aac36a651b5bc + # via -r testing/cucumber/requirements.in +psycopg-binary==3.3.4 \ + --hash=sha256:018fbed325936da502feb546642c982dcc4b9ffdea32dfef78dbf3b7f7ad4070 \ + --hash=sha256:0579252a1202cd73e4da137a1426e2dae993ae44e757605344282af3a082848c \ + --hash=sha256:136f199a407b5348b9b857c504aff60c77622a28482e7195839ce1b51238c4cc \ + --hash=sha256:13a7f380824c35896dcac7fe0f61440f7ca49d6dc73f3c13a9a4471e6a3b302e \ + --hash=sha256:17a21953a9e5ff3a16dab692625a3676e2f101db5e40072f39dbee2250194d68 \ + --hash=sha256:1dc1f79fd16bb1f3f4421417a514607539f17804d95c7ed617265369d1981cae \ + --hash=sha256:1fbaa292a3c8bb61b45df1ad3da1908ccee7cb889db9425e3557d9e34e2a4829 \ + --hash=sha256:22cdbf5f91ef7bb91fe0c5757e1962d3127a8010256eefd9c61fcaf441802097 \ + --hash=sha256:26df2717e59c0473e4465a97dfb1b7afebaa479277870fd5784d1436470db47c \ + --hash=sha256:276904e3452d6a23d474ef9a21eee19f20eed3d53ddd2576af033827e0ba0992 \ + --hash=sha256:28b7398fdd19db3232c884fb24550bdfe951221f510e195e233299e4c9b78f97 \ + --hash=sha256:2c09aad7051326e7603c14e50636db9c01f78272dc54b3accff03d46370461e6 \ + --hash=sha256:32a6fbf8481e3a370d0d72b860d35948a693cb01281da217f7b2f307636e591a \ + --hash=sha256:41f2ec0fea529832982bcb6c9415de3c86264ebe562b77a467c0fbcd7efbba8d \ + --hash=sha256:46893c26858be12cc49ca4226ed6a60b4bfccadd946b3bebb783a60b38788228 \ + --hash=sha256:47c656a8a7ba6eb0cff1801a4caaa9c8bdc12d03080e273aff1c8ac39971a77e \ + --hash=sha256:494ca54901be8cf9eb7e02c25b731f2317c378efa44f43e8f9bd0e1184ae7be4 \ + --hash=sha256:514404ed543efd620c85602b747df2a23cf1241b4067199e1a66f2d2757aaa41 \ + --hash=sha256:574ea21a9651958f1535c5a1c649c7409e9168bcbffa29a3f2f961f58b322949 \ + --hash=sha256:580ae30a5f95ccd90008ec697d3ed6a4a2047a516407ad904283fa42086936e9 \ + --hash=sha256:5ab28a2a7649df3b72e6b674b4c190e448e8e77cf496a65bd846472048de2089 \ + --hash=sha256:5c4ab71be17bdca30cb34c34c4e1496e2f5d6f20c199c12bad226070b22ef9bf \ + --hash=sha256:612a627d733f695b1de1f9b4bd511c15f999a5d8b915d444bbd7dd71cf3370da \ + --hash=sha256:6402a9d8146cf4b3974ded3fd28a971e83dc6a0333eb7822524a3aa20b546578 \ + --hash=sha256:6b9016b1714da4dd5ecaaa75b82098aa5a0b87854ce9b092e21c27c4ae23e014 \ + --hash=sha256:71e55ccbdfae79a2ed9c6369c3008a3025817ff9d7e27b32a2d84e2a4267e66e \ + --hash=sha256:7465bfe6087d2d5b42d4c53b9b11ca9f218e477317a4a162a10e3c19e984ba8e \ + --hash=sha256:75a9067e236f9b9ae3535b66fe99bddb33d39c0de10112e49b9ab11eee53dc31 \ + --hash=sha256:773d573e11f437ce0bdb95b7c18dc58390494f96d43f8b45b9760436114f7652 \ + --hash=sha256:77df19583501ea288eaf15ac0fe7ad01e6d8091a91d5c41df5c718f307d8e31b \ + --hash=sha256:7f7668f30b9dd5163197e5cbf4e0efd54e00f0a859cc566ce56cfc31f4054839 \ + --hash=sha256:8c0056529e68dbe9184cd4019a1f3d8f3a4ead2f6fc7a5afcf27d3314edd1277 \ + --hash=sha256:94596f9e7633ee3f6440711d43bb70aa31cc0a46a900ab8b4201a366ace5c9e7 \ + --hash=sha256:ab8cca8ef8fb1ccf5b048ae5bd78ba55b9e4b5d472e3ce5ca39ff4d2a9c249e4 \ + --hash=sha256:ad3bc94054876155549fdaedf4a46d1ec69d39a5bcee377148afe498e84c4b8e \ + --hash=sha256:b56b603ebcea8aa10b46228b8410ba7f13e7c2ee54389d4d9be0927fd8ce2a70 \ + --hash=sha256:b6f5a29e9c775b9f12a1a717aa7a2c80f9e1db6f27ba44a5b59c80ac61d2ffcf \ + --hash=sha256:b7bfff1ca23732b488cbca3076fc11bc98d520ee122514fdb17a8e20d3338f5a \ + --hash=sha256:bdef84570ebbce1d42b4e7ea952d21c414c5f118ad02fee00c5625f35e134429 \ + --hash=sha256:c37e024c07308cd06cf3ec51bfd0e7f6157585a4d84d1bce4a7f5f7913719bf8 \ + --hash=sha256:c677c4ad433cb7150c8cd304a0769ae3bcfbe5ea0676eb53faa7b1443b16d0d3 \ + --hash=sha256:cf7f73a4a792bc5db58a4b385d8a1467e8d468f7548702fb0ed1e9b7501b1c13 \ + --hash=sha256:cffc3408d77a27973f33e5d909b624cce683db5fc25964b02fe0aae7886c1007 \ + --hash=sha256:d7b4d40c153fa352ab3cca530f3a0baedf7621b2ebcbd7f084009522c21788fc \ + --hash=sha256:dbfdb9b6cc79f31104a7b162a2b921b765fcc62af6c00540a167a8de47e4ed38 \ + --hash=sha256:df1d567fc430f6df15c9fcf67d87685fc49bdb325adc0db5af1adfb2f44eb5c9 \ + --hash=sha256:e2631da29253a98bd496e6c4813b24e09a4fe3fb2a9e88513305d6f8747cce95 \ + --hash=sha256:e7510c37550f91a187e3660a8cc50d4b760f8c3b8b2f89ebc5698cd2c7f2c85d \ + --hash=sha256:eb05ee1c2b817d27c537333224c9e83c7afb86fe7296ba970990068baf819b16 \ + --hash=sha256:eb4eed2079c01a4850bf467deacfab56d356d4225040170af03dc9958321242d \ + --hash=sha256:ee17a2cf4943cde261adfad1bbc5bf38d6b3776d7afff74c7cabcbeaeb08c260 \ + --hash=sha256:f80e3f2b5331dbbf0901bcb658056c03eeb2c1ef31d774afb0d61598b242e744 \ + --hash=sha256:f9b1c2533af01cd7648378599f82b0b8ae32f293296e6eec5753a625bc97ef28 \ + --hash=sha256:fa1cbc10768a796c96d3243656016bf4e337c81c71097270bb7b0ad6210d9765 \ + --hash=sha256:fbd1d4ed566895ad2d3bf4ddfd8bae90026930ddf29df3b9d91d32c8c47866a7 + # via psycopg pycryptodome==3.23.0 \ --hash=sha256:0011f7f00cdb74879142011f95133274741778abba114ceca229adbf8e62c3e4 \ --hash=sha256:11eeeb6917903876f134b56ba11abe95c0b0fd5e3330def218083c7d98bbcb3c \ @@ -314,29 +377,37 @@ pycryptodome==3.23.0 \ --hash=sha256:dea827b4d55ee390dc89b2afe5927d4308a8b538ae91d9c6f7a5090f397af1aa \ --hash=sha256:e3f2d0aaf8080bda0587d58fc9fe4766e012441e2eed4269a77de6aea981c8be \ --hash=sha256:eb8f24adb74984aa0e5d07a2368ad95276cf38051fe2dc6605cbcf482e04f2a7 - # via -r requirements.in + # via -r testing/cucumber/requirements.in pypdf==6.11.0 \ --hash=sha256:062b51c81b0910e6d2755e99e1c5547a0a23b7d0a32322af66240d8edcfabe87 \ --hash=sha256:769394d5756d5b304c9b6bef88b54b1816b328e7e6fc9254e625529a15ed4ab8 - # via -r requirements.in + # via -r testing/cucumber/requirements.in qrcode==8.2 \ --hash=sha256:16e64e0716c14960108e85d853062c9e8bba5ca8252c0b4d0231b9df4060ff4f \ --hash=sha256:35c3f2a4172b33136ab9f6b3ef1c00260dd2f66f858f24d88418a015f446506c - # via -r requirements.in + # via -r testing/cucumber/requirements.in reportlab==4.5.1 \ --hash=sha256:06fce8cb56c83307cfa4909cdf4e6a2ddbb44e5d6ef4d2edca896d7e9769f091 \ --hash=sha256:9fdf68f4de9171ec66acb4a5feed8f8ca2af43479e707a6fbb0daa75d88e5494 - # via -r requirements.in + # via -r testing/cucumber/requirements.in requests==2.34.2 \ --hash=sha256:2a0d60c172f83ac6ab31e4554906c0f3b3588d37b5cb939b1c061f4907e278e0 \ --hash=sha256:f288924cae4e29463698d6d60bc6a4da69c89185ad1e0bcc4104f584e960b9ed - # via -r requirements.in + # via -r testing/cucumber/requirements.in six==1.17.0 \ --hash=sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274 \ --hash=sha256:ff70335d468e7eb6ec65b95b99d3a2836546063f63acc5171de367e834932a81 # via # behave # parse-type +typing-extensions==4.15.0 \ + --hash=sha256:0cea48d173cc12fa28ecabc3b837ea3cf6f38c6d1136f85cbaaf598984861466 \ + --hash=sha256:f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548 + # via -r testing/cucumber/requirements.in +tzdata==2026.2 \ + --hash=sha256:9173fde7d80d9018e02a662e168e5a2d04f87c41ea174b139fbef642eda62d10 \ + --hash=sha256:bbe9af844f658da81a5f95019480da3a89415801f6cc966806612cc7169bffe7 + # via psycopg urllib3==2.7.0 \ --hash=sha256:231e0ec3b63ceb14667c67be60f2f2c40a518cb38b03af60abc813da26505f4c \ --hash=sha256:9fb4c81ebbb1ce9531cce37674bbc6f1360472bc18ca9a553ede278ef7276897 diff --git a/testing/test-payg.sh b/testing/test-payg.sh new file mode 100644 index 000000000..d3683c448 --- /dev/null +++ b/testing/test-payg.sh @@ -0,0 +1,89 @@ +#!/bin/bash +# +# Cucumber harness for the PAYG shadow-mode scenarios. Brings up the +# saas-profile docker-compose, waits for backend health, pipes the seed SQL +# into the test postgres, then invokes Behave against features/payg/. +# +# Companion to testing/test.sh (which covers the proprietary-flavor stack +# and skips features/payg via behave.ini's exclude_re). Kept as a separate +# entrypoint so the saas variant can be reviewed + iterated on without +# touching the main cucumber harness. + +set -euo pipefail + +PROJECT_ROOT="$(cd "$(dirname "$0")/.." && pwd)" +COMPOSE_FILE="$PROJECT_ROOT/testing/compose/docker-compose-saas.yml" +SEED_FILE="$PROJECT_ROOT/testing/compose/payg/saas-seed.sql" + +cd "$PROJECT_ROOT" + +# Swap behave.ini for the run. The project-default behave.ini excludes +# features/payg so the proprietary CI (which boots without the saas profile) +# can't try to run scenarios that need PAYG tables. behave's exclude_re takes +# priority over a path argument, so even `behave features/payg` would find +# zero features against the default config. The PAYG harness needs a config +# without the payg exclusion — restored on exit. +BEHAVE_INI="$PROJECT_ROOT/testing/cucumber/behave.ini" +BEHAVE_INI_BACKUP="$BEHAVE_INI.payg-backup" + +cleanup() { + # Always dump container logs before tearing the stack down — the workflow + # has a "Dump saas container logs on failure" step but it runs AFTER this + # cleanup, by which point the containers are gone. Cheap diagnostic on + # success, essential on failure. (Earlier ERR-trap conditional didn't + # fire reliably under `set -e` when behave was the script's last command.) + echo "===== STIRLING_BACKEND_LOG_DUMP_START =====" + docker compose -f "$COMPOSE_FILE" logs --tail 500 stirling-pdf-saas || true + echo "===== STIRLING_BACKEND_LOG_DUMP_END =====" + echo "==> Tearing down saas compose stack" + docker compose -f "$COMPOSE_FILE" down -v || true + if [ -f "$BEHAVE_INI_BACKUP" ]; then + mv "$BEHAVE_INI_BACKUP" "$BEHAVE_INI" + fi +} +trap cleanup EXIT + +cp "$BEHAVE_INI" "$BEHAVE_INI_BACKUP" +cat > "$BEHAVE_INI" <<'EOF' +# Temporary behave.ini written by testing/test-payg.sh for the PAYG harness +# run only. Restored on exit. The project default (in git) excludes +# features/payg so the proprietary-flavor CI doesn't try to run them. +[behave] +exclude_re = features/enterprise +EOF + +echo "==> Building + starting saas compose stack" +docker compose -f "$COMPOSE_FILE" up -d --build + +echo "==> Waiting for backend health (max 180s)" +deadline=$(( SECONDS + 180 )) +until curl -fsS http://localhost:8080/api/v1/info/status > /dev/null 2>&1; do + if [ $SECONDS -ge $deadline ]; then + echo "Backend did not become healthy in 180s" + docker compose -f "$COMPOSE_FILE" logs --tail 200 stirling-pdf-saas + exit 1 + fi + sleep 3 +done +echo "Backend healthy." + +echo "==> Seeding test team / user / wallet_policy (PAYG_SHADOW)" +docker compose -f "$COMPOSE_FILE" exec -T postgres-saas \ + psql -U postgres -d postgres < "$SEED_FILE" + +echo "==> Running PAYG cucumber scenarios" +cd "$PROJECT_ROOT/testing/cucumber" +PAYG_BASE_URL="${PAYG_BASE_URL:-http://localhost:8080}" \ +PAYG_API_KEY="${PAYG_API_KEY:-payg-cucumber-key}" \ +PAYG_DB_HOST="${PAYG_DB_HOST:-localhost}" \ +PAYG_DB_PORT="${PAYG_DB_PORT:-5433}" \ +PAYG_DB_USER="${PAYG_DB_USER:-postgres}" \ +PAYG_DB_PASSWORD="${PAYG_DB_PASSWORD:-postgres}" \ +PAYG_DB_NAME="${PAYG_DB_NAME:-postgres}" \ +PAYG_DB_SCHEMA="${PAYG_DB_SCHEMA:-stirling_pdf}" \ +python -m behave features/payg \ + -f behave_html_formatter:HTMLFormatter -o report-payg.html \ + -f pretty \ + --junit --junit-directory junit-payg + +echo "==> PAYG cucumber run complete"