PAYG B-3 / S-3: cucumber suite for shadow-mode flows + CI workflow (#6522)

## What this PR is End-to-end cucumber coverage for the PAYG shadow charging engine (the filter + interceptor stack from #6519), wired into CI via a new `docker-compose-tests-saas.yml` workflow that runs only on PAYG-touching PRs. Stacked on #6519. ## Automated scenarios (run by `docker-compose-tests-saas.yml`) See [`testing/cucumber/features/payg/shadow_charges.feature`](../tree/payg-s3-cucumber/testing/cucumber/features/payg/shadow_charges.feature): | Scenario | Validates | |---|---| | First tool call writes a CHARGED row | Filter + interceptor fire end-to-end | | Lineage join — second call on output | `JobService.joinOrOpen` matching; no new shadow row | | 4xx leaves the row CHARGED | "Customer paid for the attempt" semantics | | ZIP-returning tool records per-PDF OUTPUT | `PaygOutputExtractor` unpacks + records signatures | | Multi-file input writes a single shadow row | Multi-input group sizing | | `X-Stirling-Automation` sets PIPELINE source | Header → `JobSource` detection | All 6 run locally via `./testing/test-payg.sh` and will run on CI for any PR that touches `app/saas/**`, the PAYG cucumber features, the saas compose stack, or the workflow itself. ## Manual-only scenarios — documented in design doc, not in this suite Two parts of the shadow engine are deliberately not automated; the engine paths are unit-tested in `PaygChargeInterceptorTest.afterCompletion_5xx_opened_*`, and the manual procedures (which require a temporary throw endpoint or a container restart with a flag flipped) live in [`notes/PAYG_DESIGN.md` §7.5.2 "PAYG cucumber: manual-only scenarios"](../tree/payg-s3-cucumber/notes/PAYG_DESIGN.md). - **5xx first-step failure → REFUNDED + CLOSED.** No reliably-5xx-ing endpoint exists; manual procedure adds a throw endpoint, runs, asserts, removes. - **Kill-switch (`PAYG_FILTER_ENABLED=false`).** Needs a container restart mid-suite; manual procedure tears down, flips env, brings up, asserts zero shadow rows. If either gets a hot-reload path (test-only throw endpoint shipped behind a profile gate, or admin endpoint for the kill switch), automate it in a follow-up and drop the manual procedure. ## CI workflow `.github/workflows/docker-compose-tests-saas.yml` (new) — self-contained, not wired into `build.yml`'s `files-changed` matrix so the saas-cucumber job fails and succeeds independently. Triggers only on PAYG-relevant paths. No JaCoCo coverage in v1 (saas compose doesn't have the coverage override; can add later). ## Test infrastructure (recap) - **`testing/compose/docker-compose-saas.yml`** — Stirling-PDF backend with `STIRLING_FLAVOR=saas` + Postgres holding the `stirling_pdf` schema. Supabase JWT auto-config disabled; API-key auth via `SECURITY_CUSTOMGLOBALAPIKEY` is the live path the cucumber tests exercise. - **`testing/compose/payg/saas-init.sql`** + **`saas-seed.sql`** — schema bootstrap + idempotent seed (team / user / wallet_policy). - **`testing/cucumber/features/payg/shadow_charges.feature`** — the 6 scenarios above. - **`testing/cucumber/features/steps/payg_step_definitions.py`** — step defs using `requests` (HTTP) + `psycopg` (direct DB inspection). Direct DB reads are deliberate — we want to see the filter's side effects, not relay them through another API layer. - **`testing/test-payg.sh`** — companion runner to `testing/test.sh`. Brings up the saas compose, waits for health, seeds, runs behave, tears down. - **`behave.ini`** excludes `features/payg` from the default behave run (the saas-cucumber CI job invokes it explicitly). ## Why a separate harness from `testing/test.sh` The existing `test.sh` covers the proprietary-flavour stack (no PAYG tables, no saas profile). Coupling two CI matrices that fail and succeed independently into one script is asking for trouble. Keep the saas-cucumber job focused on its own concerns; once the harness is mature, the wider team can decide whether to merge them. ## Tracked in `notes/PAYG_DESIGN.md` §7.5 (PR-S3) + §7.5.2 (manual scenarios).
2026-07-14 10:34:06 +02:00 · 2026-06-09 14:47:40 +00:00
parent 347ae9ebbf
commit ff96a80947
14 changed files with 1376 additions and 18 deletions
@@ -0,0 +1,136 @@
+name: Docker Compose Cucumber tests (saas / PAYG)
+
+# Self-contained CI job for the PAYG shadow-mode cucumber scenarios.
+# Triggers only on PAYG-relevant paths so we don't add CI minutes to every PR
+# that doesn't touch the saas flavour.
+#
+# Companion to `docker-compose-tests.yml` (which runs against the
+# proprietary-flavour stack and skips features/payg via behave.ini's
+# exclude_re). Kept as a separate workflow so the saas matrix can fail and
+# succeed independently without touching the main cucumber harness.
+
+on:
+  pull_request:
+    paths:
+      - "app/saas/**"
+      - "testing/cucumber/features/payg/**"
+      - "testing/cucumber/features/steps/payg_step_definitions.py"
+      - "testing/cucumber/requirements.txt"
+      - "testing/compose/docker-compose-saas.yml"
+      - "testing/compose/payg/**"
+      - "testing/test-payg.sh"
+      - ".github/workflows/docker-compose-tests-saas.yml"
+  push:
+    branches: [main]
+    paths:
+      - "app/saas/**"
+      - "testing/cucumber/features/payg/**"
+      - "testing/cucumber/features/steps/payg_step_definitions.py"
+      - "testing/cucumber/requirements.txt"
+      - "testing/compose/docker-compose-saas.yml"
+      - "testing/compose/payg/**"
+      - "testing/test-payg.sh"
+      - ".github/workflows/docker-compose-tests-saas.yml"
+
+permissions:
+  contents: read
+
+jobs:
+  pick:
+    uses: ./.github/workflows/_runner-pick.yml
+
+  docker-compose-tests-saas:
+    needs: pick
+    runs-on: ${{ needs.pick.outputs.is_fork == 'true' && 'ubuntu-latest' || 'depot-ubuntu-24.04-4' }}
+    permissions:
+      actions: write
+      contents: read
+      checks: write
+    env:
+      DEPOT_TOKEN: ${{ secrets.DEPOT_TOKEN }}
+
+    steps:
+      - name: Harden Runner
+        uses: step-security/harden-runner@ab7a9404c0f3da075243ca237b5fac12c98deaa5 # v2.19.3
+        with:
+          egress-policy: audit
+
+      - name: Checkout Repository
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+
+      - name: Set up JDK 25
+        uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654 # v5.2.0
+        with:
+          java-version: "25"
+          distribution: "temurin"
+
+      - name: Cache Gradle dependency artifacts
+        uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
+        with:
+          path: |
+            ~/.gradle/wrapper
+            ~/.gradle/caches/modules-2/files-2.1
+            ~/.gradle/caches/modules-2/metadata-2.*
+          key: gradle-deps-saas-${{ runner.os }}-jdk-25-${{ hashFiles('**/gradle/wrapper/gradle-wrapper.properties', '**/*.gradle', '**/*.gradle.kts', 'settings.gradle', 'settings.gradle.kts', 'gradle/libs.versions.toml') }}
+
+      - name: Setup Gradle
+        uses: gradle/actions/setup-gradle@50e97c2cd7a37755bbfafc9c5b7cafaece252f6e # v6.1.0
+        with:
+          gradle-version: 9.3.1
+          cache-disabled: true
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
+
+      - name: Expose GitHub runtime for Buildx cache
+        uses: crazy-max/ghaction-github-runtime@04d248b84655b509d8c44dc1d6f990c879747487 # v4.0.0
+
+      # No "Install Docker Compose" step: Ubuntu runners ship with `docker compose`
+      # v2 (built into the Docker CLI). test-payg.sh uses the v2 form throughout
+      # (`docker compose …`, no hyphen), so the legacy v1 `docker-compose` binary
+      # isn't needed. Avoids a `curl | sudo install` without checksum verification
+      # (Aikido flagged this when copy-pasted from docker-compose-tests.yml).
+
+      - name: Set up Python
+        uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
+        with:
+          python-version: "3.12"
+          cache: "pip"
+          cache-dependency-path: ./testing/cucumber/requirements.txt
+
+      - name: Pip requirements
+        run: |
+          pip install --require-hashes --only-binary=:all: -r ./testing/cucumber/requirements.txt
+
+      - name: Run PAYG Cucumber Tests
+        env:
+          MAVEN_USER: ${{ secrets.MAVEN_USER }}
+          MAVEN_PASSWORD: ${{ secrets.MAVEN_PASSWORD }}
+          MAVEN_PUBLIC_URL: ${{ secrets.MAVEN_PUBLIC_URL }}
+        run: |
+          chmod +x ./testing/test-payg.sh
+          ./testing/test-payg.sh
+
+      - name: Dump saas container logs on failure
+        if: failure()
+        run: |
+          docker compose -f testing/compose/docker-compose-saas.yml logs --tail 500 stirling-pdf-saas || true
+          docker compose -f testing/compose/docker-compose-saas.yml logs --tail 200 postgres-saas || true
+
+      - name: Upload PAYG Cucumber Report
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: payg-cucumber-report
+          path: testing/cucumber/report-payg.html
+          retention-days: 7
+          if-no-files-found: warn
+
+      - name: PAYG Cucumber Test Report
+        if: always()
+        uses: dorny/test-reporter@a43b3a5f7366b97d083190328d2c652e1a8b6aa2 # v3.0.0
+        with:
+          name: PAYG Cucumber Tests
+          path: testing/cucumber/junit-payg/*.xml
+          reporter: java-junit
+          fail-on-error: false