mirror of
https://github.com/arsvendg/Stirling-PDF.git
synced 2026-07-14 18:44:05 +02:00
ConnorYohandGitHub
ff96a80947
## What this PR is End-to-end cucumber coverage for the PAYG shadow charging engine (the filter + interceptor stack from #6519), wired into CI via a new `docker-compose-tests-saas.yml` workflow that runs only on PAYG-touching PRs. Stacked on #6519. ## Automated scenarios (run by `docker-compose-tests-saas.yml`) See [`testing/cucumber/features/payg/shadow_charges.feature`](../tree/payg-s3-cucumber/testing/cucumber/features/payg/shadow_charges.feature): | Scenario | Validates | |---|---| | First tool call writes a CHARGED row | Filter + interceptor fire end-to-end | | Lineage join — second call on output | `JobService.joinOrOpen` matching; no new shadow row | | 4xx leaves the row CHARGED | "Customer paid for the attempt" semantics | | ZIP-returning tool records per-PDF OUTPUT | `PaygOutputExtractor` unpacks + records signatures | | Multi-file input writes a single shadow row | Multi-input group sizing | | `X-Stirling-Automation` sets PIPELINE source | Header → `JobSource` detection | All 6 run locally via `./testing/test-payg.sh` and will run on CI for any PR that touches `app/saas/**`, the PAYG cucumber features, the saas compose stack, or the workflow itself. ## Manual-only scenarios — documented in design doc, not in this suite Two parts of the shadow engine are deliberately not automated; the engine paths are unit-tested in `PaygChargeInterceptorTest.afterCompletion_5xx_opened_*`, and the manual procedures (which require a temporary throw endpoint or a container restart with a flag flipped) live in [`notes/PAYG_DESIGN.md` §7.5.2 "PAYG cucumber: manual-only scenarios"](../tree/payg-s3-cucumber/notes/PAYG_DESIGN.md). - **5xx first-step failure → REFUNDED + CLOSED.** No reliably-5xx-ing endpoint exists; manual procedure adds a throw endpoint, runs, asserts, removes. - **Kill-switch (`PAYG_FILTER_ENABLED=false`).** Needs a container restart mid-suite; manual procedure tears down, flips env, brings up, asserts zero shadow rows. If either gets a hot-reload path (test-only throw endpoint shipped behind a profile gate, or admin endpoint for the kill switch), automate it in a follow-up and drop the manual procedure. ## CI workflow `.github/workflows/docker-compose-tests-saas.yml` (new) — self-contained, not wired into `build.yml`'s `files-changed` matrix so the saas-cucumber job fails and succeeds independently. Triggers only on PAYG-relevant paths. No JaCoCo coverage in v1 (saas compose doesn't have the coverage override; can add later). ## Test infrastructure (recap) - **`testing/compose/docker-compose-saas.yml`** — Stirling-PDF backend with `STIRLING_FLAVOR=saas` + Postgres holding the `stirling_pdf` schema. Supabase JWT auto-config disabled; API-key auth via `SECURITY_CUSTOMGLOBALAPIKEY` is the live path the cucumber tests exercise. - **`testing/compose/payg/saas-init.sql`** + **`saas-seed.sql`** — schema bootstrap + idempotent seed (team / user / wallet_policy). - **`testing/cucumber/features/payg/shadow_charges.feature`** — the 6 scenarios above. - **`testing/cucumber/features/steps/payg_step_definitions.py`** — step defs using `requests` (HTTP) + `psycopg` (direct DB inspection). Direct DB reads are deliberate — we want to see the filter's side effects, not relay them through another API layer. - **`testing/test-payg.sh`** — companion runner to `testing/test.sh`. Brings up the saas compose, waits for health, seeds, runs behave, tears down. - **`behave.ini`** excludes `features/payg` from the default behave run (the saas-cucumber CI job invokes it explicitly). ## Why a separate harness from `testing/test.sh` The existing `test.sh` covers the proprietary-flavour stack (no PAYG tables, no saas profile). Coupling two CI matrices that fail and succeed independently into one script is asking for trouble. Keep the saas-cucumber job focused on its own concerns; once the harness is mature, the wider team can decide whether to merge them. ## Tracked in `notes/PAYG_DESIGN.md` §7.5 (PR-S3) + §7.5.2 (manual scenarios).
90 lines
3.6 KiB
Bash
90 lines
3.6 KiB
Bash
#!/bin/bash
|
|
#
|
|
# Cucumber harness for the PAYG shadow-mode scenarios. Brings up the
|
|
# saas-profile docker-compose, waits for backend health, pipes the seed SQL
|
|
# into the test postgres, then invokes Behave against features/payg/.
|
|
#
|
|
# Companion to testing/test.sh (which covers the proprietary-flavor stack
|
|
# and skips features/payg via behave.ini's exclude_re). Kept as a separate
|
|
# entrypoint so the saas variant can be reviewed + iterated on without
|
|
# touching the main cucumber harness.
|
|
|
|
set -euo pipefail
|
|
|
|
PROJECT_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
|
COMPOSE_FILE="$PROJECT_ROOT/testing/compose/docker-compose-saas.yml"
|
|
SEED_FILE="$PROJECT_ROOT/testing/compose/payg/saas-seed.sql"
|
|
|
|
cd "$PROJECT_ROOT"
|
|
|
|
# Swap behave.ini for the run. The project-default behave.ini excludes
|
|
# features/payg so the proprietary CI (which boots without the saas profile)
|
|
# can't try to run scenarios that need PAYG tables. behave's exclude_re takes
|
|
# priority over a path argument, so even `behave features/payg` would find
|
|
# zero features against the default config. The PAYG harness needs a config
|
|
# without the payg exclusion — restored on exit.
|
|
BEHAVE_INI="$PROJECT_ROOT/testing/cucumber/behave.ini"
|
|
BEHAVE_INI_BACKUP="$BEHAVE_INI.payg-backup"
|
|
|
|
cleanup() {
|
|
# Always dump container logs before tearing the stack down — the workflow
|
|
# has a "Dump saas container logs on failure" step but it runs AFTER this
|
|
# cleanup, by which point the containers are gone. Cheap diagnostic on
|
|
# success, essential on failure. (Earlier ERR-trap conditional didn't
|
|
# fire reliably under `set -e` when behave was the script's last command.)
|
|
echo "===== STIRLING_BACKEND_LOG_DUMP_START ====="
|
|
docker compose -f "$COMPOSE_FILE" logs --tail 500 stirling-pdf-saas || true
|
|
echo "===== STIRLING_BACKEND_LOG_DUMP_END ====="
|
|
echo "==> Tearing down saas compose stack"
|
|
docker compose -f "$COMPOSE_FILE" down -v || true
|
|
if [ -f "$BEHAVE_INI_BACKUP" ]; then
|
|
mv "$BEHAVE_INI_BACKUP" "$BEHAVE_INI"
|
|
fi
|
|
}
|
|
trap cleanup EXIT
|
|
|
|
cp "$BEHAVE_INI" "$BEHAVE_INI_BACKUP"
|
|
cat > "$BEHAVE_INI" <<'EOF'
|
|
# Temporary behave.ini written by testing/test-payg.sh for the PAYG harness
|
|
# run only. Restored on exit. The project default (in git) excludes
|
|
# features/payg so the proprietary-flavor CI doesn't try to run them.
|
|
[behave]
|
|
exclude_re = features/enterprise
|
|
EOF
|
|
|
|
echo "==> Building + starting saas compose stack"
|
|
docker compose -f "$COMPOSE_FILE" up -d --build
|
|
|
|
echo "==> Waiting for backend health (max 180s)"
|
|
deadline=$(( SECONDS + 180 ))
|
|
until curl -fsS http://localhost:8080/api/v1/info/status > /dev/null 2>&1; do
|
|
if [ $SECONDS -ge $deadline ]; then
|
|
echo "Backend did not become healthy in 180s"
|
|
docker compose -f "$COMPOSE_FILE" logs --tail 200 stirling-pdf-saas
|
|
exit 1
|
|
fi
|
|
sleep 3
|
|
done
|
|
echo "Backend healthy."
|
|
|
|
echo "==> Seeding test team / user / wallet_policy (PAYG_SHADOW)"
|
|
docker compose -f "$COMPOSE_FILE" exec -T postgres-saas \
|
|
psql -U postgres -d postgres < "$SEED_FILE"
|
|
|
|
echo "==> Running PAYG cucumber scenarios"
|
|
cd "$PROJECT_ROOT/testing/cucumber"
|
|
PAYG_BASE_URL="${PAYG_BASE_URL:-http://localhost:8080}" \
|
|
PAYG_API_KEY="${PAYG_API_KEY:-payg-cucumber-key}" \
|
|
PAYG_DB_HOST="${PAYG_DB_HOST:-localhost}" \
|
|
PAYG_DB_PORT="${PAYG_DB_PORT:-5433}" \
|
|
PAYG_DB_USER="${PAYG_DB_USER:-postgres}" \
|
|
PAYG_DB_PASSWORD="${PAYG_DB_PASSWORD:-postgres}" \
|
|
PAYG_DB_NAME="${PAYG_DB_NAME:-postgres}" \
|
|
PAYG_DB_SCHEMA="${PAYG_DB_SCHEMA:-stirling_pdf}" \
|
|
python -m behave features/payg \
|
|
-f behave_html_formatter:HTMLFormatter -o report-payg.html \
|
|
-f pretty \
|
|
--junit --junit-directory junit-payg
|
|
|
|
echo "==> PAYG cucumber run complete"
|