fix(policies): poll runs to completion with progress, soft-retry when queue is full (#6690)

## What & why

Production reports of policy enforcement "hanging" traced to
large/many-page documents: the watermark step's flatten-to-image
(`convertPDFToImage`) on a 500+ page PDF takes minutes, exceeding both
the client poll cap and the backend per-step timeout. This makes the
slow case graceful instead of looking broken, and makes load-shedding
non-fatal.

### Poll runs to completion (no false "hang")
The client poll loop used a flat ~150s cap that was **shorter than the
backend's 300s per-step timeout**, so it abandoned long-but-healthy runs
mid-flight. The budget is now sized to the backend's real worst case —
`stepCount × per-step timeout + grace`, learned from the first status
report — so the client always polls long enough to surface the run's
**actual** terminal state (success or the backend's real error) rather
than a misleading client-side timeout.

### Per-step progress
The activity feed now shows `Enforcing… · step n/m` (from
`currentStep`/`stepCount`), so a slow step shows movement instead of a
dead spinner.

### Soft-retry on queue rejection
Under load the shared `JobQueue` rejects runs ("queue full"), which
previously surfaced as a hard failure needing a manual Retry. The
backend now tags that rejection with a stable `POLICY_QUEUE_FULL`
errorCode; the client treats it as transient backpressure and
**auto-retries the file in place** with exponential backoff (≈4s→64s, ~2
min), shown as a soft "Busy — retrying…" row, falling back to the manual
Retry only once the retry budget is spent.

## Testing
- **Frontend unit tests** (30 pass across the policies suite), including
a new `usePolicyAutoRun.retry.test.tsx` that drives the real controller
orchestration (poll → `POLICY_QUEUE_FULL` → relabel → backoff → in-place
re-dispatch), plus poll-budget, step-progress, and activity-feed relabel
cases.
- **Backend** `PolicyEngineTest` case asserting a queue-rejected run
carries the `POLICY_QUEUE_FULL` code.
- Typecheck clean on all three flavors (proprietary/saas/core); prettier
+ spotless clean.
- Poll-budget + progress + real-error surfacing were also verified live
end-to-end against a 599-page run (survived past the old cap, showed
step progress, reported the backend's real 300s-timeout failure,
recovered after a simulated network drop).

## Not included (follow-ups)
- The underlying flatten-to-image cost itself (bounded-memory/streaming
flatten, revisiting `convertPDFToImage` default and the 300s timeout) —
the real perf fix, deliberately out of scope here.
This commit is contained in:
Reece Browne
2026-06-16 17:32:12 +00:00
committed by GitHub
parent f33f4f8f75
commit f127d4f575
9 changed files with 516 additions and 16 deletions
@@ -59,6 +59,10 @@ public class PolicyEngine {
// files. See ResourceMonitor#shouldQueueJob(int).
private static final int RUN_RESOURCE_WEIGHT = 50;
// errorCode marking a run that was never admitted (job queue full under load). Transient: the
// client treats it as "busy" and retries, rather than as a terminal processing failure.
private static final String QUEUE_FULL_CODE = "POLICY_QUEUE_FULL";
private final PolicyExecutor stepExecutor;
private final TaskManager taskManager;
private final PolicyRunRegistry registry;
@@ -239,7 +243,8 @@ public class PolicyEngine {
if (!completion.isDone()) {
String message = "Policy run could not be queued: " + ex.getMessage();
log.error("Policy run {} was not admitted: {}", run.getRunId(), ex.getMessage());
run.fail(message);
// Transient admission rejection, not a processing failure (see QUEUE_FULL_CODE).
run.failWithCode(message, QUEUE_FULL_CODE, null);
taskManager.setError(run.getRunId(), message);
completion.complete(run);
}
@@ -336,6 +336,27 @@ class PolicyEngineTest {
assertEquals(PolicyRunStatus.PENDING, registry.get(handle.runId()).getStatus());
}
@Test
void runRejectedWhenQueueFullCarriesTransientErrorCode() {
when(resourceMonitor.shouldQueueJob(anyInt())).thenReturn(true);
// Admission rejected (queue full): the queued future completes exceptionally.
CompletableFuture<Object> rejected = new CompletableFuture<>();
rejected.completeExceptionally(
new RuntimeException("Job queue full, please try again later"));
doReturn(rejected).when(jobQueue).queueJob(anyString(), anyInt(), any(), anyLong());
PolicyRunHandle handle =
engine.submit(
definition(new PipelineStep(ROTATE, Map.of())),
PolicyInputs.of(List.of(pdf("input", "input.pdf"))),
PolicyProgressListener.NOOP);
PolicyRun run = registry.get(handle.runId());
assertEquals(PolicyRunStatus.FAILED, run.getStatus());
// Tagged transient so the client backs off and retries instead of hard-failing.
assertEquals("POLICY_QUEUE_FULL", run.getErrorCode());
}
@Test
void resumeIsNotYetImplemented() {
assertThrows(UnsupportedOperationException.class, () -> engine.resume("any", List.of()));