mirror of
https://github.com/arsvendg/Stirling-PDF.git
synced 2026-07-14 10:34:06 +02:00
fix(policies): poll runs to completion with progress, soft-retry when queue is full (#6690)
## What & why
Production reports of policy enforcement "hanging" traced to
large/many-page documents: the watermark step's flatten-to-image
(`convertPDFToImage`) on a 500+ page PDF takes minutes, exceeding both
the client poll cap and the backend per-step timeout. This makes the
slow case graceful instead of looking broken, and makes load-shedding
non-fatal.
### Poll runs to completion (no false "hang")
The client poll loop used a flat ~150s cap that was **shorter than the
backend's 300s per-step timeout**, so it abandoned long-but-healthy runs
mid-flight. The budget is now sized to the backend's real worst case —
`stepCount × per-step timeout + grace`, learned from the first status
report — so the client always polls long enough to surface the run's
**actual** terminal state (success or the backend's real error) rather
than a misleading client-side timeout.
### Per-step progress
The activity feed now shows `Enforcing… · step n/m` (from
`currentStep`/`stepCount`), so a slow step shows movement instead of a
dead spinner.
### Soft-retry on queue rejection
Under load the shared `JobQueue` rejects runs ("queue full"), which
previously surfaced as a hard failure needing a manual Retry. The
backend now tags that rejection with a stable `POLICY_QUEUE_FULL`
errorCode; the client treats it as transient backpressure and
**auto-retries the file in place** with exponential backoff (≈4s→64s, ~2
min), shown as a soft "Busy — retrying…" row, falling back to the manual
Retry only once the retry budget is spent.
## Testing
- **Frontend unit tests** (30 pass across the policies suite), including
a new `usePolicyAutoRun.retry.test.tsx` that drives the real controller
orchestration (poll → `POLICY_QUEUE_FULL` → relabel → backoff → in-place
re-dispatch), plus poll-budget, step-progress, and activity-feed relabel
cases.
- **Backend** `PolicyEngineTest` case asserting a queue-rejected run
carries the `POLICY_QUEUE_FULL` code.
- Typecheck clean on all three flavors (proprietary/saas/core); prettier
+ spotless clean.
- Poll-budget + progress + real-error surfacing were also verified live
end-to-end against a 599-page run (survived past the old cap, showed
step progress, reported the backend's real 300s-timeout failure,
recovered after a simulated network drop).
## Not included (follow-ups)
- The underlying flatten-to-image cost itself (bounded-memory/streaming
flatten, revisiting `convertPDFToImage` default and the 300s timeout) —
the real perf fix, deliberately out of scope here.
This commit is contained in:
+6
-1
@@ -59,6 +59,10 @@ public class PolicyEngine {
|
||||
// files. See ResourceMonitor#shouldQueueJob(int).
|
||||
private static final int RUN_RESOURCE_WEIGHT = 50;
|
||||
|
||||
// errorCode marking a run that was never admitted (job queue full under load). Transient: the
|
||||
// client treats it as "busy" and retries, rather than as a terminal processing failure.
|
||||
private static final String QUEUE_FULL_CODE = "POLICY_QUEUE_FULL";
|
||||
|
||||
private final PolicyExecutor stepExecutor;
|
||||
private final TaskManager taskManager;
|
||||
private final PolicyRunRegistry registry;
|
||||
@@ -239,7 +243,8 @@ public class PolicyEngine {
|
||||
if (!completion.isDone()) {
|
||||
String message = "Policy run could not be queued: " + ex.getMessage();
|
||||
log.error("Policy run {} was not admitted: {}", run.getRunId(), ex.getMessage());
|
||||
run.fail(message);
|
||||
// Transient admission rejection, not a processing failure (see QUEUE_FULL_CODE).
|
||||
run.failWithCode(message, QUEUE_FULL_CODE, null);
|
||||
taskManager.setError(run.getRunId(), message);
|
||||
completion.complete(run);
|
||||
}
|
||||
|
||||
+21
@@ -336,6 +336,27 @@ class PolicyEngineTest {
|
||||
assertEquals(PolicyRunStatus.PENDING, registry.get(handle.runId()).getStatus());
|
||||
}
|
||||
|
||||
@Test
|
||||
void runRejectedWhenQueueFullCarriesTransientErrorCode() {
|
||||
when(resourceMonitor.shouldQueueJob(anyInt())).thenReturn(true);
|
||||
// Admission rejected (queue full): the queued future completes exceptionally.
|
||||
CompletableFuture<Object> rejected = new CompletableFuture<>();
|
||||
rejected.completeExceptionally(
|
||||
new RuntimeException("Job queue full, please try again later"));
|
||||
doReturn(rejected).when(jobQueue).queueJob(anyString(), anyInt(), any(), anyLong());
|
||||
|
||||
PolicyRunHandle handle =
|
||||
engine.submit(
|
||||
definition(new PipelineStep(ROTATE, Map.of())),
|
||||
PolicyInputs.of(List.of(pdf("input", "input.pdf"))),
|
||||
PolicyProgressListener.NOOP);
|
||||
|
||||
PolicyRun run = registry.get(handle.runId());
|
||||
assertEquals(PolicyRunStatus.FAILED, run.getStatus());
|
||||
// Tagged transient so the client backs off and retries instead of hard-failing.
|
||||
assertEquals("POLICY_QUEUE_FULL", run.getErrorCode());
|
||||
}
|
||||
|
||||
@Test
|
||||
void resumeIsNotYetImplemented() {
|
||||
assertThrows(UnsupportedOperationException.class, () -> engine.resume("any", List.of()));
|
||||
|
||||
Reference in New Issue
Block a user