Add new triggers, sources and sinks to policies (#6543)

# Description of Changes
Add new triggers:
- Schedule (fires every X amount of time)
- Folder watch (fires whenever the OS tells us a folder has a new file
in it; on Mac this is technically a 2s schedule but that's just how Java
implements it)

Add new sources:
- Folder (reads from this directory)

Add new sinks:
- Inline (stores in FileStorage)
- Folder (stores in specified directory)

Still want to do S3 buckets and web hooks and stuff, but they can come
in a future PR. I'm hoping this should make it sufficient to be able to
integrate with processing folders frontend etc. I've also changed it so
that policies can have multiple sources and triggers at once, which
seems like it might be useful.
This commit is contained in:
James Brunton
2026-06-10 10:55:29 +00:00
committed by GitHub
parent e2536daeb8
commit 3675db5907
36 changed files with 2556 additions and 114 deletions
@@ -80,6 +80,7 @@ public class ApplicationProperties {
private Mcp mcp = new Mcp();
private InternalApi internalApi = new InternalApi();
private Cluster cluster = new Cluster();
private Policies policies = new Policies();
@Bean
public PropertySource<?> dynamicYamlPropertySource(ConfigurableEnvironment environment)
@@ -203,6 +204,45 @@ public class ApplicationProperties {
}
}
@Data
public static class Policies {
/**
* Absolute directories that policy folder input sources and output sinks may read from or
* write to. Empty (the default) disables folder access entirely, so a policy can never be
* pointed at an arbitrary server path. Stirling's own config directory is always
* off-limits, and folder access is always disabled in SaaS mode regardless of this list.
*/
private List<String> allowedFolderRoots = new java.util.ArrayList<>();
/** How often (seconds) the schedule trigger checks for policies whose schedule is due. */
private long scheduleSweepSeconds = 60;
/**
* How often (seconds) the folder-watch trigger reconciles its watch registrations and
* re-runs every folder-watch policy as a safety net for filesystem events that were missed
* (NFS, bind mounts, inotify-queue overflow).
*/
private long watchReconcileSeconds = 300;
/**
* How long (milliseconds) the folder-watch trigger keeps draining filesystem events after
* the first, so a burst from a single file copy coalesces into one run instead of many.
*/
private long watchQuietPeriodMs = 500;
/**
* SSE emitter timeout (milliseconds) for streamed runs; generous for long multi-step runs.
*/
private long streamTimeoutMs = 1800000;
/**
* How long (minutes) a finished run's in-memory state is retained before eviction,
* mirroring the job-result expiry so rich run state does not outlive the process. Active
* and paused runs are kept regardless of age.
*/
private int runExpiryMinutes = 30;
}
@Data
public static class PdfEditor {
private Cache cache = new Cache();