Support multiple pipeline watch directories and configurable pipeline base path (#5545)

### Motivation
- Allow operators to configure a pipeline base directory and multiple
watched folders so the pipeline can monitor several directories and
subdirectories concurrently.
- Ensure scanning traverses subdirectories while skipping internal
processing folders (e.g. `processing`) and preserve existing behavior
for finished/output paths.
- Expose the new options in the server `settings.yml.template` and the
admin UI so paths can be edited from the web console.

### Description
- Added new `pipelineDir` and `watchedFoldersDirs` fields to
`ApplicationProperties.CustomPaths.Pipeline` and kept backward
compatibility with `watchedFoldersDir`
(app/common/src/main/java/stirling/software/common/model/ApplicationProperties.java).
- Resolved pipeline base and multiple watched folder paths in
`RuntimePathConfig` and exposed `getPipelineWatchedFoldersPaths()`
(app/common/src/main/java/stirling/software/common/configuration/RuntimePathConfig.java).
- Updated `FileMonitor` to accept and register multiple root paths
instead of a single root
(app/common/src/main/java/stirling/software/common/util/FileMonitor.java).
- Updated `PipelineDirectoryProcessor` to iterate all configured watched
roots and to walk subdirectories while ignoring `processing` dirs
(app/core/src/main/java/stirling/software/SPDF/controller/api/pipeline/PipelineDirectoryProcessor.java).
- Exposed the new settings in `settings.yml.template` and the admin UI,
including a multi-line `Textarea` to edit `watchedFoldersDirs`
(app/core/src/main/resources/settings.yml.template,
frontend/src/proprietary/components/shared/config/configSections/AdminGeneralSection.tsx).
- Adjusted unit test setup to account for list-based watched folders
(app/common/src/test/java/stirling/software/common/util/FileMonitorTest.java).

### Testing
- Ran formatting and build checks with `./gradlew spotlessApply` and
`./gradlew build` using Java 21 via
`JAVA_HOME=/root/.local/share/mise/installs/java/21.0.2
PATH=/root/.local/share/mise/installs/java/21.0.2/bin:$PATH ./gradlew
...`, but both runs failed due to Gradle plugin resolution being blocked
in this environment (plugin portal/network 403), so full
compilation/formatting could not complete.
- Confirmed the code compiles locally was not possible here; unit test
`FileMonitorTest` was updated to use the new API but was not executed
due to the blocked build.
- Changes were committed (`Support multiple pipeline watch directories`)
and the repository diff contains the listed file modifications.

------
[Codex
Task](https://chatgpt.com/codex/tasks/task_b_69741ecd17c883288d8085a63ccd66f4)
This commit is contained in:
Anthony Stirling
2026-01-31 20:59:25 +00:00
committed by GitHub
parent 2ae413c5ea
commit 4f404a1ccf
9 changed files with 452 additions and 35 deletions
@@ -186,6 +186,13 @@ public class AdminSettingsController {
+ HtmlUtils.htmlEscape(key)));
}
// Validate pipeline path settings
String validationError = validatePipelinePathSetting(key, value);
if (validationError != null) {
return ResponseEntity.badRequest()
.body(Map.of("error", HtmlUtils.htmlEscape(validationError)));
}
log.info("Admin updating setting: {} = {}", key, value);
GeneralUtils.saveKeyToSettings(key, value);
@@ -642,6 +649,54 @@ public class AdminSettingsController {
return true;
}
private String validatePipelinePathSetting(String key, Object value) {
// Validate pipeline path settings
if (key.startsWith("system.customPaths.pipeline.watchedFoldersDirs")
&& value instanceof java.util.List) {
@SuppressWarnings("unchecked")
java.util.List<String> paths = (java.util.List<String>) value;
// Check for empty or all-blank paths
if (paths.isEmpty()) {
return null; // Empty is OK, will use default
}
// Validate each path
java.util.Set<String> normalizedPaths = new java.util.HashSet<>();
for (String path : paths) {
if (path != null && !path.trim().isEmpty()) {
try {
java.nio.file.Path normalized =
java.nio.file.Paths.get(path.trim()).toAbsolutePath().normalize();
String normalizedStr = normalized.toString();
// Check for duplicates
if (normalizedPaths.contains(normalizedStr)) {
return "Duplicate path detected: " + path;
}
normalizedPaths.add(normalizedStr);
} catch (java.nio.file.InvalidPathException e) {
return "Invalid path: " + path + " - " + e.getMessage();
}
}
}
// Check for overlapping paths
java.util.List<String> pathList = new java.util.ArrayList<>(normalizedPaths);
for (int i = 0; i < pathList.size(); i++) {
java.nio.file.Path path1 = java.nio.file.Paths.get(pathList.get(i));
for (int j = i + 1; j < pathList.size(); j++) {
java.nio.file.Path path2 = java.nio.file.Paths.get(pathList.get(j));
if (path1.startsWith(path2) || path2.startsWith(path1)) {
return "Overlapping paths detected: " + path1 + " and " + path2;
}
}
}
}
return null; // Valid
}
private Object getSettingByKey(String key) {
if (key == null || key.trim().isEmpty()) {
return null;