Support multiple pipeline watch directories and configurable pipeline base path (#5545)

### Motivation
- Allow operators to configure a pipeline base directory and multiple
watched folders so the pipeline can monitor several directories and
subdirectories concurrently.
- Ensure scanning traverses subdirectories while skipping internal
processing folders (e.g. `processing`) and preserve existing behavior
for finished/output paths.
- Expose the new options in the server `settings.yml.template` and the
admin UI so paths can be edited from the web console.

### Description
- Added new `pipelineDir` and `watchedFoldersDirs` fields to
`ApplicationProperties.CustomPaths.Pipeline` and kept backward
compatibility with `watchedFoldersDir`
(app/common/src/main/java/stirling/software/common/model/ApplicationProperties.java).
- Resolved pipeline base and multiple watched folder paths in
`RuntimePathConfig` and exposed `getPipelineWatchedFoldersPaths()`
(app/common/src/main/java/stirling/software/common/configuration/RuntimePathConfig.java).
- Updated `FileMonitor` to accept and register multiple root paths
instead of a single root
(app/common/src/main/java/stirling/software/common/util/FileMonitor.java).
- Updated `PipelineDirectoryProcessor` to iterate all configured watched
roots and to walk subdirectories while ignoring `processing` dirs
(app/core/src/main/java/stirling/software/SPDF/controller/api/pipeline/PipelineDirectoryProcessor.java).
- Exposed the new settings in `settings.yml.template` and the admin UI,
including a multi-line `Textarea` to edit `watchedFoldersDirs`
(app/core/src/main/resources/settings.yml.template,
frontend/src/proprietary/components/shared/config/configSections/AdminGeneralSection.tsx).
- Adjusted unit test setup to account for list-based watched folders
(app/common/src/test/java/stirling/software/common/util/FileMonitorTest.java).

### Testing
- Ran formatting and build checks with `./gradlew spotlessApply` and
`./gradlew build` using Java 21 via
`JAVA_HOME=/root/.local/share/mise/installs/java/21.0.2
PATH=/root/.local/share/mise/installs/java/21.0.2/bin:$PATH ./gradlew
...`, but both runs failed due to Gradle plugin resolution being blocked
in this environment (plugin portal/network 403), so full
compilation/formatting could not complete.
- Confirmed the code compiles locally was not possible here; unit test
`FileMonitorTest` was updated to use the new API but was not executed
due to the blocked build.
- Changes were committed (`Support multiple pipeline watch directories`)
and the repository diff contains the listed file modifications.

------
[Codex
Task](https://chatgpt.com/codex/tasks/task_b_69741ecd17c883288d8085a63ccd66f4)
This commit is contained in:
Anthony Stirling
2026-01-31 20:59:25 +00:00
committed by GitHub
parent 2ae413c5ea
commit 4f404a1ccf
9 changed files with 452 additions and 35 deletions
@@ -9,6 +9,7 @@ import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.attribute.FileTime;
import java.time.Instant;
import java.util.List;
import java.util.function.Predicate;
import org.junit.jupiter.api.BeforeEach;
@@ -34,7 +35,8 @@ class FileMonitorTest {
@BeforeEach
void setUp() throws IOException {
when(runtimePathConfig.getPipelineWatchedFoldersPath()).thenReturn(tempDir.toString());
when(runtimePathConfig.getPipelineWatchedFoldersPaths())
.thenReturn(List.of(tempDir.toString()));
// This mock is used in all tests except testPathFilter
// We use lenient to avoid UnnecessaryStubbingException in that test