Source Configuration
This page details the configuration options available when setting up source connectors in your workflows. Source configuration define how files are collected and prepared for processing.
File System Source Configuration
When using a File System as your source connector, you can configure the following settings:
Folders to process
Specify one or more folders in the connected filesystem from which to start processing files. Enter absolute paths of the file system or select a folder from the dialog and click Add Folder to add it to the list. Leave empty or use / to process from the root folder.
Process sub-folders
When enabled, the workflow processes files in all subfolders recursively, up to 10 levels deep. When disabled, only files in the specified folders are processed.
Recommendation: Enable this option if your files are organized in nested folder structures.
File types to process
Limit processing to specific file types. If no types are selected, all supported file types will be processed.
Available Options:
- PDF documents
- Text documents
- Images
Max files to process
Set the maximum number of files to process in a single workflow execution. This helps control resource usage and processing time.
Recommendations:
- Start with smaller limits when testing workflows
- Increase the limit based on your processing capacity
- Consider file size and complexity when setting this value
File processing order
Determines the order in which files are processed from the source.
Available Options:
| Option | Description | Use Case |
|---|---|---|
| Unordered | No explicit ordering; files are processed as enumerated by the operating system | Fastest option; use when order doesn't matter |
| Oldest First (FIFO) | Process files in ascending order of last-modified time | Process older files first; chronological processing |
| Newest First (LIFO) | Process files in descending order of last-modified time | Prioritize recent files; latest data first |
File ordering adds processing overhead. If you have more than 40,000 files, the ordering may not reflect all files in the source folder accurately.
File metadata deletion behavior
Controls how the workflow handles file metadata. This is used to keep track of files that have been processed before. Only file path is stored and not file content.
Available Options:
- Never delete: Use this to automatically skip files that have already been processed
- Delete after: Specify a duration after which this file metadata will be deleted and files will be reprocessed as a result
Use Cases:
- Periodic reprocessing of updated documents that fall with the same name
- Regular audits or compliance checks
- Data refresh workflows
Configuration Storage
All source configuration settings are stored in the workflow's backend configuration as JSON. These settings can be modified after workflow creation by editing the workflow configuration.