Skip to main content

Task Pipeline Introduction

Unstract's Task Pipeline handles file transfer and processing workflows between storage systems. Unlike ETL Pipelines, it focuses on moving files rather than data transformation.

Key Features​

Smart File Handling​

Unstract's Task pipelines are designed for maximum efficiency and reliability:

  • Keeps track of new files in each pipeline run to avoid reprocessing
  • Skips duplicate files (same name, path, and content) automatically
  • Executes files in parallel for faster batch processing
  • Retries execution on failure to ensure reliability

Getting Started​

To create your first Task Pipeline, see Set Up Task Pipeline

Execution and Monitoring​

Execution Modes:

  • Manual: Instant execution via UI (Task Pipeline > Actions > Sync Now)
  • Scheduled: Automated cron-based execution configured in pipeline settings
  • API: REST integration via Task Pipeline > Actions > Download Postman Collection

Monitoring:

  • Real-time status: Dashboard view and run details
  • Execution history: MANAGE > Logs > Task Sessions

Notifications:

  • Task Pipelines > Actions > Setup Notifications

Next Steps​