Skip to main content

Set Up ETL Pipeline

This guide shows you how to create and configure an ETL Pipeline in Unstract.

Prerequisites​

  1. Export a Prompt Studio project as a tool
  2. Configure your source filesystem connector
  3. Configure your destination database connector

Creating the ETL Pipeline​

  1. Navigate to Workflows in the side navigation
  2. Create a new workflow following the Workflow Setup Guide
  3. Configure your workflow:
    • Source: Select your filesystem connector
    • Destination: Select your database connector
    • Tool: Select your exported Prompt Studio project
  4. Click Deploy as ETL

Option 2: Direct ETL Creation​

  1. Navigate to ETL Pipelines in the side navigation
  2. Click New ETL Pipeline
  3. Select an existing workflow or create a new one
  4. Configure ETL-specific settings (see below)

ETL-Specific Configuration​

Scheduling​

Set up automated execution with cron scheduling:

  1. For Display Name, enter a descriptive name for your ETL pipeline
  2. In Cron Schedule, specify when the ETL should trigger (in UTC time)
    • Example: 0 2 * * * (runs daily at 2:00 AM UTC)

ETL Deployment Configuration

Deployment Confirmation​

Once deployed, the ETL will automatically trigger at the configured time.

Managing Your ETL Pipeline​

Manual Execution​

Trigger your ETL pipeline on demand:

  • ETL Pipeline → Actions → Manual Sync

Monitoring Execution​

View execution logs and status:

  • ETL Pipeline → Actions → View Logs

Or navigate to: MANAGE → Logs → ETL Sessions

File Processing Behaviour

Once a file is processed in an ETL pipeline, it is tracked to avoid reprocessing. If you wish to reprocess the files, you can clear the processed file history.

  • Clear Processed File History: ETL Pipeline → Actions → Clear Processed File History

Notifications​

Set up alerts for your ETL pipeline:

ETL Pipeline → Actions → Setup Notifications