Skip to main content

Create an ETL Workflow for exported Tool

  1. Export the Prompt studio project as a tool

  2. Now that you've exported a Prompt studio project, let's create a Workflow that we'll launch as an ETL pipeline. Navigate to Workflows in the side navigation menu and create a new workflow.

  3. Let's configure the workflow using the new streamlined workflow setup interface: img Create New Workflow

    The workflow setup is now organized into 4 main configuration steps: img Create New Workflow

    1. Configure Source Connector:

      • Select and configure your data input connector
      • Choose File System from the dropdown
      • Click the Configure button to configure the input file system
      • You'll see your configured connector (e.g., "gallagher") displayed below

    2. Configure Destination Connector:

      • Select and configure your data output connector
      • Choose Database from the dropdown
      • Click the Configure button to configure the database system
      • You'll see your configured connector (e.g., "gallagher_dev") displayed below

    3. Select Exported Prompt Studio project:

      • Choose an exported Prompt Studio project for processing your data
      • Your exported project will be displayed (e.g., "gallagher_staging_aug27")
      • Use Change Prompt Studio project link to select a different project
      • Click Configure Settings to modify the tool settings

    4. Deploy Workflow:

      • Deploy your workflow for processing
      • Use Deploy as ETL button to deploy workflows as unstructured data ETL Pipelines
      • Alternatively, use Deploy Workflow for other deployment options img Create New Workflow
      • For Display Name enter the name for the ETL pipeline
      • In Cron Schedule choose the time at which your ETL needs to be triggered.It should in UTC time.

  4. Once you deployed the ETL pipeline, it will automatically trigger at the configured time.

    • For Execution logs Click on ETL Pipeline-->Actions button of your ETL task-->View Logs
    info
    1. Once the file is processed in ETL, it will be cached. To run the cached file again, Clear cache and Clear Processed File History under ETL Pipeline-->Actions button of your ETL task
    2. To trigger the ETL pipeline manually, click on Manual Sync under ETL Pipeline-->Actions button of your ETL task

    img Create New Workflow