Skip to main content

Create an ETL Workflow for exported Tool

  1. Export the Prompt studio project as a tool

  2. Now that you've exported a Prompt studio project, let's create a Workflow that we'll launch as an ETL. First off, let's create a new empty Workflow:

    • From the side navigation menu, choose Workflows 🞂 New Workflow. This should bring up the new Workflow dialog: img Create New Workflow
    • For Workflow Name enter any name for your workflow.
    • For Description enter some detail about the type of workflow
    • Click on Create Workflow. This should land you in an empty workflow in the Workflow Studio, which allows you to build sophisticated workflows.

  3. Let's configure the workflow. img Create New Workflow

    • Input Configuration: To build an ETL Pipeline, choose File System from this list and click on the settings button near the dropdown and configure the input file system
    • Output Configuration: To build an ETL Pipeline, choose Database from this list and click on the settings button near the dropdown and configure the database system
    • Tools Pane: this pane lists all built in and user created custom Tools available in the platform. Prompt Studio projects you export are made available as Tools in this pane from where you an drop and drop them into the Workflow Chain to construct Workflows you want.
    • Workflow Chain: Search for your prompt studio exported project under Tools, and drag and drop it in the workflow container(in between Input and output configuration)
    • Tool Settings Pane: This is where settings for various tools are made available. You'll need to select a Tool from the Workflow Chain in order to change the settings of Tools from here.Below is the detailed screenshot for point no.5 img Create New Workflow
    • Deploy As ETL Pipeline: use this button to deploy workflows as unstructured data ETL Pipelines.

  4. Now, click on the Deploy As ETL button and you should see a dialog pop up: img Create New Workflow

    • For Display Name enter the name for the ETL pipeline
    • In Cron Schedule choose the time at which your ETL needs to be triggered.It should in UTC time.

  5. Once you deployed the ETL pipeline, it will automatically trigger at the configured time.

    • For Execution logs Click on ETL Pipeline-->Actions button of your ETL task-->View Logs
    info
    1. Once the file is processed in ETL, it will be cached. To run the cached file again, Clear cache and Clear Processed File History under ETL Pipeline-->Actions button of your ETL task
    2. To trigger the ETL pipeline manually, click on Manual Sync under ETL Pipeline-->Actions button of your ETL task

    img Create New Workflow