Optional: Set up a Challenger LLM

info

LLMEval is only part of Unstract Cloud and Unstract Enterprise offerings. This section assumes that you have a working Google Cloud Platform (GCP) account, using which we'll configure a Gemini Pro LLM for use with the Unstract Platform as an evaluator LLM.

Unstract Cloud and Enterprise editions feature LLMEval, an advanced feature that improves accuracy by employing a second LLM to evaluate the output of the first one that does the structuring. In this section, we'll look at how to set up Google's Gemini Pro LLM for use in the Unstract Platform as an Evaluator LLM.

note

Please note that it is always a good idea to choose an LLM from a completely different provenance or origin as the evaluator LLM as compared to the LLM that is used to structure documents. When two LLMs from very different origins that feature potentially different internal architectures agree that a field extraction is right, the probability that it is indeed right, is high.

Set up a GCP Service Account for Vertex AI

What we're looking for is to gain access to Google's Gemini Pro LLM. It is made available under the Vertex AI umbrella. We'll create a service account in GCP which will have permissions to use Vertex AI and thus the Gemini Pro APIs.

Although it might seem as if this is a long and confusing process, it is not if you understand what we're trying to achieve. GCP uses Service Accounts to associate roles and services. It's a great security practice that lets you granularly control which services a Service Account has access to while also controlling who has access to it. Here, we'll create a Service Account with access to the Vertex APIs. We'll create and download a private key with which we're able to access that Service Account. Finally, any service you'll want to use under GCP has to be explicitly enabled on a per project basis. After we do all this, we'll add the Gemini Pro LLM to Unstract.

Let's get started.

info

Everything described here is done in the context of a project. Make sure you've selected the right project to be the active one in the GCP console.

First off, let's create a new Service Account. Go to:

Main menu (top left hamburger menu) 🞂 IAM & Admin 🞂 Service Accounts 🞂 Create Service Account. You should now see a screen similar to the following:

• For Service account name, enter Unstract
• For Service account ID, enter unstract
• For Service account description, enter Unstract LLM Profile
• Click on Create and Continue

Now, we'll grant this service account access to the Vertex AI services.

• In the Select a role field, type vertex ai service agent and choose it from the list.
• This should take you to the optional next step.

• We will ignore this optional step by simple clicking on Done.
• This should take us to the service accounts list page which should look similar to the screenshot below.

• You might recall that we named our service account Unstract. Find it and click on the Actions disclosure, which reveals a menu from where you can click on Manage keys to create a key which will allow us to access this service account.

• We'll need to create a key to access this service account. For this, click on the Add key button and choose Create new key.
• This should bring you to the Create private key for "Unstract" dialog.

• Choose the key type to be JSON and click Create. This should download the key to your computer.

• Excellent! We now have a way to access the service account. We now need to enable Vertex AI for your in your GCP project.

• Go to Main menu (top left hamburger menu) 🞂 APIs & Services 🞂 Enabled APIs & services. You should now see a screen similar to the following:

• You should see a screen similar to the one below:

• Now, click on the Enable APIs and Services button. You'll be taken to a screen where you can search for APIs and services that you can enable for your project, much like the one below:

• Search for "vertext ai api" and select it from the list below. You should now be taken to the Vertex AI API page.

• Click on the Enable button to enable the Vertex AI APIs for your project.
• This concludes our work in the GCP console. Let's now move on to the Unstract console.
• Open the downloaded private key in a text editor. It should look something like this:

• You'll need to select the whole text and copy it, you'll also need to copy the value of the project_id key as well, which in example is smiling-rythm-168501. Yours will be something else, of course.
• Sign into the Unstract Platform
• From the side navigation menu, choose Settings 🞂 LLMs
• Click on the New LLM Profile button
• From the list of supported LLMs, choose VertexAI. You should see a dialog box where you enter details.

• For Name, enter a name for this connector.
• In Credentialsfield, paste in the whole downloaded service account key we copied from the text editor earlier.
• For Models field,enter gemini-1.5-pro.For more models refer to Gemini Models.
• For Max output tokens,enter 8192. For more models and its token limit refer to Gemini Models.
• For Project, enter the Google project name in which the Vertex AI is enabled (or) Check the project_id in credentials json.
• Leave Max Retries value to its default.
• Click on Test Connection and ensure it succeeds. You can finally click on Submit and that should create a new Evaluator LLM Profile for use in your Unstract projects.