Optional: Set up a Challenger LLM
LLMEval is only part of Unstract Cloud and Unstract Enterprise offerings. This section assumes that you have a working Google Cloud Platform (GCP) account, using which we'll configure a Gemini Pro LLM for use with the Unstract Platform as an evaluator LLM.
Unstract Cloud and Enterprise editions feature LLMEval, an advanced feature that improves accuracy by employing a second LLM to evaluate the output of the first one that does the structuring. In this section, we'll look at how to set up Google's Gemini Pro LLM for use in the Unstract Platform as an Evaluator LLM.
Please note that it is always a good idea to choose an LLM from a completely different provenance or origin as the evaluator LLM as compared to the LLM that is used to structure documents. When two LLMs from very different origins that feature potentially different internal architectures agree that a field extraction is right, the probability that it is indeed right, is high.
Set up a GCP Service Account for Vertex AI
What we're looking for is to gain access to Google's Gemini Pro LLM. It is made available under the Vertex AI umbrella. We'll create a service account in GCP which will have permissions to use Vertex AI and thus the Gemini Pro APIs.
Although it might seem as if this is a long and confusing process, it is not if you understand what we're trying to achieve. GCP uses Service Accounts to associate roles and services. It's a great security practice that lets you granularly control which services a Service Account has access to while also controlling who has access to it. Here, we'll create a Service Account with access to the Vertex APIs. We'll create and download a private key with which we're able to access that Service Account. Finally, any service you'll want to use under GCP has to be explicitly enabled on a per project basis. After we do all this, we'll add the Gemini Pro LLM to Unstract.
Let's get started.
Everything described here is done in the context of a project. Make sure you've selected the right project to be the active one in the GCP console.
First off, let's create a new Service Account. Go to:
Main menu
(top left hamburger menu) 🞂 IAM & Admin
🞂 Service Accounts
🞂 Create Service Account
. You should now see a screen similar to the following:
- For
Service account name
, enterUnstract
- For
Service account ID
, enterunstract
- For
Service account description
, enterUnstract LLM Profile
- Click on
Create and Continue
Now, we'll grant this service account access to the Vertex AI services.
- In the
Select a role
field, typevertex ai service agent
and choose it from the list. - This should take you to the optional next step.
- We will ignore this optional step by simple clicking on
Done
. - This should take us to the service accounts list page which should look similar to the screenshot below.
- You might recall that we named our service account
Unstract
. Find it and click on theActions
disclosure, which reveals a menu from where you can click onManage keys
to create a key which will allow us to access this service account.
- We'll need to create a key to access this service account. For this, click on the
Add key
button and chooseCreate new key
. - This should bring you to the
Create private key for "Unstract"
dialog.
-
Choose the key type to be
JSON
and clickCreate
. This should download the key to your computer. -
Excellent! We now have a way to access the service account. We now need to enable Vertex AI for your in your GCP project.
-
Go to
Main menu
(top left hamburger menu) 🞂APIs & Services
🞂Enabled APIs & services
. You should now see a screen similar to the following:
- You should see a screen similar to the one below:
- Now, click on the
Enable APIs and Services
button. You'll be taken to a screen where you can search for APIs and services that you can enable for your project, much like the one below:
- Search for "vertext ai api" and select it from the list below. You should now be taken to the Vertex AI API page.
- Click on the
Enable
button to enable the Vertex AI APIs for your project. - This concludes our work in the GCP console. Let's now move on to the Unstract console.
- Open the downloaded private key in a text editor. It should look something like this:
- You'll need to select the whole text and copy it, you'll also need to copy the value of the
project_id
key as well, which in example issmiling-rythm-168501
. Yours will be something else, of course. - Sign into the Unstract Platform
- From the side navigation menu, choose
Settings
🞂LLMs
- Click on the
New LLM Profile
button - From the list of supported LLMs, choose
VertexAI
. You should see a dialog box where you enter details.
- For
Name
, enter a name for this connector. - In
Credentials
field, paste in the whole downloaded service account key we copied from the text editor earlier. - For
Models
field,entergemini-1.5-pro
.For more models refer to Gemini Models. - For
Max output tokens
,enter8192
. For more models and its token limit refer to Gemini Models. - For
Project
, enter the Google project name in which the Vertex AI is enabled (or) Check theproject_id
in credentials json. - Leave
Max Retries
value to its default. - Click on
Test Connection
and ensure it succeeds. You can finally click onSubmit
and that should create a new Evaluator LLM Profile for use in your Unstract projects.