📄️ LLMChallenge: Introduction
Large Language Models can work really well for document data extraction use cases. The trouble with them however, is that they can hallucinate. And worse, there is no reliably way to detect if an LLM has indeed hallucinated. If Large Language Models are to be deployed in production for at-scale use cases, users need to be sure that there are no hallucinations happening. Wrong extractions undermine trust in the system.
📄️ Developing and verifying LLMChallenge
When developing your prompts in Prompt Studio, you can enable LLMChallenge and see how it affects your extractions. Of course, in Prompt Studio, you're only developing prompts and defining the extraction schema. Ultimately, LLMChallenge runs as part of a deployed Prompt Studio project as one of the following integration types:
📄️ Deploying LLMChallenge
We discussed earlier in Developing and verifying LLMChallenge that while in Prompt Studio, you're only verifying if LLMChallenge is doing a good job of required the structured extraction. LLMChallenge needs to be deployed as part of API Deployments, ETL Pipelines, Human Quality Review or Task Pipelines workflow. In this section, let's see how, when deploying one of the previously mentioned types of workflows, we can enable LLMChallenge.
📄️ Post-deployment verification
In this section, we'll see how to request metadata as part of the API response so that we can get the LLMChallenge log and costs.