Skip to main content
Version: 2.0.0

On-Prem Deployment Guide

This guide provides comprehensive instructions for deploying LLMWhisperer in an on-premises environment. LLMWhisperer is deployed inside a Kubernetes cluster, packaged as a Helm chart.

Overview​

LLMWhisperer On-Prem is a self-hosted deployment that runs entirely within your infrastructure. It includes:

  • LLMWhisperer Backend — the core text extraction API service
  • LLMWhisperer Dashboard — a web UI for usage monitoring and management
  • OCR Workers — document processing workers that scale based on load
  • RabbitMQ — message broker for distributed task processing
  • Redis — caching layer for performance optimization

1. Infrastructure Prerequisites​

Kubernetes Cluster​

  • Recommended version: >= 1.29 (latest tested: 1.33)
  • Node autoscaling should be enabled
  • Supports both single Availability Zone and multi-AZ deployments. Multi-AZ requires enabling HA mode for stateful workloads (RabbitMQ quorum queues, Redis multi-AZ overlay)
  • Ingress controller as a K8s cluster add-on for load balancer creation (recommended)
    • Ingress requires a maximum timeout of 900 seconds to work as expected
  • In-house or cloud provider observability stack (recommended)

PostgreSQL Database​

  • Supported versions: 15 or 17
  • Minimum specs: 1 vCPU, 8 GiB RAM, 50 GiB SSD
  • Autoscale enabled (recommended)
  • A dedicated database for LLMWhisperer should be created within the PostgreSQL instance

DNS & SSL​

  • A domain for pointing to LLMWhisperer (e.g., llmwhisperer.<customer-domain>.com)
  • An active SSL certificate is required for the domain

Node Profile​

Add 50 GiB SSD for application data to each machine.

Machine TypeLabelTaint (NoSchedule)MinMax
8 vCPU and 32 GiBservice: llmwhispererservice: llmwhisperer160

GPU Nodes (Optional — for document insights mode)

Cloud ProviderInstance TypeGPU FamilyLabelTaint (NoSchedule)MinMax
AWSg6.xlargeNVIDIA L4 Tensor Coreservice: llmwhisperer-gpuservice: llmwhisperer-gpu11
GCPg2-standard-4NVIDIA L4 Tensor Coreservice: llmwhisperer-gpuservice: llmwhisperer-gpu11
warning

It is expected that the workloads are to be deployed on non-spot nodepools.

2. Configuration​

Files Provided by Unstract Team​

The following files will be provided by the Unstract team:

FileDescription
artifact-key.jsonGCP service account key for Helm chart registry login and container image pull
sample.onprem.values.yamlSample Helm chart values (non-sensitive configuration)
onprem-profile.values.yamlProfile values for resource allocation and scaling configuration
tip

The sample.onprem.values.yaml and onprem-profile.values.yaml files are bundled inside each Helm chart release. You can extract them for any version — see Download Configuration Files for instructions.

Required Configuration Values​

These values must be provided by the customer or the Unstract team to deploy LLMWhisperer:

VariableDescriptionSource
DB_LLMW_HOSTPostgreSQL hostCustomer
DB_LLMW_USERNAMEPostgreSQL usernameCustomer
DB_LLMW_PASSWORDPostgreSQL passwordCustomer
DB_LLMW_NAMEPostgreSQL database nameCustomer
ENCRYPTION_KEYEncryption key for sensitive data — must be backed up securelySelf-generated
LICENSE_PORTAL_API_KEYLicense portal API keyUnstract Team
endpoint (azureOcrBilling)Azure Cognitive Services OCR endpoint — required for v2.59.x and earlier; not required from v2.60.0 onwardsUnstract Team
apiKey (azureOcrBilling)Azure OCR API key — required for v2.59.x and earlier; not required from v2.60.0 onwardsUnstract Team
INITIAL_PASSWORDInitial admin password for the dashboardCustomer
X_CELERY_BROKER_USERNAMERabbitMQ usernameCustomer
X_CELERY_BROKER_PASSWORDRabbitMQ passwordCustomer
warning

The ENCRYPTION_KEY is used to encrypt data at rest and is required when retrieving the data. Do not rotate, delete, or lose this key — doing so will render existing encrypted data inaccessible.

Using Kubernetes Secrets (existingSecret)​

Each configuration section in sample.onprem.values.yaml supports two approaches for providing sensitive values:

Option 1: Inline values — Provide values directly in the values file. Suitable for initial setup and testing.

global:
  sharedConfigs:
    database:
      DB_LLMW_HOST: "postgres.example.com"
      DB_LLMW_USERNAME: "postgres"
      DB_LLMW_PASSWORD: "your-password"
      DB_LLMW_NAME: "llmwhisperer"

Option 2: Kubernetes secrets (recommended for production) — Pre-create Kubernetes secrets with the matching variable names as keys, then reference the secret name via existingSecret. This avoids storing sensitive values in the Helm values file.

global:
  sharedConfigs:
    database:
      existingSecret: "llmwhisperer-db-credentials"

The following configuration sections support existingSecret:

SectionExample Secret NameKeys
global.sharedConfigs.databasellmwhisperer-db-credentialsDB_LLMW_HOST, DB_LLMW_USERNAME, DB_LLMW_PASSWORD, DB_LLMW_NAME, DB_LLMW_PORT
global.sharedConfigs.redisllmwhisperer-redis-credentialsREDIS_HOST, REDIS_PORT, REDIS_DB, REDIS_PASSWORD, REDIS_USER
global.sharedConfigs.workerRedisllmwhisperer-worker-redis-credentialsWORKER_REDIS_HOST, WORKER_REDIS_PORT, WORKER_REDIS_DB, WORKER_REDIS_PASSWORD
global.sharedConfigs.celeryBrokerllmwhisperer-celery-broker-credentialsX_CELERY_BROKER_BASE_URL, X_CELERY_BROKER_USERNAME, X_CELERY_BROKER_PASSWORD, X_CELERY_BACKEND_URL
global.sharedConfigs.apiKeysllmwhisperer-api-keysENCRYPTION_KEY
global.sharedConfigs.dashboardCredentialsllmwhisperer-dashboard-credentialsINITIAL_USER_NAME, INITIAL_PASSWORD
global.sharedConfigs.licensellmwhisperer-license-secretLICENSE_PORTAL_URL, LICENSE_PORTAL_API_KEY, LLMW_API_KEYS
global.azureOcrBillingazure-ocr-billing-credentialsendpoint, apiKey (required for v2.59.x and earlier; not required from v2.60.0 onwards)

Example of creating a Kubernetes secret:

kubectl create secret generic llmwhisperer-db-credentials \
  --namespace $NAMESPACE \
  --from-literal=DB_LLMW_HOST="postgres.example.com" \
  --from-literal=DB_LLMW_USERNAME="postgres" \
  --from-literal=DB_LLMW_PASSWORD="your-password" \
  --from-literal=DB_LLMW_NAME="llmwhisperer" \
  --from-literal=DB_LLMW_PORT="5432"

API Key Authentication (Optional)​

You can enable request-level API key authentication for the LLMWhisperer backend. When enabled, every request must include a valid API key in the unstract-key header. Each key is associated with a key_id for per-key usage auditing.

To enable, set the LLMW_API_KEYS environment variable under the license section in your values file:

global:
  sharedConfigs:
    license:
      LLMW_API_KEYS: "key_id1:api_key_1,key_id2:api_key_2"

The format is a comma-separated list of key_id:key pairs, where each key_id must be a valid UUID. You can generate one with:

python3 -c "import uuid; print(uuid.uuid4())"

For example:

LLMW_API_KEYS: "550e8400-e29b-41d4-a716-446655440000:sk-abc123,6fa459ea-ee8a-3ca4-894e-db77e160355e:sk-def456"

Callers must then include the API key in their requests:

curl -H "unstract-key: sk-abc123" https://llmwhisperer.example.com/api/v2/whisper
tip

This feature is opt-in. If LLMW_API_KEYS is not set or is empty, the backend accepts all requests without authentication — preserving backward compatibility.

3. Installation (One-Time)​

Step 1: Check Cluster Connectivity​

kubectl cluster-info

Step 2: Deploy RabbitMQ Operator (Once Per Cluster)​

RabbitMQ operator is used for provisioning the RabbitMQ cluster within the namespace using its CRD. Refer to the official documentation.

kubectl apply -f "https://github.com/rabbitmq/cluster-operator/releases/download/v2.11.0/cluster-operator.yml"
tip

Pin a specific operator version (e.g. v2.11.0) instead of releases/latest, so installs stay reproducible and match the version you mirror for air-gapped clusters.

Private registry / air-gap

The RabbitMQ operator ships as a raw manifest, so there is no Helm --set to override its image. Download the pinned manifest, repoint the operator image to your mirror, and apply the edited file:

VERSION=v2.11.0
curl -fsSL "https://github.com/rabbitmq/cluster-operator/releases/download/${VERSION}/cluster-operator.yml" -o cluster-operator.yml
sed -i.bak 's#rabbitmqoperator/cluster-operator:[^[:space:]]*#harbor.corp.example.com/llmwhisperer/cluster-operator:2.11.0#' cluster-operator.yml
kubectl apply -f cluster-operator.yml

The RabbitMQ broker image (set on the RabbitmqCluster CR) is separate and follows global.image.registry, so it is already covered by your private-registry redirect — see Appendix e.

Scheduling the operator onto a node pool

The RabbitMQ broker pods need no special handling — they auto-derive node affinity from global.nodeSelector and tolerate the matching taint, so they follow your Node Profile along with the rest of the platform.

The operator, being a raw manifest, does not read global.nodeSelector. If your cluster requires every pod to carry an explicit nodeSelector/toleration, pin it with a one-liner on the downloaded cluster-operator.yml — the same edit-then-apply flow you use to repoint its image. yq inserts the structured fields cleanly (no indentation guesswork):

yq -i 'select(.kind == "Deployment" and .metadata.name == "rabbitmq-cluster-operator") |= (.spec.template.spec.nodeSelector.service = "llmwhisperer" | .spec.template.spec.tolerations = [{"key": "service", "operator": "Equal", "value": "llmwhisperer", "effect": "NoSchedule"}])' cluster-operator.yml
kubectl apply -f cluster-operator.yml

No yq on the box, or the operator is already applied? A kubectl patch does the same to the live Deployment (safe to re-run):

kubectl -n rabbitmq-system patch deployment rabbitmq-cluster-operator --type merge -p '
spec:
  template:
    spec:
      nodeSelector:
        service: llmwhisperer
      tolerations:
        - {key: service, operator: Equal, value: llmwhisperer, effect: NoSchedule}
'

These fields do not survive an operator upgrade — re-applying the upstream cluster-operator.yml resets the Deployment to the manifest's contents. Bake the change into your copy of the manifest (the yq edit above) so it is versioned, or re-run the patch after each operator upgrade.

Step 3: Create Namespace​

export NAMESPACE=<namespace_name>
kubectl create namespace $NAMESPACE

Step 4: Authenticate Helm Registry​

cat artifact-key.json | helm registry login -u _json_key --password-stdin https://us-central1-docker.pkg.dev

Step 5: Create Image Pull Secret​

kubectl create secret docker-registry artifact-registry \
  --namespace $NAMESPACE \
  --docker-server=us-central1-docker.pkg.dev \
  --docker-username=_json_key \
  --docker-password="$(cat artifact-key.json)"

Validate the secret was created successfully:

kubectl get secret artifact-registry -n $NAMESPACE -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d

Step 6: Configure Values File​

  1. Create a copy of sample.onprem.values.yaml as onprem.values.yaml
  2. Fill in all values marked with # <REQUIRED> — refer to the Configuration section for details on each value

Step 7: Install Helm Chart​

  • Requires 3 x 8 vCPU 32 GB nodes by default
  • Processes ~1,800 pages/hour, maximum concurrency of 10 pages, response time of 12–14 seconds
  • With HPA enabled: up to ~7,200 pages/hour, concurrency of 30 pages, response time of 15–16 seconds (uses ~9 x 8 vCPU machines)
  • Capacity can be further tuned based on the processing modes in use
warning

The Helm release name must be whisperer and must not be changed. Internal service communication relies on this name, and using a different release name will cause the deployment to fail.

helm install whisperer oci://us-central1-docker.pkg.dev/pandoras-tamer/charts/llmwhisperer \
  --version <version> \
  -f /path/to/onprem.values.yaml \
  -f /path/to/onprem-profile.values.yaml \
  -n $NAMESPACE

Replace <version> with the target release version (see Version History).

4. Deployment Validation​

Health Checks​

ServicePortNetwork TypeEndpoint
whisperer-backend3006HTTP/health/ping
llmwhisperer-dashboard3007HTTP/health/ping

Validation Steps​

  1. Check that all pods in the namespace are running without restarts:

    kubectl get pods -n $NAMESPACE

  2. Validate the ingress configured for both the LLMWhisperer dashboard and the backend

  3. Log in to the dashboard using the credentials configured in onprem.values.yaml

  4. Validate the backend API — refer to the API documentation

5. Upgrading​

warning

Do not reuse older onprem.values.yaml or onprem-profile.values.yaml files as-is. Each release may add, rename, or remove Helm values, so before every upgrade re-extract both files from the target release version and reconcile them with the ones you deploy with:

  • Diff your existing onprem.values.yaml against the sample.onprem.values.yaml bundled in the target release and bring over any added, renamed, or changed keys — keeping your own configured values.
  • Deploy the onprem-profile.values.yaml bundled in the target release rather than an older copy, since resource and scaling defaults change between releases. If you overrode anything in it, re-apply those overrides on top of the new file.

See Download Configuration Files to obtain both files for a specific version. Value changes that need action are called out under Upgrade Notes for each release in the Version History.

  1. Configure onprem.values.yaml as required for the target release version
  2. Run the upgrade command:
helm upgrade whisperer oci://us-central1-docker.pkg.dev/pandoras-tamer/charts/llmwhisperer \
  --version <version> \
  -f /path/to/onprem.values.yaml \
  -f /path/to/onprem-profile.values.yaml \
  -n $NAMESPACE
info

If you are on AWS Ingress and upgrading from a version older than v2.36.0, ensure the following annotation is present: alb.ingress.kubernetes.io/target-type: ip (see Appendix a).

6. Admin Login / Onboarding​

Once LLMWhisperer is successfully deployed, log in to the LLMWhisperer Dashboard using the INITIAL_USER_NAME and INITIAL_PASSWORD configured during installation.

For a walkthrough of the dashboard screens (Home, Usage, License), see the LLMWhisperer Dashboard Guide.

Appendix​

a. Ingress Configuration​

All ingress types must support a 900-second timeout.

AWS ALB Ingress Controller

  • Ingress configuration in EKS Auto Mode

  • Required annotation:

    # REF: https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/how-it-works/#ip-mode
    alb.ingress.kubernetes.io/target-type: ip

Nginx Ingress Controller

Required annotations (Community Version syntax):

# Default is 60. Must be increased to 900.
nginx.ingress.kubernetes.io/proxy-read-timeout: "900"
# Default is 1 MB. Must be increased for large document uploads.
# REF: https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/
nginx.org/client-max-body-size: "200m"
warning

Avoid using nginx.ingress.kubernetes.io/rewrite-target annotation. In Community NGINX Controller versions >= v0.22.0, the old rewrite-target: / syntax causes authentication failures (401 Unauthorized responses). If you encounter login issues, remove any rewrite-target annotations from your ingress configuration.

b. Outgoing Data (OCR Billing)​

In an on-prem deployment, the only outgoing data from the OCR containers is billing information sent to Unstract for metering purposes (see routing details below). No document content leaves your infrastructure.

Required outbound whitelisting

VersionEndpoints to whitelist
v2.60.0 and laterUnstract License Portal (LICENSE_PORTAL_URL, default https://license-portal.unstract.com)
Pre-v2.60.0Unstract License Portal (LICENSE_PORTAL_URL) and Azure Cognitive Services OCR endpoints
info

From v2.60.0 onwards, all outgoing billing traffic is routed through the centralized OCR billing proxy hosted by the Unstract License Portal, so Azure Cognitive Services endpoints no longer need to be whitelisted.

For versions prior to v2.60.0, billing information is sent directly to Azure in addition to license traffic going to the License Portal. You can find details about how Azure container billing works here.

tip

Upgrading from a pre-v2.60.0 release? Existing azureOcrBilling.endpoint and apiKey values can stay in your values.yaml (they are ignored) or be removed.

The following billing data is sent to Unstract for license metering:

{
    "subscription_id": "<subscription_id:uuid4>",
    "deployment_id": "<deployment_id:uuid4>",
    "page_count_total": "<total_page_count:int>",
    "native_text_page_count": "<non_ocr_page_count:int>",
    "low_cost_page_count": "<low_cost_page_count:int>",
    "high_quality_page_count": "<high_quality_page_count:int>",
    "form_page_count": "<form_page_count:int>",
    "from_date": "<timestamp>",
    "to_date": "<timestamp>"
}

c. Multi-AZ Deployment​

By default, LLMWhisperer deploys in a single Availability Zone. For production environments that require high availability across zones, a multi-AZ overlay is provided that configures the stateful workloads accordingly.

What the overlay enables​

ComponentMulti-AZ Behavior
RabbitMQHA mode with a 3-node cluster using quorum queues (Raft-based replication). Replaces classic queues via a dual-queue migration strategy for zero-downtime transitions.
RedistopologySpreadConstraints spread replicas across zones (maxSkew: 1) with podAntiAffinity to prevent co-location on the same node.

Prerequisites​

  • Kubernetes cluster with nodes spanning at least 2 Availability Zones
  • Node labels with zone topology (standard topology.kubernetes.io/zone label)

Enabling multi-AZ​

Include the multiaz.values.yaml overlay during install or upgrade:

helm install whisperer oci://us-central1-docker.pkg.dev/pandoras-tamer/charts/llmwhisperer \
  --version <version> \
  -f /path/to/onprem.values.yaml \
  -f /path/to/onprem-profile.values.yaml \
  -f /path/to/multiaz.values.yaml \
  -n $NAMESPACE

For upgrades:

helm upgrade whisperer oci://us-central1-docker.pkg.dev/pandoras-tamer/charts/llmwhisperer \
  --version <version> \
  -f /path/to/onprem.values.yaml \
  -f /path/to/onprem-profile.values.yaml \
  -f /path/to/multiaz.values.yaml \
  -n $NAMESPACE

The multiaz.values.yaml file is bundled inside the Helm chart — see Download Configuration Files for extraction instructions.

info

For single-AZ deployments, simply omit the multiaz.values.yaml file. No additional configuration is needed.

Validation​

After deploying with the multi-AZ overlay:

  1. Verify RabbitMQ cluster has 3 replicas and quorum queues:

    kubectl exec -n $NAMESPACE <rabbitmq-pod> -- rabbitmqctl list_queues name type

  2. Verify Redis pods are spread across zones:

    kubectl get pods -n $NAMESPACE -l app=redis -o wide

d. Useful Commands​

Kubernetes:

kubectl get pod -n <namespace>
kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace>

Helm:

helm list -n <namespace>

helm show values oci://us-central1-docker.pkg.dev/pandoras-tamer/charts/llmwhisperer --version <version>

helm rollback whisperer <revision-number> -n <namespace>

helm uninstall whisperer -n <namespace>

e. Listing Required Container Images (Private Registry / Air-Gap)​

To mirror container images into your own private registry (for example in an air-gapped environment), use the list-onprem-images.sh helper. It renders the Helm chart with your values files and prints every image reference the deployment requires. Download it and run it with the llmwhisperer target:

curl -fsSLO https://docs.unstract.com/unstract/files/list-onprem-images.sh
chmod +x list-onprem-images.sh

./list-onprem-images.sh llmwhisperer \
  -c oci://us-central1-docker.pkg.dev/pandoras-tamer/charts/llmwhisperer \
  -v <version> \
  -f onprem.values.yaml -f onprem-profile.values.yaml \
  -o images.txt
Pass every override you deploy with

The image list is exactly what your values cause the chart to render. Mirror your actual helm install/helm upgrade command: pass every -f values file you deploy with, in the same order — onprem.values.yaml, onprem-profile.values.yaml, and any overlay such as multiaz.values.yaml. Any image gated behind an override you omit here will be missing from images.txt, never mirrored, and fail to pull at deploy time.

The script accepts -f values files, not --set. If your deploy uses --set flags, fold those same values into a values file and pass it with -f too.

You do not need to match global.image.registry — the script always lists against the source registry (where to pull from), regardless of the destination registry in your values.

Run ./list-onprem-images.sh --help for all options.

Once you have the list, mirror each image into your registry and redirect the chart to it by setting global.image.registry (the full path up to, but not including, the image name) in your onprem.values.yaml:

global:
  image:
    registry: <full path>   # e.g. harbor.corp.example.com/llmwhisperer
tip

For the complete walkthrough — authenticating to both registries, the pull/tag/push mirror loop, image-pull secrets, and overriding the RabbitMQ Cluster Operator image (installed from the manifest in Step 2, not via Helm) — see the Mirroring Container Images guide.

f. Observability & Tracing (OpenTelemetry)​

Available from v2.62.0 onwards. LLMWhisperer ships with built-in OpenTelemetry instrumentation for distributed tracing. Traces cover HTTP requests, Celery tasks, database queries, Redis operations, and outbound HTTP calls, across the backend and all workers.

Tracing is disabled by default — you turn it on with a flag in your values file. No application code changes are needed, so a standard upgrade requires no action.

note

This is the LLMWhisperer equivalent of the Unstract platform observability guide. The concepts are the same, but the values keys differ — LLMWhisperer uses global.backend.config with ENABLE_TRACING, not the platform's otel.enabled block. Use the keys on this page for LLMWhisperer.

Prerequisites​

An OTLP-compatible endpoint reachable from the LLMWhisperer namespace. Typically an OpenTelemetry Collector running in-cluster, or a SaaS backend that accepts OTLP directly (Honeycomb, Datadog, Grafana Tempo, Jaeger). Note its endpoint as host:port — by convention gRPC listens on 4317 and HTTP on 4318.

1. Enable tracing​

The tracing settings live under global.backend.config in sample.onprem.values.yaml. Despite the backend key, this config is shared by the backend and every worker — all workers are deployed from the same underlying chart.

Set the following in your onprem.values.yaml:

global:
  backend:
    config:
      ENABLE_TRACING: "true"
      OTEL_TRACES_EXPORTER: "otlp"
      OTEL_EXPORTER_OTLP_ENDPOINT: "otel-collector.observability.svc.cluster.local:4317"
      OTEL_EXPORTER_OTLP_PROTOCOL: "grpc"
      OTEL_EXPORTER_OTLP_INSECURE: "true"
      OTEL_PYTHON_EXCLUDED_URLS: "/health"

All three of the first keys are required to actually export traces: ENABLE_TRACING is the master switch, OTEL_TRACES_EXPORTER defaults to "none" (traces are dropped unless set to "otlp"), and OTEL_EXPORTER_OTLP_ENDPOINT has no default. Every value is a string — quote them, including "true" and "false".

Exporter options:

  • In-cluster Collector (recommended). Point the endpoint at your Collector service and let it hold the credentials for your upstream backend. Plaintext (OTEL_EXPORTER_OTLP_INSECURE: "true") is acceptable for this hop.

  • Direct SaaS export. Export straight to a vendor by switching protocol and adding an auth header — for example, Honeycomb:

    OTEL_EXPORTER_OTLP_ENDPOINT: "https://api.honeycomb.io"
    OTEL_EXPORTER_OTLP_PROTOCOL: "http/protobuf"
    OTEL_EXPORTER_OTLP_HEADERS: "x-honeycomb-team=YOUR_API_KEY"
Avoid API keys in config

Everything under global.backend.config is rendered into a ConfigMap in plaintext, so an API key placed in OTEL_EXPORTER_OTLP_HEADERS is stored unencrypted and visible to anyone with read access to the namespace. Prefer the in-cluster Collector option and keep vendor credentials on the Collector.

2. Apply the change​

Upgrade as described in 5. Upgrading, passing the same values files you normally deploy with. Tracing is activated by wrapping the workload startup command with opentelemetry-instrument, so the pods are restarted by this upgrade.

Custom command/args overrides disable tracing

The instrumentation wrapper is only applied to workloads that use the chart's default uv-based startup command. If you have overridden command or args for the backend or any worker in your own values, that workload will start without tracing, even with ENABLE_TRACING: "true". Re-sync those overrides from the current sample file.

3. Verify​

Confirm the instrumentation wrapper is applied. The workload should start under opentelemetry-instrument:

kubectl get deploy whisperer-backend -n <namespace> \
  -o jsonpath='{.spec.template.spec.containers[0].args}'

If opentelemetry-instrument is absent from the output, tracing is not active for that workload — the usual cause is a custom command/args override (see the warning above). Repeat for any worker deployment you want to check.

Confirm traces are being produced. Backend and worker log lines carry a [TraceID: ...] field. It reads - when no span is active, and shows a real trace ID once tracing is working:

kubectl logs deploy/whisperer-backend -n <namespace> | grep 'TraceID'

Confirm spans arrive. Run a document through extraction and check your Collector or tracing backend. Spans should appear from the backend and from each worker involved in the extraction, and the trace ID in the logs above lets you correlate a specific request with its trace.

Service names in traces​

Each workload reports itself under OTEL_SERVICE_NAME (llmw_backend, llmw_worker_ingestion, llmw_worker_page_extraction, and so on), which is how you tell them apart in your tracing backend.

Leave OTEL_SERVICE_NAME unset to keep workers distinguishable

sample.onprem.values.yaml ships OTEL_SERVICE_NAME: "llmw_backend" inside the global.backend.config block. Because the global config takes precedence over each workload's own default, keeping that line makes every worker report as llmw_backend, collapsing them into a single service in your traces.

Comment out or remove OTEL_SERVICE_NAME from global.backend.config and each workload falls back to its own correct name. Only set it globally if you deliberately want all workloads reported as one service.

Tuning​

Sampling. By default every request is traced; the per-span overhead is negligible next to the per-document OCR/LLM cost. At high request volumes, head-sample to reduce overhead:

global:
  backend:
    config:
      OTEL_TRACES_SAMPLER: "parentbased_traceidratio"
      OTEL_TRACES_SAMPLER_ARG: "0.1"   # keep ~10% of traces

Transport security. OTEL_EXPORTER_OTLP_INSECURE: "true" sends traces in plaintext. That is fine for an in-cluster or localhost Collector. If your collector is reached cross-node or off-cluster, set it to "false" and use a TLS endpoint — otherwise trace data crosses the network unencrypted.

Excluded URLs. OTEL_PYTHON_EXCLUDED_URLS takes a comma-separated list of URL substrings to skip, which keeps health probes out of your traces. Defaults to "/health".

Metrics and logs. Only tracing is supported. OTEL_METRICS_EXPORTER and OTEL_LOGS_EXPORTER are kept at "none".

Full option reference​

KeyDefaultDescription
ENABLE_TRACING"false"Master switch for OpenTelemetry tracing
OTEL_TRACES_EXPORTER"none""otlp" to export traces, "none" to drop them
OTEL_EXPORTER_OTLP_ENDPOINT""Required when tracing is enabled. Collector endpoint
OTEL_EXPORTER_OTLP_PROTOCOL"grpc"Transport protocol: "grpc" or "http/protobuf"
OTEL_EXPORTER_OTLP_INSECURE"true""true" for a plaintext (non-TLS) connection
OTEL_EXPORTER_OTLP_HEADERSunsetHeaders for direct SaaS export, e.g. an auth key
OTEL_SERVICE_NAMEper workloadService name in traces — see the warning above
OTEL_PYTHON_EXCLUDED_URLS"/health"Comma-separated URL substrings to exclude
OTEL_TRACES_SAMPLERunsetHead sampling strategy, e.g. "parentbased_traceidratio"
OTEL_TRACES_SAMPLER_ARGunsetSampler argument, e.g. "0.1" to keep ~10% of traces

Disabling tracing​

Set ENABLE_TRACING: "false" (or remove the block) and upgrade. No other cleanup is required.