Skip to main content

On-Prem Observability & Tracing (OpenTelemetry)

Unstract's services ship with OpenTelemetry (OTEL) instrumentation built in. Distributed tracing is disabled by default; you turn it on with a single flag in your on-prem values — no application code changes, no custom image, and no per-service configuration. Once enabled, traces for HTTP requests, Celery tasks, database queries, Redis operations, and outbound HTTP calls are exported to any OTLP-compatible collector or backend (an in-cluster OpenTelemetry Collector, Jaeger, Grafana Tempo, Honeycomb, etc.).

Prerequisite

An OTLP endpoint reachable from the cluster must already exist. The recommended pattern is to run an OpenTelemetry Collector inside the cluster and let it forward to your chosen backend; alternatively, export directly to a SaaS backend such as Honeycomb. See the exporter options.

How it works​

Instrumentation is no-code / auto-instrumentation: when tracing is enabled the chart launches every traced service under the opentelemetry-instrument wrapper (Celery workers) or with --with-tracing (the backend/platform/runner/x2text web services), which patches the common libraries (Flask/gunicorn, Celery, psycopg2, redis, requests) at import time. You don't wire this per service — one flag, otel.enabled, covers the whole platform, and each service reports its own service.name automatically so traces are attributable per component.

1. Enable tracing​

In your on-prem.values.yaml, set otel.enabled: true and point the exporter at your collector. The otel block is already present (commented) in sample.on-prem.values.yaml — edit it there:

otel:
  # Master switch — turns tracing on for every service.
  enabled: true
  configMap:
    # Your OTLP collector endpoint (host:port or URL).
    OTEL_EXPORTER_OTLP_ENDPOINT: "otel-collector.observability.svc.cluster.local:4317"
    OTEL_EXPORTER_OTLP_PROTOCOL: "grpc"       # "grpc" or "http/protobuf"
    OTEL_EXPORTER_OTLP_INSECURE: "true"       # in-cluster plaintext only; omit for TLS
    OTEL_PYTHON_EXCLUDED_URLS: ".*/health,.*/ping"

That's the entire configuration — OTEL_TRACES_EXPORTER is set to otlp automatically when otel.enabled: true (metrics and logs exporters stay off), and every worker/web service is wrapped by the chart. There is no per-service command/args to maintain.

Exporter options​

Only the configMap block changes depending on where you send traces:

Option A — in-cluster OpenTelemetry Collector (recommended). The app exports to a Collector in the cluster (plaintext, in-cluster traffic), and the Collector holds backend credentials and forwards upstream — keeping secrets out of app config. Use the block shown above.

Option B — export directly to Honeycomb. Requires an API key sent as an OTLP header and TLS (do not set OTEL_EXPORTER_OTLP_INSECURE):

otel:
  enabled: true
  configMap:
    OTEL_EXPORTER_OTLP_ENDPOINT: "https://api.honeycomb.io"   # EU: https://api.eu1.honeycomb.io
    OTEL_EXPORTER_OTLP_PROTOCOL: "http/protobuf"
    OTEL_EXPORTER_OTLP_HEADERS: "x-honeycomb-team=YOUR_HONEYCOMB_API_KEY"
    OTEL_PYTHON_EXCLUDED_URLS: ".*/health,.*/ping"
API keys are sensitive

OTEL_EXPORTER_OTLP_HEADERS carries your Honeycomb API key. The otel.configMap values render into a Kubernetes ConfigMap (not a Secret). If your security policy prohibits credentials in ConfigMaps, prefer Option A and keep the key on the in-cluster Collector instead.

2. Apply​

Run your normal Helm upgrade — the otel block lives in on-prem.values.yaml, so no extra -f is needed:

helm upgrade unstract-platform oci://us-central1-docker.pkg.dev/pandoras-tamer/charts/unstract-platform \
  --version <version> \
  -f /path/to/on-prem.values.yaml \
  -f /path/to/on-prem.secret.yaml \
  -n unstract

See the Deployment Guide for the full upgrade workflow.

3. Verify​

  1. Check a web service picked up tracing — its startup logs should read:

    kubectl logs deploy/unstract-backend -n unstract | grep -iE 'opentelemetry|instrumentation'
    # Running Gunicorn with OpenTelemetry instrumentation

  2. Confirm spans arrive — trigger a document execution, then look for the traces in your collector / backend (e.g. the Honeycomb dataset named after the service). A single execution produces one distributed trace spanning the API request and the Celery worker tasks.

  3. Log correlation — with tracing on, application logs carry the active trace_id, so logs can be tied back to their trace.

4. Tuning and notes​

  • Sampling. With no sampler set, every trace is captured (100%). For high-throughput deployments you can bound the overhead by head-sampling — add to the configMap block:

    otel:
      configMap:
        OTEL_TRACES_SAMPLER: "parentbased_traceidratio"
        OTEL_TRACES_SAMPLER_ARG: "0.1"   # keep ~10% of traces

  • Memory. The x2text service is given extra memory headroom automatically when tracing is enabled — no manual resource override needed.

  • Metrics and logs exporters stay none; this enables traces only.

  • Excluded URLs. OTEL_PYTHON_EXCLUDED_URLS keeps health/readiness probes out of traces. Adjust the pattern to your ingress/probe paths.

  • Transport security. Use OTEL_EXPORTER_OTLP_INSECURE: "true" only for plaintext in-cluster traffic. For any endpoint reached over the network (e.g. Honeycomb), use TLS — omit OTEL_EXPORTER_OTLP_INSECURE.

  • Disabling tracing. Set otel.enabled: false and upgrade again.