Using a Managed Redis and RabbitMQ

LLMWhisperer deploys its own Redis and RabbitMQ inside your cluster by default. You can replace either one — or both — with a managed service such as Google Memorystore, Amazon ElastiCache, Amazon MQ, Azure Cache for Redis or CloudAMQP.

Nothing else about the deployment changes. This is a values change plus one helm upgrade, and it can be reverted the same way.

info

Available from LLMWhisperer v2.65.0. See Version History.

Before you start

You need, reachable from the cluster's network:

• Redis — host, port and password. TLS is optional but recommended.
• RabbitMQ — an amqps:// endpoint, username and password.

You can move just one of them. The two halves below are independent.

Requirements

Redis mode	Non-cluster only. LLMWhisperer connects with a standard Redis client, so a cluster-mode endpoint returns MOVED on most keys
Redis version	The bundled server is Redis 8.x. Validated against Google Memorystore
Redis certificate	Must be signed by a publicly trusted CA to be verified — see Certificate verification
RabbitMQ version	The bundled broker is 3.13. A 4.x broker, including Amazon MQ, needs one extra setting
Egress from the cluster	Your Redis port (6378/6379/6380 depending on provider), 5671 for AMQPS, and 443 if you set the broker management URL

Two things that are easy to overlook when sizing the managed instances:

• On-prem stores page data in the worker Redis (WORKER_DATA_STORAGE_PROVIDER: redis), not just cache. Size the instance for extracted page content, not for key counts.
• If rabbitmq.ha.enabled is true, the application still declares quorum queues — that flag is independent of rabbitmq.enabled, so it keeps applying to a managed broker. The broker must support quorum queues, and queues that already exist there as classic will conflict.

Drain the queues first on a live installation

This switches clients to the new endpoints and removes the bundled servers in the same upgrade. Anything still queued on the old RabbitMQ is never picked up, and Celery results still held in the old Redis are not carried over.

The RabbitMQ data does go with the broker. The Redis data does not: its volumes outlive the server and are reattached if you roll back — see Rolling back.

Stop submitting new documents and let in-flight extractions finish before you upgrade. Nothing migrates the old queues or results for you.

Quick setup

1. Add the endpoints to your values file

Add these under the existing global.sharedConfigs section of your onprem.values.yaml, and set the two toggles at the bottom of the file:

global:
  sharedConfigs:
    redis:
      REDIS_HOST: "cache.example.com"
      REDIS_PORT: "6379"
      REDIS_PASSWORD: "<redis-password>"
      REDIS_SENTINEL_MODE: "False"
      REDIS_SSL: "true"

    celeryBroker:
      X_CELERY_BROKER_BASE_URL: "amqps://broker.example.com:5671"
      # Optional — only feeds the queue-depth metrics
      X_CELERY_BROKER_MANAGEMENT_URL: "https://broker.example.com"
      X_CELERY_BROKER_USERNAME: "<broker-username>"
      X_CELERY_BROKER_PASSWORD: "<broker-password>"
      X_CELERY_BACKEND_URL: "rediss://cache.example.com:6379/2?ssl_cert_reqs=required"

# Stop deploying the in-cluster servers
redis:
  enabled: false

rabbitmq:
  enabled: false

Two of these are easy to miss, so the chart refuses to install without them rather than starting and failing at runtime:

Setting	Why
REDIS_SENTINEL_MODE: "False"	Defaults to "True". Sentinel mode uses a different code path that ignores your managed endpoint entirely.
X_CELERY_BACKEND_URL	A separate URL for Celery results. It does not follow redis above, so it keeps pointing at the in-cluster server unless you set it.

Set REDIS_PORT yourself — nothing checks it

It defaults to 26379, the Sentinel port, and a managed endpoint is usually 6379 or 6378. There is no install-time guard for this one: omit it and the upgrade succeeds, then every Redis call fails at runtime against a port the endpoint is not listening on.

REDIS_HOST takes a hostname or an IP address and must not include the port — it is read as-is, so cache.example.com:6379 becomes a hostname containing a colon and fails to resolve. The port goes in REDIS_PORT. Write an IPv6 address without brackets (2001:db8::1, not [2001:db8::1]); brackets belong in a URL, not in this field. To give one combined string instead, see Using a URL instead of discrete keys.

The port above is an example. Use whatever TLS port your provider gives you — 6379 on ElastiCache, 6378 on Memorystore, 6380 on Azure Cache — and use the same port in X_CELERY_BACKEND_URL. They are configured separately, so a mismatch leaves Redis working while Celery results silently go nowhere.

Leave the password out of X_CELERY_BACKEND_URL — it is filled in from REDIS_PASSWORD. That only happens when the URL's host and port match REDIS_HOST and REDIS_PORT exactly, so that the primary's credential is never sent to a server it was not issued for. Spelling one as a DNS name and the other as an IP counts as a different server: no password is sent and Redis answers NOAUTH. Write both the same way.

A rediss:// backend URL must carry ?ssl_cert_reqs=, or Celery rejects it.

2. Apply the change

Run your normal upgrade — see Upgrading. Nothing about this change exempts you from the usual steps there, in particular reconciling your values file against the target release's sample.

3. Verify

The usual Deployment Validation steps apply unchanged. The only check specific to this change is that the bundled servers are gone:

kubectl get pods -n $NAMESPACE

No redis-node-* pods if you disabled Redis, and no rabbitmq-server-* pods if you disabled RabbitMQ. Everything else restarts and returns to Running. Then send a document through the API — if it returns a result, both services are working.

That is the whole procedure for an endpoint with a publicly trusted certificate. Google Memorystore signs its certificate with a per-instance private CA, which cannot be verified — see Certificate verification before you enable TLS against it. The rest of this page is only needed if one of the sections below applies to you.

Rolling back

Remove the block you added, set redis.enabled and rabbitmq.enabled back to true, and upgrade again the same way. No manual cleanup is needed.

Rolling back takes a few minutes. Until the Redis replicas have finished syncing the API returns 500; this is expected — wait for all three redis-node-* pods to be Running before deciding anything is wrong.

RabbitMQ comes back empty; Redis may not

The redis-node StatefulSet keeps its data on PersistentVolumeClaims, and disabling the subchart removes the pods, not the claims — and the bundled server runs with AOF persistence. Rolling back reattaches those volumes, so Redis can come up holding data from before you switched away, and the claims keep costing storage in the meantime.

To come back clean, list the leftover claims while Redis is still disabled and nothing is using them, then delete them before rolling back:

kubectl get pvc -n $NAMESPACE \
  -l app.kubernetes.io/name=redis,app.kubernetes.io/instance=<your-helm-release-name>

Match on the instance label, not the name: anything else running a Redis in that namespace labels its claims name=redis too. Confirm each claim belongs to this release before deleting it.

Redis reference

Certificate verification

REDIS_SSL: "true" alone verifies the endpoint's certificate against the system trust store. That is all a publicly-trusted endpoint needs — Amazon ElastiCache and Azure Cache for Redis both qualify.

None of the keys in this section is required. Leave them out and the certificate is verified: REDIS_SSL_CERT_REQS defaults to required. Add one only if your endpoint falls into a case below.

A privately-signed certificate cannot be verified in this version

Google Memorystore signs its certificate with a per-instance private CA, and there is no supported way to get that CA into the pods that connect to Redis. REDIS_SSL_CA_CERTS and the ssl_ca_certs= URL parameter both name a path inside the pod, and the per-service volumes lists that would place it there are replaced rather than merged by Helm — several deployments already use them for the worker heartbeat probe, so overriding one breaks its liveness check.

The only option against such an endpoint is to disable verification:

      REDIS_SSL_CERT_REQS: "none"

and to match it on the Celery result backend, which is a separate connection:

      X_CELERY_BACKEND_URL: "rediss://cache.example.com:6379/2?ssl_cert_reqs=none"

Traffic is still encrypted, but the endpoint is no longer authenticated — the connection is not protected against an impersonated server. Prefer a provider whose certificate is publicly trusted, such as Amazon ElastiCache or Azure Cache for Redis, where TLS needs nothing beyond REDIS_SSL: "true".

An endpoint addressed by IP, or by a name the certificate does not carry.

Prefer connecting by a name the certificate actually covers — most providers publish a DNS name alongside the IP, and using it keeps verification intact with no extra settings. Only if no such name exists:

      REDIS_SSL_CHECK_HOSTNAME: "false"

This still verifies that the certificate chains to a trusted CA, but stops checking that it was issued for the host you asked for — so a trusted certificate belonging to a different host would be accepted if the connection were redirected.

Hostname checking is on by default, and is forced off when REDIS_SSL_CERT_REQS is "none".

Databases

LLMWhisperer uses three Redis databases: 0 for cache, 1 for the worker cache, and 2 for Celery results.

If your endpoint exposes only database 0 — Azure Managed Redis, Redis Enterprise and Redis Cloud, and any cluster-mode tier — point all three at 0:

      REDIS_DB: "0"
    workerRedis:
      WORKER_REDIS_DB: 0

and end X_CELERY_BACKEND_URL with /0. The keys do not collide.

Using a URL instead of discrete keys

REDIS_URL works in place of the host/port/password keys:

      REDIS_URL: "rediss://:<password>@cache.example.com:6379/0"

The password may be in the URL or supplied separately as REDIS_PASSWORD; a separate value fills a gap the URL leaves and never overrides one. REDIS_DB still applies and overrides the database in the URL.

warning

With existingSecret, put REDIS_URL in your Secret, not in the values file. A URL set in the values file is not delivered to the pods on that path, and they get no endpoint at all.

The worker cache

The worker cache inherits the primary's host, port and password automatically and only carries its own database. You do not need to repeat the endpoint.

Set WORKER_REDIS_* keys only if the worker cache lives on a different server. The password is inherited only while the host and port match — a different port is treated as a different server, and the password is not carried across.

Keeping the password out of the values file

existingSecret works here as it does for every other configuration section — see Using Kubernetes Secrets.

One point is specific to Redis, and it is a split rather than a single rule. The pods read these settings from the Secret, but the install-time checks read them from your values file — the chart cannot see inside a Secret you created. A few keys therefore have to be in both places.

Keep these in the values file, or the upgrade is rejected:

global:
  sharedConfigs:
    redis:
      REDIS_SENTINEL_MODE: "False"
      existingSecret: "my-redis-secret"

    celeryBroker:
      X_CELERY_BACKEND_URL: "rediss://cache.example.com:6379/2?ssl_cert_reqs=required"

Putting REDIS_SENTINEL_MODE only in the Secret does not work: the values-file default is "True", so the chart still sees Sentinel mode and refuses to install. X_CELERY_BACKEND_URL behaves the same way — its default names the in-cluster Redis.

Put these in the Secret. Setting existingSecret on a section stops the chart building that section's Secret, and the pods receive nothing else for it:

Section	Its Secret must carry
redis	REDIS_HOST, REDIS_PORT, REDIS_DB, REDIS_PASSWORD, REDIS_USER, REDIS_SENTINEL_MODE, and any REDIS_SSL* keys
celeryBroker — only if you set its existingSecret too	X_CELERY_BACKEND_URL, alongside the broker keys it already holds

danger

REDIS_SENTINEL_MODE and X_CELERY_BACKEND_URL each belong in both places. In the values file they get you past the install checks; in the Secret they are what the pods actually read. Set either one in the values file alone and the upgrade succeeds — and then the pods come up in Sentinel mode, or keep writing Celery results to the Redis you have just removed, or to nowhere. Neither is reported.

RabbitMQ reference

TLS

The amqps:// scheme is enough on its own. It enables TLS and verifies the broker's certificate against the system trust store — no extra settings. Amazon MQ and CloudAMQP both work this way.

danger

Never add ?ssl_cert_reqs= to X_CELERY_BROKER_BASE_URL. Unlike the Redis backend URL, the broker URL does not accept it, and the workers fail at startup with TypeError: _wrap_socket_sni() got an unexpected keyword argument 'ssl_cert_reqs'.

warning

A broker with a privately-signed certificate is not supported in this version, for the same reason as Redis: X_CELERY_BROKER_SSL_CA_CERTS names a path inside the pod, and the CA cannot be mounted there. Use a broker whose certificate is publicly trusted — Amazon MQ and CloudAMQP both are.

note

Broker TLS verifies the certificate chain but not the hostname, in every mode. A certificate issued to a different host by a trusted CA is accepted.

RabbitMQ 4.x, including Amazon MQ

Celery declares the pidbox queues for its remote-control protocol as transient and non-exclusive, and RabbitMQ 4.x refuses that by default — so the workers die at startup with:

(541) INTERNAL_ERROR - Feature transient_nonexcl_queues is deprecated

Amazon MQ now defaults to RabbitMQ 4.x, so this applies there. Turn remote control off:

      CELERY_WORKER_ENABLE_REMOTE_CONTROL: "false"

This disables celery inspect / celery control diagnostics. Task execution, retries, scheduling and monitoring are unaffected.

note

Amazon MQ does not allow the broker-side configuration that would otherwise permit these queues, so this setting is the only option there.

Virtual hosts

To use a vhost other than the default, put it on the end of the URL:

      X_CELERY_BROKER_BASE_URL: "amqps://broker.example.com:5671/myvhost"

Troubleshooting

Symptom	Cause
Install fails with a message about REDIS_SENTINEL_MODE	Left at its default "True". Set it to "False" in the values file — putting it only in an existingSecret does not satisfy the check.
Install fails saying X_CELERY_BACKEND_URL still points at the in-cluster Redis	It is a separate URL. Point it at the managed endpoint.
Install fails saying no external Redis is configured	REDIS_HOST has been explicitly blanked. It defaults to redis, so simply forgetting it passes the guard and leaves the pods pointing at the server you just removed.
Pods start but every Redis call times out	TLS against a non-TLS port, or the reverse. Check REDIS_SSL matches the port.
SSL: CERTIFICATE_VERIFY_FAILED on Redis	Privately-signed certificate. See Certificate verification.
Extraction works but results never arrive	X_CELERY_BACKEND_URL disagrees with the redis settings — usually a different port, or a private CA that the URL does not name.
An IPv6 Redis host does not resolve	Brackets in REDIS_HOST. Write 2001:db8::1, not [2001:db8::1].
Workers crash-loop with _wrap_socket_sni() got an unexpected keyword argument	?ssl_cert_reqs= in X_CELERY_BROKER_BASE_URL. Remove it.
Workers crash-loop on a RabbitMQ 4.x broker	Set CELERY_WORKER_ENABLE_REMOTE_CONTROL: "false".