Single-AZ to Multi-AZ Migration

This guide covers moving an existing LLMWhisperer on-prem deployment from a single Availability Zone to the multi-AZ (HA) topology described in Multi-AZ Deployment.

The whole cutover — draining the RabbitMQ queues, recreating them as quorum queues, and cycling the workers — is packaged inside the Helm chart as pre-/post-upgrade hook Jobs. You arm it with a single value (rabbitmq.ha.migrateQueues=true) and run one helm upgrade; the chart does the rest. Nothing to download, and nothing to run by hand — which also makes it work under GitOps (Argo CD, Flux).

It is a planned-maintenance operation: LLMWhisperer is offline for the duration of the migration.

Deploying a new cluster?

If you are installing LLMWhisperer for the first time, you do not need this page. Simply include multiaz.values.yaml in your initial helm install — see Enabling multi-AZ. This guide is only for converting a deployment that is already running single-AZ. On a fresh install the queues are declared as quorum queues from the start, so there is nothing to migrate.

What changes

Component	Before (single-AZ)	After (multi-AZ)
RabbitMQ	1 broker node, classic queues	3 broker nodes, quorum queues (Raft replication), pause_minority partition handling
Redis	3 replicas, no zone constraint	3 replicas with topologySpreadConstraints across zones (maxSkew: 1)
PriorityClass	none	llmwhisperer-critical, so the scheduler can preempt lower-priority pods rather than leave HA components Pending

Why it cannot be a plain helm upgrade

A RabbitMQ classic queue cannot be converted into a quorum queue in place. The queue has to be deleted and re-declared with x-queue-type: quorum, and a queue can only be deleted safely when it is empty. So the migration must:

1. Stop accepting new work (scale the backend to zero).
2. Let the workers finish everything still in flight (drain the queues).
3. Stop the workers, delete the now-empty classic queues, bring the 3-node HA broker up, and start the workers again — at which point they re-declare their queues as quorum queues.

Two details are easy to get wrong by hand, and the hooks handle both for you:

• Workers must be restarted, not just upgraded. The HA switch reaches a worker through the RABBITMQ_HA_ENABLED environment variable in its pod. A helm upgrade that flips the value does not by itself roll the worker pods, so a worker left running would keep declaring classic queues. The migration scales the workers to zero and back so they pick up the new value.
• Queues must be deleted while nothing is connected. If a worker is running when a queue is deleted, it immediately re-declares it — as classic, if it has not picked up the flag yet.

How the automated migration works

When you run helm upgrade with rabbitmq.ha.migrateQueues=true, the chart schedules the cutover as two Helm hook Jobs that bracket the upgrade:

helm upgrade  ─┬─▶  pre-upgrade Job    stop-backend → drain → stop-workers
               │
               ├─▶  (Helm applies the multi-AZ overlay:
               │     broker 1→3, RABBITMQ_HA_ENABLED=true, Redis topology)
               │
               └─▶  post-upgrade Job   apply → wait-broker → delete-queues
                                       → start-workers → start-backend → verify

Because the pre-upgrade Job runs before Helm changes anything, its preflight checks can abort the entire upgrade with nothing scaled down if a precondition fails. The post-upgrade Job runs after the HA overlay is applied and finishes the cutover.

State that has to survive between the two Jobs — chiefly the original backend and worker replica counts — is stored in a ConfigMap named <release>-multiaz-migration-state. It is created by the migration itself (not a Helm-managed resource), so it persists across a re-run and preserves the original counts even if the first attempt fails partway through.

The Jobs run a first-party multiaz-migrate image with kubectl baked in (no runtime downloads — air-gap safe), and the chart creates the migration's RBAC automatically — a ServiceAccount, a namespaced Role (pods/exec, plus pvc/pod delete when the Redis volume reset is on), and its RoleBinding.

Namespace-scoped RBAC — the one cluster-scoped object is a PriorityClass

The migration itself needs no cluster-scoped permissions; its RBAC is namespace-only. The only cluster-scoped object in the upgrade is the llmwhisperer-critical PriorityClass that the multiaz.values.yaml overlay creates (priorityClass.create: true) — so the account running helm upgrade (or your GitOps controller) needs permission to create a PriorityClass, not to create cluster-scoped RBAC. On a locked-down cluster, that one capability is what to confirm before the window.

A consequence of the namespace-only scope: the migration's preflight zone checks read nodes / PVs / StorageClasses, which are cluster-scoped and deliberately not granted, so those checks degrade to warnings. Treat the migration's own zone verdict as advisory and confirm placement with the -o wide checks under Verify.

Prerequisites

• A chart version v2.64.4 or newer — the on-prem release that ships the migration hooks (the rabbitmq.ha.migrateQueues flag and the multiazMigration block). On an older chart, helm show values … | grep migrateQueues returns nothing; upgrade the chart first.
• The RabbitMQ Cluster Operator must already be installed in the cluster.
• helm 3.x and kubectl configured against the target cluster.
• For RabbitMQ and Redis to gain real zone redundancy, each must be able to schedule into at least two availability zones, with capacity in each. What decides this is the nodeSelector on those pods, not the cluster's overall zone coverage: a component pinned to a single-zone node pool still schedules (one topology domain satisfies the spread constraint trivially), it simply stays vulnerable to that one zone failing. The Verify step's -o wide checks show which case you are in.
• A maintenance window of about 30 minutes. The fixed work — the Helm upgrade, forming the 3-node broker cluster, deleting and recreating the queues, and cycling the workers — takes roughly 10–15 minutes; the default Redis volume reset adds to that, replacing the three Redis PVCs one at a time. The variable part is draining whatever is in flight when you start, which is why the drain step allows 5 minutes by default (multiazMigration.drainTimeout). Migrate during a quiet period so the drain stays short — or raise drainTimeout and the Job deadlines (preActiveDeadlineSeconds / postActiveDeadlineSeconds) if you expect a large backlog or a slow Redis reset, keeping --timeout above the highest deadline you set.

Zonal volumes cannot move between zones

Redis uses persistent volumes, and on most cloud providers a volume is pinned to the zone it was created in. If Redis can be scheduled into several zones but all its existing volumes sit in one, adding the zone-spread constraint leaves the recreated Redis pods Pending indefinitely — the scheduler cannot satisfy the constraint, and the volume cannot follow.

The multi-AZ overlay handles this for you: multiazMigration.allowRedisVolumeReset is on by default, so the migration re-provisions the Redis PVCs across zones inside the window, one node at a time (see Redis volume reset). To leave the volumes in place instead, turn it off.

If Redis carries a nodeSelector restricting it to a single-zone node pool, there is only one topology domain, the constraint is satisfied trivially, and no action is needed — Redis simply gains no zone redundancy from the migration.

Step 1 — Arm the migration

The migration is gated behind two values, both of which must be set on the same upgrade:

Value	Set by	Purpose
rabbitmq.ha.enabled=true	multiaz.values.yaml overlay	Switches on the 3-node HA broker and Redis zone spread.
rabbitmq.ha.migrateQueues=true	you, for this one upgrade	Runs the offline classic→quorum cutover Jobs.

The multiaz.values.yaml overlay sets rabbitmq.ha.enabled=true but leaves migrateQueues=false on purpose, so that adding the overlay never takes a deployment offline by itself. You opt into the cutover explicitly by adding --set rabbitmq.ha.migrateQueues=true.

The overlay is bundled inside the Helm chart — see Download Configuration Files for how to extract it.

The chart refuses a misconfiguration

Setting migrateQueues=true without ha.enabled=true fails the Helm render with a clear message rather than quietly skipping the migration, so a mistyped values file is caught before the upgrade runs.

Air-gapped installs: mirror the migration image

The hook Jobs run the multiaz-migrate image, which only appears in the render when the migration is armed. Surface it so you can mirror it into your registry alongside the other on-prem images:

helm template whisperer oci://us-central1-docker.pkg.dev/pandoras-tamer/charts/llmwhisperer \
  --version <version> \
  -f /path/to/onprem.values.yaml \
  -f /path/to/multiaz.values.yaml \
  --set rabbitmq.ha.migrateQueues=true \
  | grep -i multiaz-migrate

The image follows global.image.registry like every other first-party image, so once mirrored it is picked up by your existing private-registry redirect — no extra configuration.

Step 2 — Run the migration

Inside the maintenance window, run your normal upgrade command with the overlay and the migrate flag added:

helm upgrade whisperer oci://us-central1-docker.pkg.dev/pandoras-tamer/charts/llmwhisperer \
  --version <version> \
  -n $NAMESPACE \
  -f /path/to/onprem.values.yaml \
  -f /path/to/multiaz.values.yaml \
  --set rabbitmq.ha.migrateQueues=true \
  --timeout 30m

Raise --timeout well above the default 5 minutes

helm upgrade --timeout bounds each Kubernetes wait separately — in Helm's own words, "time to wait for any individual Kubernetes operation (like Jobs for hooks)". It applies to the pre-upgrade Job's wait and the post-upgrade Job's wait independently, so size it against the longest single hook, not the sum. The binding one is the post-upgrade Job (multiazMigration.postActiveDeadlineSeconds, default 900s / 15m); the pre-upgrade Job's preActiveDeadlineSeconds is 600s / 10m.

At the default --timeout of 5 minutes, a hook still draining or waiting for the cluster to form aborts the upgrade mid-cutover. The examples here use --timeout 30m — comfortably above the post hook's ceiling. Note that --timeout does not extend the Jobs' own activeDeadlineSeconds: if a hook needs longer (for example a slow Redis volume reset), raise its deadline, and raise --timeout to stay above it — --timeout still bounds how long Helm waits. See If a hook fails.

Avoid --atomic for the migration upgrade

--atomic rolls the release back automatically if the upgrade does not complete in time — which, partway through an offline cutover, is exactly what you do not want. If the upgrade times out or a hook fails, recover by re-running the same command instead (see If a hook fails).

Mirror every -f you deploy with

Pass every -f values file you normally deploy with, in the same order, plus the overlay. Helm applies exactly the values you give it; an override you leave out here is an override that silently disappears from your deployment.

Watch it run

Both Jobs stream their progress to their logs. In another terminal:

# pre-upgrade phase (stop-backend, drain, stop-workers)
kubectl -n $NAMESPACE logs -f job/whisperer-multiaz-migrate-pre

# post-upgrade phase (broker formation, delete/recreate queues, restart, verify)
kubectl -n $NAMESPACE logs -f job/whisperer-multiaz-migrate-post

(Replace whisperer with your release name.) Finished Jobs are kept for a day (multiazMigration.ttlSecondsAfterFinished) so their logs stay available for audit before they are garbage-collected.

What the hooks do, step by step

Phase	Step	Action
pre	stop-backend	Scales the backend to 0 so no new extraction requests are accepted. In-flight documents keep processing.
pre	drain	Polls every application queue until both messages_ready and messages_unacknowledged reach 0. Unacknowledged matters as much as ready: a message being processed right now would be lost if its queue were deleted.
pre	stop-workers	Scales every worker to 0, so nothing can re-declare a queue during the cutover and every worker picks up the HA flag when it restarts.
—	Helm applies overlay	Broker goes 1→3, RABBITMQ_HA_ENABLED=true, Redis gets its zone-spread topology.
post	apply	Re-asserts scale-to-0 (the upgrade can reset a worker that has autoscaling disabled) and waits for RABBITMQ_HA_ENABLED=true to actually land in the pods.
post	wait-broker	Waits for all 3 RabbitMQ pods to be Ready and to report healthy cluster membership.
post	delete-queues	Deletes the drained classic queues. Every delete uses --if-empty, so a queue that somehow still holds a message is refused rather than silently destroyed.
post	start-workers	Restores the recorded worker replica counts. On startup the workers re-declare their queues as quorum queues. Fails loudly if the pods do not actually see RABBITMQ_HA_ENABLED=true.
post	start-backend	Restores the backend replica count and waits for it to pass its readiness probe.
post	verify	Confirms every queue is now quorum and the broker has 3 ready pods, then checks that RabbitMQ and Redis are genuinely spread across zones — warning if they all landed in one.

When allowRedisVolumeReset is on — the overlay default — the post phase also re-provisions the Redis volumes across zones, one node at a time, so Redis gains real zone redundancy. See Redis volume reset.

Redis volume reset (on by default)

A zonal Redis volume cannot follow its pod to another zone, so the multi-AZ overlay sets multiazMigration.allowRedisVolumeReset: true — the migration re-provisions the Redis PVCs across zones as part of the run. This is what lets Redis actually gain zone redundancy instead of returning to its original zone.

It is destructive but safe. Redis runs as a replicated set with Sentinel, so the dataset lives on every node: the PVCs are deleted and recreated one node at a time, each rebuilt node resyncs fully from the surviving members before the next is touched, a node is never wiped unless another confirmably holds a current copy, and the migration stops rather than touching a second node if the first does not come back. (The migration Job is granted pvc/pod delete permission for this.)

To skip it — for example Redis is pinned to a single-zone node pool (so it gains no redundancy anyway), or you would rather move the volumes yourself — set it to false:

helm upgrade whisperer oci://us-central1-docker.pkg.dev/pandoras-tamer/charts/llmwhisperer \
  --version <version> \
  -n $NAMESPACE \
  -f /path/to/onprem.values.yaml \
  -f /path/to/multiaz.values.yaml \
  --set rabbitmq.ha.migrateQueues=true \
  --set multiazMigration.allowRedisVolumeReset=false \
  --timeout 30m

With the reset off, the queue cutover still runs; only the Redis volumes are left where they are.

GitOps deployments (Argo CD, Flux)

Because the cutover is packaged as ordinary chart resources, it runs the same way under a GitOps controller — there is no script to invoke out-of-band. The pre-/post-upgrade hooks map to your controller's sync hooks (Argo CD runs them as PreSync/PostSync).

To migrate, add multiaz.values.yaml and rabbitmq.ha.migrateQueues: true to your application's Helm values and sync. Two things to get right:

• Do not arm migrateQueues on an app's first sync. Argo CD maps the pre-upgrade hook to a PreSync that runs on every sync, including a brand-new app's first one — where the backend Deployment does not exist yet, so the pre hook fails trying to scale it. Install once with migrateQueues unset, then set it true on a later sync to run the cutover.
• Raise the sync timeout — not progressDeadlineSeconds. Let the sync run at least as long as the post-upgrade Job's deadline, e.g. argocd app sync --timeout 1800. A Deployment's progressDeadlineSeconds does not govern hook waits, so raising it has no effect here.
• Unset migrateQueues afterwards (see below) and commit that change, so a later routine sync does not re-enter the migration path.

Verify

The post-upgrade Job ends with a verification pass, so a green helm upgrade already means the queues came back as quorum. To confirm independently:

# every application queue should read `quorum`
kubectl exec -n $NAMESPACE <rabbitmq-pod> -- rabbitmqctl list_queues name type

# RabbitMQ should have 3 ready pods, spread across zones
kubectl get pods -n $NAMESPACE -l app.kubernetes.io/name=rabbitmq -o wide

# Redis should be spread across zones too
kubectl get pods -n $NAMESPACE -l app.kubernetes.io/name=redis -o wide

Finally, run a document end-to-end through the playground or the API before you close the maintenance window.

A queue that comes back as classic

verify flags any queue that reappears as classic. checkbox_detection is a known case and is harmless for on-prem deployments, where its worker is disabled by default. For any other queue, contact Unstract support before resuming production traffic.

The migration's own zone verdict is advisory

The post hook tries to report whether RabbitMQ and Redis are spread across zones, but under its namespace-only RBAC it cannot read node zones — so it warns that everything is in one zone on every run, including a fully successful one. Do not act on that warning by itself. The reliable signal is the -o wide output above: if it shows the pods genuinely confined to one zone, the node pool is single-zone — widen it across zones and let the pods reschedule for real redundancy.

After the migration

Once the migration has succeeded, unset rabbitmq.ha.migrateQueues — remove the --set flag (or set it back to false in your values / GitOps repo). Leave multiaz.values.yaml in place: that is now your steady-state topology.

Leaving migrateQueues=true set is safe — on an already-migrated release the hooks detect the completed state and exit as a no-op before doing anything — but unsetting it keeps later upgrades hook-free and avoids coupling a routine upgrade to the migration's preconditions and longer timeout.

Then remove the hook resources Helm leaves behind. Helm never garbage-collects hook objects, so the migration's ServiceAccount (which holds pods/exec, plus pvc/pod delete if the Redis reset ran), Role, RoleBinding, and the script ConfigMap otherwise sit in the namespace indefinitely — the kind of residue an on-prem security review flags months later:

kubectl -n $NAMESPACE delete serviceaccount,role,rolebinding,configmap \
  -l app=llmwhisperer,component=multiaz-migration

If a hook fails

Capture the logs first. The next helm upgrade deletes the failed Job — and its pod logs — before creating the replacement (the before-hook-creation policy), so read them before you re-run; they are the first thing support will ask for:

kubectl -n $NAMESPACE logs job/whisperer-multiaz-migrate-pre
kubectl -n $NAMESPACE logs job/whisperer-multiaz-migrate-post

The migration is idempotent and resumable. Once you have fixed the underlying cause, re-run the same command. Helm recreates the hook Jobs, and the persisted <release>-multiaz-migration-state ConfigMap preserves the original replica counts, so nothing is lost and the counts are restored correctly even after a failed-then-retried run. No queue is ever deleted unless it is empty.

Common causes:

Symptom	Cause and fix
The upgrade aborts after ~5 minutes	--timeout was left at its default. Re-run with --timeout 30m (or higher).
The post Job is killed at its deadline (often mid Redis volume reset)	The Job's own activeDeadlineSeconds fired — raising --timeout alone won't help, because it does not extend the Job's deadline. The default Redis volume reset replaces three PVCs one at a time and can approach the 15-minute post ceiling on its own. Raise multiazMigration.postActiveDeadlineSeconds, then set --timeout above the new deadline and re-run.
The drain step times out with messages outstanding	A queue is not being consumed — usually a disabled or unhealthy worker whose queue still holds messages. Enable that worker long enough to drain it, or purge the queue if the messages are disposable, then re-run.
RabbitMQ pods stay Pending after the upgrade	No node capacity in one of the zones. Check the node pool's zone coverage and autoscaling limits, then re-run.
Redis pods stay Pending after the upgrade	The Redis volume reset is on by default; if you turned it off (allowRedisVolumeReset=false) while the volumes are pinned to one zone, re-enable it — or move the Redis volumes across zones yourself — then re-run.
start-workers fails saying pods see RABBITMQ_HA_ENABLED unset	The overlay was not part of the upgrade's values. Confirm multiaz.values.yaml is passed (it sets rabbitmq.ha.enabled=true) and re-run.

If you cannot finish the migration

If the cause cannot be resolved inside the window — no capacity in the second zone, a pending quota request, a bad image mirror — bring the deployment back up rather than leaving it scaled to zero. The pre hook recorded the original replica counts in the state ConfigMap before scaling anything down:

kubectl -n $NAMESPACE get configmap whisperer-multiaz-migration-state -o yaml

Scale the backend and each worker back to those counts with kubectl scale. A plain helm rollback or helm upgrade will not restore them: with autoscaling.enabled the chart omits replicas, and an HPA does not scale a target sitting at 0. (The vendored migration script also ships a restore subcommand that does this; on the hook path the manual scale-back is the equivalent.) This is a clean abort only before delete-queues has run — once the classic queues are deleted and recreated as quorum, finish the migration rather than reverting.

Configuration reference

The multiazMigration block in values.yaml tunes the hook Jobs. The defaults suit most clusters; the ones you are most likely to touch are allowRedisVolumeReset and the deadlines.

Value	Default	Purpose
rabbitmq.ha.migrateQueues	false	Arms the offline cutover for this upgrade. Requires rabbitmq.ha.enabled=true.
multiazMigration.allowRedisVolumeReset	true in multiaz.values.yaml (chart base false)	Re-provision the Redis PVCs across zones during the migration, one node at a time (and widen RBAC to allow pvc/pod deletes). On by default via the overlay; set false to leave Redis volumes in place.
multiazMigration.drainTimeout	300	Seconds to wait for the queues to empty during drain (assumes a quiet-window migration).
multiazMigration.waitTimeout	300	Seconds to wait for each rollout / broker cluster-formation step.
multiazMigration.preActiveDeadlineSeconds	600	Hard ceiling on the pre-upgrade Job. --timeout must stay above it (Helm waits on each hook separately).
multiazMigration.postActiveDeadlineSeconds	900	Hard ceiling on the post-upgrade Job — the longer of the two, so the one --timeout must clear.
multiazMigration.ttlSecondsAfterFinished	86400	How long finished Jobs (and their logs) are kept before garbage collection.
multiazMigration.backoffLimit	1	Job retry count; retries rely on the migration's idempotency.
multiazMigration.image	multiaz-migrate:v1.0.0	The kubectl-baked image the Jobs run. Inherits global.image.registry.
multiazMigration.nodeSelector / tolerations	inherit global	Where the Job pods schedule.
multiazMigration.resources	50m/64Mi … 500m/256Mi	Job pod resource requests/limits.

Rolling back

To return to single-AZ, run helm upgrade without multiaz.values.yaml and repeat an equivalent drain-and-delete sequence — quorum queues cannot be converted back to classic in place either.

A plain helm upgrade without the overlay is not a rollback

Dropping the overlay part-way through a migration leaves the backend and workers scaled to 0 (see If you cannot finish the migration) and sets RABBITMQ_HA_ENABLED=false against queues that are now quorum. In practice the multi-AZ topology is a superset of single-AZ, so a rollback is rarely the right response to a problem during the window — fixing the cause and re-running the migration usually is.