Migrating to the PostgreSQL Queue

Task processing moves from RabbitMQ to PostgreSQL in v0.180.0. This page covers upgrading across that change. It applies once — if your installation is already on v0.180.0 or later, follow the normal Upgrading steps instead.

Read this before upgrading from v0.179.0 or earlier

v0.180.0 is the only release that runs both systems at once, and that overlap is what lets already-queued work finish.

Upgrading from v0.179.0 or earlier straight to a release after v0.180.0 removes the RabbitMQ workers in the same step that switches processing over, so anything still queued at that moment is stranded. There is no error and nothing fails — the work simply never runs. Choose one of the two paths below.

Which path?

Followed as written, neither path loses work. The choice is about what the upgrade costs you: Path A costs a pause in processing, Path B costs a second upgrade. Work is lost if you upgrade past v0.180.0 without draining first — and, on Path A, if anything is still able to enqueue while you drain.

Can you pause processing while you upgrade?

Answer	Path	What it costs you
Yes	A — upgrade straight to the latest version	One upgrade. You stop submitting work and drain the queues first, so nothing is still in flight when you upgrade.
No	B — upgrade to v0.180.0 first, then continue	Two upgrades, no pause. Work keeps flowing while the old queues drain in the background.

Path A — with a pause

Stop every producer — new submissions through the UI and API, and your scheduled pipelines (see below).
Wait for running executions to finish.
Check the queues are empty — see Checking the queues below. Repeat until it prints READY.
Upgrade to the latest supported version.
Re-enable your scheduled pipelines, and check they are listed and firing.

Step 3 is the one that matters: anything still queued when you upgrade is lost.

Disable your scheduled pipelines for this window

A scheduled pipeline is a producer, so leaving it enabled undoes the drain. If one fires between step 3 and step 4 it does not simply miss a run — it dispatches: an execution record is created and its work is published to RabbitMQ, which the upgrade then leaves with no consumer.

That record is not recovered afterwards. The platform's recovery sweeps look either for executions that were never dispatched, or for ones whose files have all finished — and this is neither. It stays unfinished indefinitely and shows in your execution history as such.

Disable the pipelines before step 2 and re-enable them after step 4. Keep that order: re-enabling on the new version is what makes it safe, because a resumed pipeline there no longer fires an immediate catch-up run against the stale schedule it held while paused.

Schedules themselves survive the upgrade either way — they move to the new scheduler and resume from their next due time.

Path B — no pause

Upgrade to v0.180.0. No configuration changes needed.
Check both worker sets are running — the new PostgreSQL workers and your existing Celery workers.
Let it run. New work goes to PostgreSQL immediately; the Celery workers finish what was already queued.
Check the queues have drained — see Checking the queues below. Repeat until it prints READY.
Upgrade to the latest supported version when convenient.

Checking the queues

Run this against your RabbitMQ pod. rabbitmqctl ships with the broker and is not installed in the worker containers — running it there gives you command not found.

# find the pod first:  kubectl get pods -n unstract | grep rabbit
kubectl -n unstract exec <rabbitmq-pod> -- \
  rabbitmqctl -q --no-table-headers list_queues name messages \
| awk '
    { lines++ }
    $2 !~ /^[0-9]+$/ { printf "  unexpected output: %s\n", $0; bad=1; next }
    $1 ~ /\.pidbox$|^celery_periodic_logs$|^celery_log_task_queue$|^dashboard_metric_events$/ { next }
    $2 > 0 { printf "  still draining: %s (%s messages)\n", $1, $2; n++ }
    END {
      if (bad)      { print "CHECK FAILED - unexpected output. Do not upgrade."; exit 1 }
      if (lines==0) { print "CHECK FAILED - no queues listed. Do not upgrade."; exit 1 }
      if (n)        { print "NOT READY - wait a few minutes and run again"; exit 1 }
      print "READY - the old queues are empty"
    }'

It prints one of four things, and only READY means it is safe to upgrade:

Output	Meaning
`READY - the old queues are empty`	Safe to upgrade.
`NOT READY` + the queues still holding work	Wait and run it again.
`CHECK FAILED - no queues listed`	The command produced nothing — wrong pod name, a pod that is not RabbitMQ, or a broker that is down. Not an all-clear.
`CHECK FAILED - unexpected output`	Something other than `queue<TAB>count` rows arrived. Read the lines it echoes.

The two CHECK FAILED cases exist because errors from kubectl and rabbitmqctl go to stderr, which never enters the pipe. Without them an empty result would look identical to a drained broker. The command also exits non-zero unless it prints READY, so it can gate a script.

--no-table-headers is required and is a different flag from -q: -q suppresses the Listing queues ... banner but not the name<TAB>messages header row, which would otherwise be counted as a queue holding work.

Why the check skips three queues

None of these tells you whether your work has drained:

celery_periodic_logs — has no consumer. If your installation ran a version that published to it, it may hold a large backlog that will never drain. See The celery_periodic_logs backlog below.
dashboard_metric_events — carried metric aggregation, not execution work. Its consumer is removed at v0.180.0, so from then on it is an orphan with nothing attached; anything left in it is aggregation that will not run. It does not affect execution or your data.
celery_log_task_queue — execution-log traffic. Its consumer stays up through v0.180.0 to drain what is already there, so it is expected to have messages during a pre-upgrade check. After the upgrade log traffic moves to Redis, so this queue should fall to zero and stay there — if you see it growing on v0.180.0, that is a real problem, not an expected one.

The rule behind the exclusions: a queue holding messages with no consumer is an orphan, not a slow drain, and waiting will not change it. The consumers column is what distinguishes the two cases.

If you cannot use `kubectl exec`

pods/exec is a separate RBAC subresource that some hardened clusters deny. The broker's management API gives the same numbers over pods/portforward, which is a different permission:

kubectl -n unstract port-forward svc/unstract-rabbitmq 15672:15672

Then, in another terminal, load the broker credentials. They are the same ones you set as CELERY_BROKER_USER / CELERY_BROKER_PASS in your values file; if you would rather not read them out of that file, take them from the Secret the RabbitMQ operator creates:

RABBIT_USER=$(kubectl -n unstract get secret unstract-rabbitmq-default-user \
  -o jsonpath='{.data.username}' | base64 -d)
RABBIT_PASS=$(kubectl -n unstract get secret unstract-rabbitmq-default-user \
  -o jsonpath='{.data.password}' | base64 -d)

Then query the queues:

curl -sf -u "$RABBIT_USER:$RABBIT_PASS" \
  'http://localhost:15672/api/queues/%2F?columns=name,messages,consumers'

curl -sf returns a non-zero exit and no output on an authentication failure, so an empty result means the credentials are wrong — not that the queues are empty.

This returns consumers directly, which is what the orphan rule above needs. The management UI in a browser works over the same port-forward.

Seeing every queue at once

The check above answers one question — is it safe to upgrade. To see the whole picture instead, including which queues still have a consumer attached, use:

kubectl -n unstract exec <rabbitmq-pod> -- \
  rabbitmqctl -q --no-table-headers list_queues name messages consumers \
| sort -k2 -nr \
| awk 'BEGIN{printf "%-38s %9s %9s\n","QUEUE","MESSAGES","CONSUMERS"
              print "------------------------------------------------------------"}
        {printf "%-38s %9s %9s%s\n",$1,$2,$3,
          ($2>0 && $3==0 ? "   <-- BACKLOG, NO CONSUMER" : ($3==0 ? "   <-- no consumer" : ""))}'

It flags the two cases separately:

QUEUE                                   MESSAGES CONSUMERS
------------------------------------------------------------
celery_periodic_logs                       47396         0   <-- BACKLOG, NO CONSUMER
scheduler                                      0         1
file_processing                                0         2
dashboard_metric_events                        0         0   <-- no consumer

The same marker on an execution queue is the one that matters: it means work is queued with nothing to run it. A plain no consumer on an empty queue is harmless — that is a worker that is not deployed.

The `celery_periodic_logs` backlog

You do not need to drain or clear it before upgrading, whatever number it shows. It has no consumer, it never drains, and it holds no execution work — so it does not affect the upgrade or your data, and the check above excludes it deliberately.

It is a defect rather than a design, and worth knowing about for one reason. The queue was written to by a periodic job whose consumer was never deployed; the fix was to stop the publisher. On an installation still running a version that publishes to it, the backlog grows at roughly 86,000 messages a day and can consume enough broker disk to trip RabbitMQ's disk alarm — and a disk alarm blocks every publisher on the broker, which on v0.179.0 means every execution dispatch.

So if a Path A drain appears stuck — nothing draining, NOT READY indefinitely — check the broker for a disk alarm before assuming the queues are simply busy:

kubectl -n unstract exec <rabbitmq-pod> -- rabbitmqctl status | grep -A3 -i alarm

If an alarm is set, contact support before continuing. Free disk or purge that one queue, and normal draining resumes.

What not to do

Each of these fails without an error message.

Do not disable the Celery workers in the same upgrade as Path B step 1. That strands whatever is still queued — no consumer, no error, no failure on the sending side.
Do not disable the schedule mirror. The chart refuses to deploy, which is the safe outcome. Working around it by re-enabling the old scheduler causes schedules to fire twice.
Do not disable the PostgreSQL worker fleet as a way to revert. It looks like the undo lever and is not — you get a platform that accepts uploads, reports success, and runs nothing. To revert, redeploy the previous image and chart together.
Do not roll back and assume schedules follow. They must be handed back explicitly — contact support for the procedure.
Do not re-enable the old scheduler afterwards. Your schedules already belong to the new one, so anything the old scheduler still holds will fire twice.

Notes

RabbitMQ is still required for v0.180.0. It is removed in a later release.
Queue names are unchanged in v0.180.0, so execution work already queued is picked up by the same consumers as before. That is why Path B needs no pause. The exception is the three queues the check excludes — dashboard_metric_events loses its consumer at v0.180.0, and celery_periodic_logs never had one.
Schedules move automatically. Existing schedules are copied into the new scheduler and ownership is handed over during the upgrade. There is no catch-up burst — they start from their next due time rather than firing once for every interval missed while the upgrade ran.
Upgrade from a version before the PostgreSQL queue (v0.179.0 or earlier), or from a fully current one. A database left partway through an earlier PostgreSQL-queue rollout has migration records with no matching files, and the upgrade will not resolve them correctly. If you are unsure which state you are in, contact support first.

If the upgrade reports a schedule-migration failure

The schedule migration runs after the database migration. On a jump across several versions, database index rebuilds can still be running when it starts, and the schedule step fails. It fails loudly rather than half-completing, so you will see it.

Re-run the upgrade once the database migration has finished. The schedule migration is a post-upgrade hook, so running the same helm upgrade command again re-runs it — with the settings your deployment needs, which depend on which workers you have enabled. Both the upgrade and the migration are idempotent, so repeating them after a partial or a successful run is safe.

Do not run the migration command by hand

It takes flags the chart derives from your values. Passing the wrong ones can hand the metrics schedules to a scheduler that has no worker to run them, which stops those jobs with no error — the same silent stall the migration exists to avoid. Re-running helm upgrade gets this right for your deployment; running the command directly does not.

If it fails again, capture the Job's output and contact support:

kubectl -n unstract logs job/pg-schedule-mirror

Schedules that were not transferred fire on neither scheduler, so this is worth escalating rather than working around.

Which path?​

Path A — with a pause​

Path B — no pause​

Checking the queues​

Why the check skips three queues​

If you cannot use kubectl exec​

Seeing every queue at once​

The celery_periodic_logs backlog​

What not to do​

Notes​

If the upgrade reports a schedule-migration failure​