Skip to main content
Version: 2.0.0

AWS Marketplace Troubleshooting

How to read the LLMWhisperer appliance's logs and fix the common issues. Every command runs on the instance: connect with Session Manager or SSH as described in the Deployment Guide.

Over Session Manager, pass --no-pager

In a Session Manager shell, journalctl and systemctl repaint their first screen, so lines appear twice. Pass --no-pager, as the commands below do. Run journalctl with sudo: the Session Manager user cannot read the system journal without it.

Start Here​

sudo python3 /usr/local/lib/llmw/llmw_provision.py --status    # license and stack state
systemctl --no-pager list-units 'llmw*' 'llmwhisperer*' --all # the appliance's services
sudo docker compose -f /opt/llmwhisperer/docker-compose.yml ps # the containers

A healthy appliance shows "state": "provisioned", "first_boot_complete": true and "stack": "running". llmwhisperer.service shows active (exited), which is normal for this service. All containers are running, and those with a health check are healthy. Every state and stack value is explained in the Deployment Guide.

Logs​

Appliance services​

ServiceWhat it doesLogs
llmwhisperer-firstbootFirst boot: license, sizing, credentials, first start of the stack. Does not run again once it has rendered the instance's configuration.sudo journalctl --no-pager -u llmwhisperer-firstboot
llmw-provisionRenews the license every few hours, and finishes a first boot that could not get a license. Runs every 15 minutes.sudo journalctl --no-pager -u llmw-provision
llmwhispererStarts and stops the stack (docker compose up / down).sudo journalctl --no-pager -u llmwhisperer

For example, boot and license together, this boot only:

sudo journalctl --no-pager -b -u llmwhisperer-firstboot -u llmw-provision -u llmwhisperer

Containers​

sudo docker compose -f /opt/llmwhisperer/docker-compose.yml logs --tail 200 <service>
ServiceWhat it is
llm-whisperer-backendThe document API (port 3006)
celery-workersThe document-processing workers
llm-whisperer-dashboardThe dashboard (port 3007)
x-to-pdfOffice and other document conversion
llmw-dbPostgreSQL
llmw-redisRedis
rabbit-mqRabbitMQ

sudo docker compose -f /opt/llmwhisperer/docker-compose.yml ps --services lists every service running on your instance.

Without a shell: the serial console​

First boot and license provisioning also write to the serial console, so you can read them without logging in: in the EC2 console, Actions → Monitor and troubleshoot → Get system log, or:

aws ec2 get-console-output --instance-id <instance-id> --latest --output text

What to send to support​

Email support@unstract.com with your AWS account ID, the AWS Region, the instance type, the version you launched, and the output of:

sudo python3 /usr/local/lib/llmw/llmw_provision.py --status
sudo journalctl --no-pager -b -u llmwhisperer-firstboot -u llmw-provision -u llmwhisperer
sudo docker compose -f /opt/llmwhisperer/docker-compose.yml ps
sudo docker compose -f /opt/llmwhisperer/docker-compose.yml logs --tail 500 <service> # for the service involved
Never send these
  • /opt/llmwhisperer/credentials — your API key and dashboard login
  • .env or any other *.env file in /opt/llmwhisperer — they hold the instance's passwords and license credentials
  • the output of docker inspect or docker compose config, which include the same values

Read through any log before sending it, and remove anything that looks like a key, token or password.

Common Issues​

The license is not provisioned​

On a new instance, --status shows a state other than provisioned, and the stack is not started (stack stays needs_start). (On an instance that already has a license, a failed renewal also shows here, but the stack keeps running.) The license state table lists every value. The usual causes:

stateUsual causeFix
refusedThe AWS account the instance runs in has no active subscription to the listing: it was never subscribed, the subscription has not become active yet, or it has ended. The subscription must be in the same account as the instance.Subscribe in that account. The appliance retries on its own; to retry now, run sudo systemctl start llmw-provision.service.
no_instance_roleNo IAM role is attached, typically after a one-click launch.Attach an IAM instance role, then run sudo systemctl start llmw-provision.service.
unavailableNo outbound HTTPS (443) from the instance: no NAT gateway, no internet gateway route, no public IP in a public subnet, or an outbound security group or network ACL rule that blocks 443.Fix the route or rule. The appliance retries on its own. If the network is fine, contact support.

The reason is also shown in the login banner when you connect over SSH, and on the serial console.

You do not need to reboot. When the appliance obtains the license, it finishes first boot and starts the stack by itself. With a working network that happens on the next retry, up to about 26 minutes later, plus the time to start the stack. llmw-provision.service showing activating for several minutes is normal while it retries an unreachable license service: it keeps trying for up to 20 minutes.

To follow it live:

sudo journalctl --no-pager -f -u llmw-provision -u llmwhisperer-firstboot   # Ctrl-C to stop

The API or dashboard is not reachable​

  1. Is the stack up? --status should show "stack": "running".
  2. Does it answer on the instance? curl --fail --show-error http://127.0.0.1:3006/health/ping should succeed with HTTP 200. If it does not, see The stack does not start.
  3. Is the client allowed? For CloudFormation templates with DashboardCidr, the document API (3006) uses AllowedCidr; the dashboard (3007) uses DashboardCidr and is closed to inbound traffic when that parameter is empty. 0.0.0.0/0 permits access from any IPv4 address. The currently published v2.65.0 template has no separate dashboard range: port 3007 follows AllowedCidr, and this image's usage/license routes do not require login. For a direct AMI launch, check the ranges in your own security group. If the dashboard is closed, reach it through Session Manager port forwarding.
  4. Is the address right? The stack outputs give the private IP. From outside the VPC, use the instance's public IP. An instance only has one if its subnet auto-assigns public IPs (or you associated an Elastic IP), and the subnet routes to an internet gateway.
  5. Did the address change? A public IP assigned by the subnet changes when the instance is stopped and started. Use an Elastic IP for a stable address.
  6. Is it the key? If the API answers but rejects the request with 401, check the unstract-key header against /opt/llmwhisperer/credentials.

To test without changing the security group, use Session Manager port forwarding.

The stack does not start​

--status shows "state": "provisioned" but stack stays needs_start or unknown well after first boot should have finished (35 minutes after launch), or stack is running but curl -s http://127.0.0.1:3006/health/ping gets no answer, or sudo systemctl --no-pager status llmwhisperer-firstboot shows failed.

  1. Read the first-boot log: sudo journalctl --no-pager -u llmwhisperer-firstboot. The usual cause is the API not answering in time because a container keeps restarting.
  2. Find that container: sudo docker compose -f /opt/llmwhisperer/docker-compose.yml ps -a, then read its logs.
  3. Start the stack again: sudo systemctl restart llmwhisperer.service (start does nothing once the service is already active).

If it still does not start, send the diagnostics to support.

The first high_quality, form or table request is slow​

The first high_quality, form or table request after the appliance starts can take several minutes while the OCR engines initialise; later requests are processed normally. This happens after first boot and after every restart of the stack or the instance.

high_quality, form and table stop working while native_text and low_cost still work​

These three modes need ongoing outbound HTTPS to the Unstract license service. If the instance loses it, they stop processing after roughly 100 to 150 minutes, while native_text and low_cost keep working until the license expires. Restore outbound HTTPS (443) from the instance. If those modes do not resume, restart the stack with sudo systemctl restart llmwhisperer.service. Their first requests afterwards can take several minutes while the OCR engines initialise.

Documents are refused with 402 InvalidLicense​

The appliance has no valid license, for example because the subscription has ended. The containers stay up. Check state, message and licence_expires_on in --status. After subscribing again in the same AWS account, launch a new instance; contact support if you need to keep this one.

Webhooks never arrive, or documents fetched from a URL fail​

With OutboundAccess set to https-only (the default), the instance can reach only TCP 443. A webhook receiver or a document URL on http://, or on any other port, is unreachable. Update the stack with OutboundAccess set to all, or add an outbound rule for that destination to the security group.

Usage does not appear on the AWS bill​

  • The instance role must allow aws-marketplace:MeterUsage, and the instance needs outbound HTTPS to the AWS Marketplace Metering Service.
  • Usage is reported every hour. An hour with no documents processed sends nothing, which is normal.
  • Reported usage normally appears in AWS Cost Explorer the next day.

If usage is still missing after the expected billing delay, confirm the instance role and outbound HTTPS access, then contact Unstract support with your AWS account ID, Region, instance ID, and the approximate time pages were processed.

Safe to Do, and Not​

ActionSafe?
sudo systemctl start llmw-provision.serviceYes, any time. It is what the 15-minute timer runs.
sudo systemctl restart llmwhisperer.serviceYes. Requests open at that moment are dropped, and documents being processed are processed again after the restart. Stored data survives.
Reboot, or stop and start the instanceYes. Everything starts again by itself.
docker compose down -vNo. -v deletes the database, Redis and RabbitMQ volumes.
Editing the license files in /opt/llmwhisperer by handNo. The appliance manages them.