Is my instance healthy?

This guide is for the person who runs a Lakana instance. It lists what to look at to tell whether the instance works, the command or address that answers each question, and what a bad answer looks like. The commands are for the Compose bundle (Deploying with Compose); lakana worker list and the others also work against any other instance you have an administrator's access to.

After this guide you can check the services, the workers, the queue, the catalog database, the object storage and the disk, you can read the one line a stuck or failed run gives you, you know which lines of the logs matter, and you know what to collect before you ask a question in public.

The short version

Run these in order. The first that looks wrong is where to read on.

1. docker compose ps --all: every service is running and healthy, and the one-off services (migrate, bucket, setup) have Exited (0). (The services)
2. The address /healthz answers. (The health route)
3. lakana worker list: as many workers are alive as you started. (The workers)
4. lakana query list --env dev --status queued does not grow. (The queue)
5. The disks have room. (Disk space)

Nothing here is a service level, and none of it replaces looking at what your own people notice: a dashboard that is slow or a job that did not run is a signal too.

The services

From the directory of the bundle:

docker compose ps --all

What each line tells you:

The one-off services migrate, bucket and setup run once and end; Exited (0) is their healthy state. One that exited with another number is the reason the services behind it did not start: read its log with docker compose logs migrate (or bucket, setup). A service that restarts again and again shows Restarting; its log says why.

The health route

The server has one health route, /healthz. It needs no sign-in, so a script, a load balancer or Docker can ask it:

curl -s "$URL/healthz"

$URL is the public address of the instance. It answers:

{"status":"ok"}

That proves one thing: the server process is running and answers HTTP. It does not look at the database, the storage or the workers, so a server that answers can still fail every query. The image's own check asks the same route from inside the container every 10 seconds, which is what the healthy of the server service in docker compose ps comes from. There is no second route that checks the database or the storage; the rest of this page is how you check them.

A server that does not answer at all is a stopped or restarting container, a proxy or a firewall in front of it, or a wrong address. docker compose ps and docker compose logs server tell which.

The workers

A worker is the process that runs queries and tasks. It reports in to the database every few seconds. It is alive while its last report is recent, and lost once it has been silent for 60 seconds (LAKANA_RUN__LEASE_SECONDS). A lost worker's row stays for an hour, and what it was running is ended by the worker that holds the scheduler role. The commands need the right to manage environments on the instance, which an administrator has:

lakana worker list

lakana scheduler status

Healthy looks like this:

• As many alive rows as you started. LAKANA_WORKERS plus LAKANA_WORKERS_INTERACTIVE plus LAKANA_WORKERS_PIPELINE in the bundle's .env. Fewer means a worker did not start or cannot reach the database; a row with the status lost is one that stopped reporting.
• Every kind of work you use has an alive worker. The column takes says all, interactive or pipeline. See Seeing that a kind has no worker.
• The workers take work. The column doing shows idle or the query, deploy or task a worker runs. An alive worker that stays idle while queries wait is the case to look into: the reason of each waiting query says why (see A stuck run).
• One worker holds the scheduler role. lakana scheduler status names it, and when it last reported. The role ends what lost workers held and runs the hourly sweep, so a holder that has not reported for much longer than a minute, or no holder, is a problem: When a worker is lost says what frees it.

An empty list prints No worker is registered. followed by what to do. With the Compose bundle that is docker compose logs worker first.

The queue

A healthy instance has queries and runs that start soon after they are submitted. Two listings show what waits; run them now and again after a few minutes. A queue that is short and changes is normal; one that only grows is not.

lakana query list --env dev --status queued

lakana runs list --env dev --status queued,running

Use the name of your own environment instead of dev. You see your own queries; with MANAGE_ENVIRONMENT on the environment you see everyone's. The waiting_for column of the query listing gives the reason each one waits. To see what failed in the last day, use this listing (on macOS, write date -u -v-1d +%Y-%m-%dT%H:%M:%SZ instead of the date command shown):

lakana runs list --env dev --status failed --since "$(date -u -d '1 day ago' +%Y-%m-%dT%H:%M:%SZ)"

More filters and paging: Find a run, a query or a deploy.

The catalog database and the object storage

Two things the instance cannot work without. Neither has a command of its own; you read them from what else works.

The catalog database. docker compose ps shows postgres as healthy when it accepts connections. A command that reads the catalog, such as lakana worker list, answers only when the server reaches the database, so an answer proves it. When the database goes away:

• every request that needs it fails, and a new server container does not start (the migrate service runs first and exits with an error);
• each worker logs The worker's heartbeat could not be written; it is tried again every 5 s. and, after a minute without success, No heartbeat was written for 60 s (LAKANA_RUN__LEASE_SECONDS): this worker claims nothing and stops the query it runs until one is written. Check that the database is reachable. Its queries and deploys are then ended as lost. When the database is back, it logs The worker's heartbeat is fresh again; it claims work again.

The object storage. seaweedfs being healthy shows the gateway answers. Whether the instance can read and write its data you only learn by doing it: run a query that reads a table of yours. The tutorial's query is an example; use a table you have:

lakana sql "SELECT * FROM gold.station_ranking ORDER BY warmth_rank"

When the storage cannot be reached, a query that reads a table fails instead of returning rows, and a request that stores or sends an artifact file answers that the storage could not be written or read and that the operator should check whether it is unreachable. Read docker compose logs seaweedfs and docker compose logs seaweedfs-core.

Disk space

Lakana itself reports no disk use and sends no alert, so look at the disk yourself. The bundle keeps its state in three named Docker volumes, all on the disk that holds Docker's data:

The worker containers also keep temporary files under /var/tmp/lakana, inside the container. The sweep removes what a worker that was killed left behind, once an hour (LAKANA_RUN__SWEEP_INTERVAL).

docker system df -v
docker compose exec worker df -h /home/lakana/.cache/lakana-worker

The first command lists every volume with its size. The second shows how full the disk under the workers' cache is. Also look at the disk of the machine itself with df -h on the host. How to reclaim the download cache of the workers is in Python assets. The documentation gives no figure for how much space your data needs; measure on your own data.

A stuck run and why

A query or a task that has not started says why, in one sentence, and a task that failed has a code, a cause and a next step. Both are shown where you look at the run:

• Waiting. lakana query status QUERY_ID --env dev shows waiting_for for a queued query, and lakana runs show RUN_ID --env dev shows it for each task that waits. Most reasons pass by themselves. The ones that need you are no_worker and no_worker_of_kind (start a worker), and a reason that does not change for a long time. All of them, and what to do, are in Why is my query or task waiting?.
• Failed. lakana runs show RUN_ID --env dev shows the code of each failed task and under the table the cause and Next:. The codes that point at the instance, rather than at a project, are these; all of them are in Why did it fail?:

Log lines that matter

Lakana's own records go to standard error, one per line with the time, the level and the logger name; the web server's lines (startup and, if switched on, one per request) have a shorter form without the time. docker compose logs shows both together. Read them with docker compose logs server and docker compose logs worker; the default level, INFO, shows all of the lines below. How to change the level and what the lines of the sweep mean: The sweep and After a restore.

What to collect before you ask

Questions and bug reports go on a public page (Support), where anyone can read them, so collect what lets a stranger understand your case and nothing that lets a stranger into your instance. In this order:

1. The versions. The CLI's and the instance's:

lakana --version

2. The state of the services. The output of docker compose ps --all, and the image you run: the LAKANA_IMAGE line of .env.

3. The failing run or query. Its id and, for a run, the code from lakana runs show. Not the SQL or the data if either is private.

4. The relevant log lines, with secrets removed. This writes the last hour of both logs to a file and replaces two kinds of secret: an access token (it starts lakana_pat_; the server never logs one itself, but a tool or a person may have pasted one), and a password in a connection address:

docker compose logs --no-color --since 1h server worker \
  | sed -E -e 's/lakana_pat_[^[:space:]]+/lakana_pat_REDACTED/g' \
           -e 's#(://[^:/@[:space:]]+:)[^@[:space:]]+@#\1REDACTED@#g' > lakana-logs.txt

Open the file and read it before you post it: the filter knows two patterns, not your data. A table name, a file path, a person's name or a SQL statement can be in a line, and a package index address can carry a credential (user:token@ in its address). Cut a line or replace a word when in doubt.

Never post the .env file, the output of docker compose config (it prints the values of .env), a token, the master key, or the storage keys. If you posted one by mistake, treat it as public: replace the token (Give a second person access), and for the master key and the storage keys start with Manage secrets.

Lakana is run by one maintainer, and a question gets an answer when time allows or not at all; Support says where to ask and what the project does and does not provide. A report with these four things in it is one that can be understood without asking you for more.

See also

• Run Lakana for other people: the whole job of an operator, in order.
• Why is my query or task waiting? and Why did it fail?.
• Deploying with Compose: the services, the memory limit, the sweep and the logs.
• The audit log: who did what.