Checks

Exact facts about data quality checks: the files, the keys, what a deploy refuses, what a check and a set-back can end as, the commands, the routes, who may read the results and what is audited. Check your data teaches them with an example.

Files and names

lakana init writes one check, checks/silver/station_temperatures/plausible_range.sql, for its sample asset (--no-sample leaves it out).

The comment

Any other key is refused, and so is any other value of on_failure; the message lists the right ones. A file without the comment takes the project's default.

The default of on_failure

The value in force for a check is, in this order: its own on_failure, then checks.on_failure in lakana.yaml (block or warn, also in an environment's override; see Projects and deploys), then block. A deploy stores the resolved value on every check, so nothing reads lakana.yaml at run time, and lakana check list shows the value that was in force. keep cannot be a default.

What a deploy refuses

A deploy and a dry run read checks/ and refuse the whole deploy if one check is wrong. Each message names the file and, where there is one, the line. All problems of a project are reported together.

A project without checks/ deploys and runs as before.

What a check can end as

Each check of a run has one result.

A blocking check that failed or errored sets the table back. A keep check does not: its result has no rollback outcome and the table keeps the data (see Keeping a table). What a set-back did is the rollback outcome of the result, and the group in lakana runs show reads it as a phrase:

A result has no rollback outcome when the check passed or warned, and for a table the project only reads.

Keeping a table

A check with on_failure: keep fails the task and leaves the table as the run wrote it. The table gets a mark: the run, the task and the check (kept_run_id, kept_task_run_id, kept_check on the table; the first check by name, followed by and N more when several failed). While the mark stands, every task of the project that builds the table is stopped before it reads the table, except as the rules below say, and ad-hoc readers (lakana sql, the SDK, the REST API) read the table as before. A table has one of three states: published, not_published (a first build failed a block check) and kept.

The rules

• Who sets the mark. Only a task that built the table, when a keep check of that table fails. A keep check of a table the project only reads fails its task and stops the assets that read it in that run, and sets no mark. A keep check that could not run because a table of another project that it reads is kept sets no mark either; its task fails and its message says the checks could not run. One exception: when the table the task builds already carries a mark, the build has replaced the data that mark names, and no check could run on the new data. The mark then moves to the new run and names the check as one that could not run, and the table's state text says it holds data no check could run on. Readers stay refused.
• Where keep is allowed. On a check of an asset of materialize: table (SQL or Python) and of a table the project only reads. A deploy refuses it on a check of an incremental asset (building it again only adds rows, so the failed rows would stay) and of a view (it holds no data); see What a deploy refuses.
• Who clears it. A run of the project that builds the table, in one of three ways: the table's own task passes all its block and keep checks after a rebuild; the check task of a later run passes (see below); or a project version that has no checks for the table rebuilds it. Each clearing writes one TABLE_KEPT_CLEARED event (why: checks_passed or rebuilt_without_checks). In rare cases (the worker stops right after the write) the mark survives a good rebuild, and the next build of the table clears it.
• A set-back follows the mark. A table remembers the states it had before the current one (kept_earlier, newest last, at most the newest eight), each with the snapshot it was a verdict on: a mark, or "no mark" when the data had none. A mark that moves to a newer run (a keep check that fails again, or a rebuild whose checks could not run) puts the state it replaces on that stack, and so does a clear: a passing check phase, a later run's check task or a rebuild without checks puts the mark it removes on the stack, with the snapshot that mark was a verdict on, and records the snapshot its own checks judged. When a set-back moves the table to a snapshot that the table's current verdict does not describe, the table takes back the newest state the stack holds for that snapshot, in the same step. That happens whoever set the current state: the run that wrote the data can be cancelled after another run's check judged its data and cleared the mark, and the table then has the earlier data and, again, its earlier mark. The entries above the one used describe data newer than the snapshot, which is gone, so they go. lakana table get and the refusal of a reader name the run whose data the table holds again, with the check and its could not run wording as they were. If that state is "no mark", the data had passed its checks or had none to pass, the table has no mark afterwards, and readers are let in again; only then. If the stack has no state for the snapshot (the history ran out of eight entries, or the table was marked before upgrading to migration 0031), nothing says whether the data passed its checks, so the table gets a mark that says the checks are not known: it names the run that was set back, refuses readers like every mark, and is cleared by the next passing checks like every mark. When the verdict already describes the snapshot the table goes back to, nothing changes. A set-back of data a mark does not name (a block failure of a later run after the table was kept in an earlier one) leaves that mark; a first build has nothing to go back to and keeps its mark. A table that has no verdict (it was never marked, or was cleared before migration 0031) is left as it is by a set-back. A set-back also acts on a mark that names another run when that mark is the verdict on the data the set-back removes (two writers set back in the reverse order of their writes). The set-back records which of these happened in the TABLE_ROLLED_BACK event (see What is audited). A table that was kept once and then cleared keeps the snapshot of its latest verdict, which a passing check phase of a later build refreshes (the state it replaces goes on the stack), so a set-back of a later write finds the verdict of the data before it.
• A later run. A run that builds an asset which reads a kept table without rebuilding it gets a check task for the kept table (kind source_check, see SDK and REST). It runs the table's checks before any reader. If they pass, the mark is cleared and the readers are built; if one fails or cannot run, the task fails, the readers are not built, and nothing is set back. If the project has no check for the table any more, the task fails too: only building the table again clears the mark. A reader that only reaches the table through views is covered too; a table asset that is not built stops the search, because it holds older data that was checked when it was built.
• Which tasks read a kept table. In a run of the project that builds the table: the table's own task (its checks, its rebuild) and every task that comes after it. Any other task is refused. A task of another project is always refused, also when that project has a check of its own on the table and the check passes, because its check may be weaker; only a run of the project that builds the table clears the mark. The count of a check may read a kept table of its own project.
• A table whose asset was removed from the project has no project that builds it, so no run clears the mark. Add the asset back and build it, or drop the table (lakana table delete TABLE --env ENV). Ad-hoc reads work throughout, and no command clears the mark by hand. A deploy or dry run that removes such an asset says so in its warnings (the SDK's Deploy.warnings, the REST DeployOut.warnings), and lakana deploy prints each as a warning: line on stderr; it lists only tables you may read, and it is not a refusal.
• lakana runs resume of a run in which the check task of a kept table failed builds that table again, also when the run was cancelled before the check ran.
• A cancel while a worker runs the checks can leave kept data without the mark, as a cancel leaves data that was not checked; a later build of the table sets or clears it. When the worker was lost, or the task waits for its checks again after a lost worker, the next worker sets the table back instead.

The messages

Commands

lakana check list

Lists check results, newest first.

The columns are run (the first 8 characters of the run id), table, check, outcome, rows, on_failure and finished_at; a narrow terminal leaves on_failure and run out first, and the line under the table says so. --output json has every field, including the whole run id, the error, the rollback outcome and the note about policies. A run without results prints Run RUN_ID has no check results.

lakana check rows

lakana check rows RUN CHECK --env ENV shows the rows a check found in a run, as they were then, also after the table was set back or when it was never published. RUN is the whole run id. CHECK is the check's name, NAMESPACE.TABLE/NAME, as lakana check list shows it; quote it in the shell if it holds a space or a quote. The messages of a failed task print the command with the name already quoted. For a name with a non-ASCII letter, a backslash or a control character, or one longer than 200 characters, a message cannot print the name as it is, so it prints CHECK and says to copy the name from lakana check list.

The output is that of lakana sql: a table, then the number of rows. The details are in The rows of a check.

lakana table maintain

For a table with failed checks, the line of lakana table maintain has one more part: Maintained ENV/TABLE: expired 2 snapshots, deleted 4 files (2048 bytes), removed 1 tag of a failed check older than 7 days (removed 2 tags of failed checks for several). A dry run prints Dry run for ENV/TABLE: would expire 2 snapshots, would delete 4 files (2048 bytes), would remove 1 tag of a failed check older than 7 days. The part is left out when no tag was removed. See The tag of a failed check.

A maintenance run stops at the first safe point after LAKANA_RUN__MAINTENANCE_TIMEOUT (30 minutes by default). It then fails with the cause timeout, and the table stays consistent; what it had expired or deleted stays so, and running lakana table maintain on the table again continues where it stopped. See A maintenance run past its time limit.

lakana runs show and lakana run

After the table of tasks, both print one group for every task with checks. A group is a heading line and a line for each check:

silver.station_temperatures   FAILED   2 of 3 checks passed, rolled back
  has_rows                    passed
  no_missing_station          FAILED   1 row
  plausible_range             warned   1 row

The heading holds the table, its state, how many of its checks passed (a warning counts as passed), and the rollback phrase. For a table the project only reads it ends with not built by this project. For the check task of a table that was kept after a failed check it ends with kept after a failed check. For a check task of a run whose project version is gone, which cannot tell the two, it ends with check task. The state is FAILED when a blocking check failed or errored, warned when a check warned or a warning check errored, and passed otherwise. A check line is passed; FAILED or warned with the number of rows; or ERRORED (errored for a warning check) with the error. A keep check reads failed (kept) with the number of rows, or ERRORED (kept) with the error; it has no rollback phrase. A result that was counted under a row filter or column mask has a line note: checked under a row filter or column mask of NAME: rows and values it cannot see were not checked under it. A run without checks prints nothing more than it did before.

Other commands

• lakana deploy --dry-run and lakana deploy print Checks: and, for each table, its checks with (warn) after one that only warns and (keep) after one that keeps. When lakana.yaml sets the default of some checks, a line follows: 2 checks take `on_failure: warn` from lakana.yaml. (1 check takes for one).
• lakana table get and lakana table list show a state of published, not published or kept, and table get says why for the last two.
• lakana runs cancel ends a run whose checks cannot finish; see Limits in Check your data. A task whose table may hold rows no check has passed is set back first, so the cancelled run ends only once a pipeline worker has done that; see A worker that is lost.
• lakana query cancel refuses the query of a check and names lakana runs cancel; see Stop a query.

The rows of a check

A result holds a count and never a row. lakana check rows makes the rows from the check's SQL.

How it works. The command queues an ordinary query of yours whose text is the check's SQL, taken from the project version the run used. The query is tied to the check's result. When it runs, the worker reads the checked table at the snapshot the check read, and every other table the SQL names as it is now, under your own grants, row filters and column masks. Nothing in the request names a snapshot.

What the rows are.

• The rows the check found in that run, as they were then, also after the table was set back, and also for a table that was never published.
• Only the columns that the table still has in the same form: under the same name, with the same type, and as the same column (not one dropped and added again under the name). A column that the run added and that was set back, or that was dropped or changed since, is not shown, because a column mask can only be set on a column the table has now. A first build that is not published keeps its own columns, so all of them are shown.
• A row filter of yours is applied to the snapshot's rows. A column mask of yours is applied to the column it names. A row filter, or a mask's expression, that reads a column which is not the same column in the snapshot and in the table now stops the read with a message: Lakana never leaves a filter out to make a read work. A mask on a column that is not shown is skipped, since nothing is shown of it.
• A check that passed is queued like any other and returns no rows.

Who may ask. The right to read the run (READ_ENVIRONMENT on the environment, or RUN) and SUBMIT_QUERY on the environment; the roles ANALYST and above have both. A caller who lacks them is refused before Lakana looks at any run or check, so the refusal says nothing about whether they exist. Then, once the check is found, the right to read every table the check's SQL names (READ_TABLE): the SQL becomes the text of the caller's query, so a caller without it is refused (403) before any other answer about the check, with a message that names none of the tables and not their number. The worker checks the right to read the tables again when it runs the query, as for every query.

The check's SQL in a project version. lakana project-version show (GET /api/v1/environments/{env}/project-versions/{id}) shows a check's sql and references only to a reader who may read every table the SQL names, read again from the SQL itself. For anyone else the entry keeps name, table, check, path, on_failure and description, and hidden_references counts the tables they may not read (0 when nothing is hidden).

In the audit log. The QUERY_COMPLETED event of such a query has, next to sql_text, tables_touched, row_count and status, the field check_result_id: the id of the check result whose rows were asked for. The event of every other query does not have the field. An administrator can so tell a read of a table's failed or earlier state from an ordinary query of the table.

A task cannot ask. A principal that is a task of a run, or a token bound to a task, is refused with A task cannot read the rows a check found: they are shown to a person's own query, never to a pipeline. Run `lakana check rows` yourself, outside the task. The REST API answers 403.

How long the rows can be shown

The tag of a failed check

A set-back keeps the failed state of the table under a snapshot tag named lakana-check-failed-<run id>, so that maintenance does not take it away at once. A tag has no creation time, so its age is the age of the snapshot it names, which is older than the tag by the time the checks took.

• One run of lakana table maintain removes a tag whose snapshot is more than seven days old, and the snapshot with it, whatever --keep-days and --keep-last say, unless another reference (a branch, another tag) names the snapshot, a remaining branch needs it as an ancestor, or a run still in progress needs it to set its table back. Such a snapshot stays without the tag, and a later maintenance run expires it by the ordinary rules once that run has ended.
• Nobody can create a ref with the same prefix, lakana-check-failed-, through the Iceberg REST catalog: a commit that sets one is refused (below), and Lakana's own set-back is the one setter. A table that an earlier release left with such a ref made by hand keeps it, and lakana table maintain removes it after seven days as before.
• The maintenance report lists the removed tags as tags_removed (names; empty when none), and so does the TABLE_MAINTAINED event in the audit log. A dry run lists the tags it would remove.
• A table that was never published stays not published when its tag and snapshot go; the next run whose checks pass publishes it as before.

When it cannot show rows

The first seven messages come back at once as the error of the command: 409 on the route and lakana.errors.Conflict in the SDK, except the fifth, which is 404 and lakana.errors.NotFound. The others are the failure of the query, after it was queued.

SDK and REST

The second route returns {"items": [...], "next_cursor": ...}, newest first, with limit from 1 to 200 (default 50). Send the same table with every page.

A result has these fields: run_id, task, name, check, path, table, table_id, on_failure, outcome, failing_rows, error, checked_snapshot_id, rollback_outcome, policy_applied, policy_note, query_run_id, started_at and finished_at. A result holds a count and never a row of the table.

Other objects carry checks, too: a run's tasks have kind (asset, or source_check for a table the project only reads, and for the check task of a kept table), kept_check (true for the check task of a table that was kept after a failed check, false otherwise, null for a check task of a run whose project version is gone, which cannot tell; the commands use it to tell the two apart and print check task for null), setting_back (true while the task has not ended and its next claim only sets its table back: its worker was lost after it may have written, or a cancel found it waiting with rows no check has passed; false once it has ended; see A worker that is lost) and checks; a deploy has checks (table to a list of the check names and their on_failure); a table has state (published, not_published or kept) and state_message. A read of a table that is not published answers 409 on the platform routes, and "not found" with the same text on the Iceberg REST catalog's load route; so does a task's read of a kept table that it may not read. The answer to a commit or a create on the Iceberg REST catalog carries the table's history (its snapshots, its references, among them the tag of a failed check, and its properties) only for a caller who may read the table, as its load would; anyone else gets the new metadata location and the table's current schema, partitioning and sort order, and the commit itself lands all the same. Such a client can commit once with each table object it holds: its next commit needs a fresh load of the table, which requires READ_TABLE. A refused commit tells such a caller nothing of the table either (below). A deploy's list of checks may carry "from": "lakana.yaml" on a check that took its on_failure from the project's default.

When a commit through the Iceberg REST catalog is refused

A caller who may not read the table, or whose load the table would refuse (it is not published, or kept from a task), gets fixed texts instead of the details, which would name snapshots, references or properties of the table. A reader gets the details. For such a caller, removing a property the table does not have changes nothing instead of failing, as in Iceberg itself.

Who may read the results

• The results of one run need what reading the run needs: READ_ENVIRONMENT on the environment, or RUN.
• Listing the results of an environment needs READ_ENVIRONMENT.

Looking at the rows of a check needs READ_ENVIRONMENT or RUN, and SUBMIT_QUERY; see The rows of a check.

Running a check needs no permission of its own. It is part of a run and reads as the run's principal; that principal needs the right to read every table the check names, as for any query. Setting a table back is not a route and not a command: only the worker that runs the task does it.

What is audited

A passed or warned check is in the results and not in the audit log. Read the events with lakana audit list --event-type CHECK_FAILED.

What a project stores

A deploy stores each check in the project version's manifest: its name, table, path, on_failure, description and SQL. A manifest with checks has the version number 4, and a project without checks keeps its old number. A worker of an earlier release refuses a version-4 manifest with the message that tells you to update the worker; upgrade the workers together with the server (see Upgrades).

A task never builds a table unchecked when its project version lists checks for it. If a run was started by a server that does not know checks, the task fails before it writes anything, with a message that says to upgrade the server (see Upgrades).

If the stored manifest holds a check entry that cannot be read, which a deploy never writes but a hand-edited or damaged manifest can, the task fails and the table is set back, or left not published when this run built it for the first time. An entry is unreadable when a field is missing or is not text, when its name is not <table>/<check>, or when two entries have the same name (both count). The other checks of the table still run and are recorded, and one more blocking result, named <table>/(unreadable), says that the project version lists a check that cannot be read. A deploy of the same commit reuses the stored project version, so it does not repair this: deploy a new commit of the project, which writes the project version afresh. An entry that does not name a table of the manifest, a checks entry that is not a list, and a version-4 manifest without any check count against every table of the run, so every task that builds a table fails. The names (none) and (unreadable) are reserved; a check file cannot have them.