Test history

Toki is a build and test system. It owns the history of every test it runs and is designed to own the history of test results fed to it from CI: what passed and failed, on which machine and platform, at which source state, how long it took, and whether it is flaky. Gin may link to that history for a revision, but it does not store or judge test results.

Mostly planned

Test history is being built in stages. Today Toki records test evidence in receipts, and the history model (test identities, per-attempt timings and machine descriptors) exists. Storage, CI import, flakiness signals, duration analysis and the query commands are planned. Commands shown on this page under "Planned" do not exist yet.

What exists today

Test evidence in receipts

Every receipt's payload can carry test evidence: the runner, the granularity of its results, the selection (which tests were selected, and which were excluded, filtered, ignored or unavailable) and every attempt's outcome per result.

Results are as fine as the runner honestly reports:

Granularity	Meaning
case	one result per test case
binary	one result per test binary
run	one result for the whole run

Today, network builds produce whole-run results. Toki reads the executed test count from libtest's test result: lines or Node's pass and fail totals.

Evidence is strict:

• it is complete only when enumeration finished, the selection is not empty and every selected test is covered by a result with a known, nonzero count;
• incomplete evidence makes the verdict incomplete, never pass, even when the command exited 0;
• a test action published without test evidence is incomplete too;
• zero-test and short runs cannot pass.

Timings and machine context

Receipt payload version 1.7 adds what history needs:

• per-attempt timings: wall time, CPU time when the runner reports it, and a failure signature (up to 4 KiB) for attempts that did not pass;
• per-attempt host information: OS and version, architecture, CPU brand, core counts, memory, whether the machine is virtualised or containerised, the authenticated worker, and load samples at the start and end of the attempt.

Load samples are the load average on Linux and macOS, the processor queue and CPU busy share on Windows, and the number of Toki actions the worker was running. Measures a platform cannot report cheaply are left out.

Timings and host data are recorded only. They never change an outcome or a verdict.

Test identity

A test is identified by a digest of its project, runner family, suite and case:

Part	Examples
runner family	rust (libtest, nextest), node-test, vitest, junit
suite	a Rust binary id; a JS test file path; a JUnit testsuite name
case	the framework's full test name; none for a binary- or run-level result

The same test run by cargo test, nextest or imported from CI has the same identity. A runner's version is recorded per run, not in the identity, so a runner upgrade keeps the history. A renamed test is a new identity.

Principles

These rules apply to everything history will do:

• Recording only. History never changes a CI verdict, and never skips, deselects or retries a test. Quarantine is out of scope.
• Honest granularity. History never invents per-case results a runner did not report.
• Exact comparisons. Flakiness compares runs of the same test at the same source tree on a comparable environment.
• Infrastructure is not the test. Attempts that failed to launch, were superseded, cancelled or abandoned, or were killed by containment, are recorded as infrastructure events, not test failures or flakes.
• Every tunable is a per-project setting, with a default and validation.

Planned

Report readers

Readers will turn runner reports into case-level evidence with timings, for both Toki runs and CI output:

Runner	Planned source	Granularity
nextest	JUnit output or libtest JSON events	case
cargo test	test result: totals; JSON events where available	run, or case with JSON
node --test	JUnit or TAP reporter	case
vitest	JUnit reporter	case
anything else	generic JUnit	case if testcase elements exist, else run

Any runner that writes JUnit XML is meant to work through the generic reader.

Storage and CI ingestion

History will live in its own SQLite database on the coordinator, separate from the execution store. Toki runs arrive through the receipt outbox; the receipt id makes redelivery harmless. CI results are planned to arrive through a toki tests import command that takes a JUnit report and its producer metadata, keyed by CI run, job and attempt.

Raw per-attempt rows are planned to default to 90 days of retention, with daily rollups per test, platform and machine kept indefinitely.

Flakiness

A flaky test will be reported with the signals that make it flaky, never as a bare flag:

• retry flake: failed, then passed within one run;
• same-tree flake: passed and failed at the same source tree on comparable environments;
• suspected flake: a failure the baseline classifies as such;
• load-sensitive flake: failures and timeouts that cluster in slow or loaded runs.

Flake rates are planned per platform and machine class, so a test that only flakes on one loaded worker shows as exactly that.

Durations across mixed machines

CI runs on a mix of machines under varying load, so raw durations are not comparable. The design calibrates each run against its own unchanged tests, using the machine class only as a prior, and records an uncertainty for each calibration. On top of that it plans:

• duration regression detection, attributed over the source history;
• classification of timeouts as load-associated, unexpected or unresolved;
• optional time limits per suite or file, in off, report or fail mode (default report).

Benchmarks are the exception: they run on a fixed machine and are compared only on that machine.

CI cost

Toki plans to measure what CI costs per run and per change (wall time, machine time, test count and cost per test), track the main line against a per-project budget, and rank bloat candidates. These are information only: Toki never removes or skips a test on its own.

Queries

Planned commands, through the same generated client schema:

toki tests history <test>
toki tests flaky [--project] [--since] [--platform]
toki tests slow
toki tests at <source>
toki tests classify <run> <test>
toki tests bloat
toki ci cost <change>
toki ci trend [--since]