Builds

This page covers what a network build does on the worker: where its source comes from, how its tree and outputs are managed, which toolchain it uses and how it returns files.

The build request

A request is a JSON document with an idempotency key, a worker selector, a timeoutSeconds deadline and a command:

{
  "key": "app-release-42",
  "worker": "class:build-host",
  "timeoutSeconds": 3600,
  "requiredPlatform": { "os": "windows", "arch": "x86_64" },
  "command": {
    "program": "cargo",
    "args": ["build", "--release"],
    "cwd": "",
    "source": {
      "repository": "https://github.com/your-org/your-repo.git",
      "commit": "0123456789abcdef0123456789abcdef01234567"
    }
  }
}

• worker is an authenticated worker subject, or class:<label>.
• cwd is an optional directory relative to the checkout.
• program is cargo, rustc or node --test. See What is Toki for the accepted forms.
• The same key with the same body returns the original build; a changed body under that key is refused, even after a coordinator restart.

The deadline bounds source preparation as well as execution. Cancelling revokes execution authority and stops the native process tree.

Required platform

requiredPlatform: { os, arch } must match the worker's authenticated metadata exactly, in addition to the selector. Both values are lowercase ASCII letters, digits or underscores, up to 32 bytes. If no compatible worker is connected, the build ends as refused with required_platform_unavailable. A compatible but busy worker may queue it. Requests without a required platform keep the older behaviour and can queue for an offline worker.

Source: Git or Gin

Git commits

source: { repository, commit } names a complete commit object name in an allowlisted HTTPS repository. The worker fetches only that commit into a disposable checkout. Uncommitted edits are not visible: commit and push them first, or publish them as a Gin snapshot.

The worker reads the commit's tree through the Git provider and reports its git/v1: source identity with the result. The coordinator does not fetch repositories itself; the authenticated worker is the source authority.

Cross-OS checkouts

On Windows, macOS and Linux the checkout uses the pinned reference configuration git-checkout/v1, and the receipt records it. The settings are passed on Git's command line, and system, global, user and environment configuration are ignored, so a machine's own Git settings and hooks never reach the build.

Setting	Windows	macOS	Linux
core.autocrlf	false	false	false
core.eol	crlf	lf	lf
core.symlinks	true	true	true
core.ignoreCase	true	true	false
core.longpaths	true

Line endings therefore follow the committed .gitattributes alone. The checkout fails closed on anything the reference cannot reproduce: filter drivers such as Git LFS, symbolic links the account cannot create, missing executable bits on Unix, and names a case-folding volume would merge (README beside readme).

Gin snapshot sources

A worker can also build published Gin snapshots, with no Git commit or export. The operator adds a gin object to the worker's builds settings, naming one or more Gin providers:

"gin": {
  "maxSnapshotBytes": 2147483648,
  "maxEntries": 500000,
  "providers": {
    "gin-main": {
      "endpoint": "https://gin.example.internal:7445",
      "serverName": "gin.example.internal",
      "caPem": "-----BEGIN CERTIFICATE-----\n…",
      "tokenFile": "/var/lib/toki/gin-token",
      "repositories": ["gin:blake3:v1:repository:<hex>"]
    }
  }
}

The endpoint is HTTPS, or plain HTTP on the worker's loopback interface for a gin serve on the same host. A request then carries ginSource: { provider, repository, workspace, snapshot } instead of source, where snapshot is a complete Gin checkpoint or tree identity. Naming both or neither is refused.

The worker calls the separately installed gin executable through Gin's versioned view contract. Gin owns the verified source tree, its placement and recovery; Toki owns the build slot and its outputs. The receipt names the snapshot as gin-snapshot/v2:<digest> and records the source policy Gin actually applied.

INFO

cacheBytes is still accepted for older worker configurations but is reported as unapplied: Toki no longer keeps its own Gin blob cache.

Sparse source views

A Gin request may set ginSource.view to a list of up to 256 snapshot paths. The worker then places only that part of the snapshot, widened to the command's Cargo input closure: manifests, lock and configuration files, whole packages and their path dependencies, and link targets.

Whenever Toki cannot bound the closure, it uses the whole snapshot instead. That includes:

• cargo test or cargo bench without --no-run, and any program other than cargo build, check, clippy, or test/bench --no-run;
• build scripts, procedural macros, and registry or Git dependencies;
• Cargo configuration or environment variables that could choose another program or linker.

If a command fails inside a view, the worker does not report that failure. It places the whole snapshot and runs the command again, reporting only the second run with a sparse source view expanded notice. Outputs built in one view are never reused in another.

Provisioned toolchains

An operator can set builds.toolchainRoot to an absolute directory holding the project's toki-toolchains-<os>-<arch>.toml and optional rust-toolchain.toml. The worker provisions that inventory before it registers and advertises its measured toolchainIdentity. Provisioning failure prevents registration.

A provisioned submission sets command.toolchainIdentity to that digest and a requiredPlatform. Only workers with a matching identity and platform are selected. The worker verifies its tools before and after the command, and the receipt binds the requested identity. A new inventory needs a supervised restart. See Toolchains.

WARNING

Selection manifests are not supported on provisioned workers yet. Toki refuses the combination rather than publish a receipt under the wrong identity.

The materialiser

For manifest-based sources, the materialiser builds one action's tree in a slot from verified store objects. A slot is a directory with one writer at a time:

Path	Holds
tree/	the action's source tree, at the same path every time
tmp/	the action's temporary files
<output>/	reusable output directories, such as Cargo's target directory
slot.json	what the tree last presented
.lock	an OS lock held for the whole action

Files are read-only except at declared writable paths. The command's environment points TMPDIR, TMP, TEMP, CARGO_TARGET_DIR and CARGO_BUILD_BUILD_DIR into the slot. On Windows, each slot gets its own MSVC program-database server.

Modification times

Cargo and MSVC decide what to rebuild from file times, so Toki sets them by content. A file whose content is unchanged keeps the time it had; every changed, added or reverted file moves to the current time, which is checked to be later than the newest output. When freshness cannot be shown, the outputs are removed and the action builds clean.

Recovery

Outputs an action may have left half-written are never reused. They are invalidated when the action was stopped, a tool was killed, processes escaped containment, the disk filled, a program database was contended, or another process held Cargo's lock. A compile error or a failing test keeps the outputs. Outputs that changed outside the lock, or were built under a different toolchain, SDK, target or configuration, also build clean.

Production Gin builds do not use the materialiser for their source tree; Gin places it. Toki still uses the same output journal to decide whether outputs can be reused.

Build artefacts

A Cargo or rustc build can ask the coordinator to keep named output files. Add artifactRequestJson to the command (protocol 1.3 or later):

{
  "version": 1,
  "outputs": [
    {"name": "application", "root": "target", "path": "release/application", "required": true},
    {"name": "symbols", "root": "target", "path": "release/application.pdb", "required": false}
  ],
  "maxBytes": 16777216,
  "retentionSeconds": 86400,
  "requestedOs": "windows",
  "requestedArch": "x86_64"
}

• root is source (the checkout) or target (the leased Cargo target directory).
• Paths are relative and use /. Symbolic links and Windows reparse points are refused; only regular files are collected. There is no globbing.
• Limits: 1–32 outputs, 1 byte to 16 MiB in total, retention from 1 second to 30 days.
• A missing required output makes the build failed, even if the command exited 0.

Declared target files are deleted before the command runs, so a reused slot cannot supply stale bytes.

A successful build's result carries artifactManifestJson, listing each file's name, opaque id, BLAKE3 digest and size. Download a file with the Download RPC using the owning client's credential; other clients are refused. Verify the size and digest before you publish a file.

By default the coordinator retains up to 1 GiB of bundles and serves four concurrent downloads. Set artifact_capacity_bytes and artifact_downloads in server.json to change this. When the retention period ends, the build reports artifact-incomplete.

Dependency selection

Toki can launch a selection manifest (toki.selection-manifest version 1) with a test run: a record of which test obligations were selected and why. Submit it as selectionManifestJson, or reference one submitted before by selectionManifestDigest.

The coordinator validates the manifest before accepting the request: it must name the command's provider (rust for cargo test, js for node --test), the exact program, arguments and directory, the required platform and the source. The worker then checks, before launching, that the source identity and the toolchain identity match the manifest.

Two helpers prepare manifests:

toki selection toolchain cargo path/to/checkout   # print the toolchain identity to bind
toki selection digest manifest.json               # validate a manifest and print its digest

Run toki selection toolchain on the executing host, in the intended checkout. If you cannot, submit rustc -vV and cargo -vV (or node --version) as builds on the target worker, save their stdout, and hash them with toki selection toolchain-output.

Selection never skips tests by itself

The selection layer always falls back to the full set of required tests. Neither first-party adapter claims dependency completeness, so Toki records selections but never authorises omitting a test.

Rust dependency inputs

For Rust, Toki can build a dependency graph from Cargo metadata, the build report and the exact source manifest, and compute which targets a change could affect. The result is diagnostic: it always carries a full-gate fallback, and the complement of the impact set is never proof that a test can be skipped. Out-of-tree includes, generated output and runtime reads are reported as unqualified external inputs.