CI/CD speed evidence and design constraints

Identifies current CI bottlenecks, cache constraints, lane-parity requirements, and safe wall-clock reduction strategies.

Research state: Reference

Summary

Current CI uses one job per affected crate, selected from the Cargo metadata reverse-dependency closure. Each job owns its applicable tests, checks, lint, benchmarks, powerset, fuzzing, and Docker coverage.

The measured baseline supports affected-crate selection, direct crate execution, post-admission preview builds, and an optional warm persistent-runner path; current workflow state belongs in CI configuration and contributor references.

Optimization work should remove repeated setup and compilation while preserving GitHub/Velnor parity and explicit cache evidence.

Shared CI compiler cache and the relevant workflow roadmap items own delivery state and sequencing; this page owns evidence and design constraints.

Research question

PR feedback, main-branch confidence, preview publishing, and tagged releases all need faster wall-clock results without dropping coverage. jackin❯ already has many good CI primitives -- path filters, PR cancellation, aggregator jobs, Swatinem/rust-cache, cargo-nextest, Docker Buildx layer caches, Bun download caching, pinned mise tools, and split workflows -- but the run triggered by commit 4c8b94bd05f84a62a04f9f235e2d846f14d04366 shows that broad Rust/runtime changes still produce a long post-merge feedback loop: roughly 12.5 minutes to green CI, then roughly 8.5 more minutes before preview artifacts publish.

The governing design is staged signal: cheap deterministic checks report quickly, expensive checks start as early as their real prerequisites allow, full coverage still runs before merge or publishing, and every cache has measured hit/miss behavior rather than folklore.

"Almost instant" has two distinct ceilings, and they need different work. A cold GitHub-hosted runner always pays a floor of toolchain install, cache restore, and compilation of whatever the change touched, so the realistic best case for a real Rust change on hosted runners is a few minutes, not seconds. The cases that can become near-instant on hosted runners are the ones where change-aware routing skips the Rust surface entirely: docs-only, workflow-only, or construct-only changes should finish in well under a minute. Rust changes only approach instant on a persistent runner whose target/, cargo registry, and Docker layers stay warm between runs, because nothing the GitHub Actions cache can do is as fast as the build directory already being on disk. The program therefore runs on two tracks: tighten change-aware routing so non-Rust changes are near-instant on hosted runners, and stand up a warm persistent lane so incremental Rust changes recompile only the edited crate.

Change-aware CI means docs-only changes do not build Rust binaries, workflow-only changes do not run unrelated Docker E2E, and construct-image changes do not force unrelated docs deployment work. Rust changes run the affected test/build surface plus cross-cutting gates they can invalidate. An uncertain dependency graph selects the broader safe set.

Current change-impact routing

The pipeline derives a compact run plan from changed paths and workflow inputs:

Change class	Required work
Docs/prose only	repo links, docs build/link check when published docs changed, Codebook docs/prose checks, deploy/live-link checks only on main docs deploy paths
GitHub workflow/tooling only	`actionlint`/`shellcheck`, affected workflow dry-run or targeted job, plus docs checks if docs changed
Rust crate-local change	fmt, schema when config/schema inputs changed, cargo check/clippy, affected package tests, dependency/audit policy when lock/tooling changed
Runtime/launch/capsule handoff change	Rust gates plus nextest Docker E2E lane because `dind_e2e` covers real Docker/runtime/capsule behavior
Construct image change	construct image build/publish path plus Docker E2E using the rebuilt construct artifact
Preview/release build logic change	preview/release archive build rehearsal, signing/SBOM/attestation checks, publish mutation only after CI gates
Unknown or cross-cutting change	broader safe set, then refine classifiers once the run shows which work was actually needed

Current evidence

Commit 4c8b94bd05f84a62a04f9f235e2d846f14d04366 supplies the broad-change baseline below: Rust runtime/image/launch code, docs, workflow files, and docker/construct inputs activated every mainline path filter. All measured runs succeeded.

Workflow	Run	Trigger	Wall time	Long pole
`CI`	`27937532691`	`push` to `main`	12m 34s	`cargo nextest prepare` 4m 33s, `cargo build validator` 6m 00s, then Docker E2E waited for the full package matrix and ran 3m 07s
`Docs`	`27937532567`	`push` to `main`	5m 06s	cold `codebook-lsp` installs in `spell-check-docs` and `spell-check-source` at about 2.5m each, docs link/build path 2m 26s, deploy live-link verification 1m 21s
`Construct Image`	`27937532647`	`push` to `main`	4m 12s	arm64 image publish 2m 59s, amd64 image publish 1m 53s, manifest publish 39s
`Publish Homebrew Preview`	`27938150807`	`workflow_run` after `CI`	8m 31s	four release-profile `cargo zigbuild` jobs at 6m 47s to 7m 26s, then publish 47s
`Renovate`	`27937532547`	`push` to `main`	3m 40s	self-hosted Renovate 3m 30s; not part of branch protection but consumes Actions capacity after every main push
`Renovate Validate`	`27937532557`	`push` to `main`	11s	no meaningful speed issue

Within CI, the fastest checks already return early: changes 5s, actionlint 13s, fmt 24s, schema-check 33s. The slow path is structural. Docker E2E belongs in the reusable nextest workflow without a serialized cache-seeding gate, allowing it to run beside crate jobs once its construct image is available.

The inherited CI matrix split item also called out the real Docker boundary directly: crates/jackin/tests/dind_e2e.rs is now 1323 LOC and exercises real docker run, PTY, runtime launch, and jackin-capsule handoff behavior. That should stay a named docker-e2e failure surface instead of being buried inside a general package-test lane. It should still belong to the nextest test system, though: the current command is already cargo nextest run -p jackin --features e2e --profile docker-e2e, so the better architecture is a dedicated nextest Docker lane inside the reusable nextest workflow, not an unrelated top-level CI job.

Current controls

Path filters in .github/workflows/ci.yml, .github/workflows/docs.yml, .github/workflows/construct.yml, and .github/workflows/preview.yml prevent unrelated workflows from doing full work.
PR workflow concurrency cancels stale PR runs while preserving non-cancelled release serialization.
Rust jobs restore ~/.rustup, mise-managed tool caches, and Swatinem/rust-cache target/cargo caches; the reusable nextest workflow centralizes the expensive test-binary build and shares it with package jobs.
Construct builds use Buildx with registry cache for published main builds and GitHub Actions cache scopes for PR/rehearsal builds.
Construct builds stage the pinned shellfirm binary before Docker Buildx runs, so the construct Dockerfile no longer carries a Rust toolchain or from-source shellfirm compile stage.
Docs jobs cache Bun's download cache and lychee's link cache, and they separate repo-link checks from full site build/link checks.
CARGO_INCREMENTAL=0 is already set on the main compile-heavy CI/preview/release paths, which is compatible with compiler-output caching via sccache.

Findings

Independent critical lanes set the floor

The measured baseline had two roughly twelve-minute critical lanes, proving that optimizing only lint or only tests cannot reduce total wall clock. The current workflow expands the affected-crate and reverse-dependent set into one job per crate, preserving attribution without testing unaffected crates.

cargo clippy --workspace --all-targets --all-features performs the full type and borrow check before linting, so it is the all-features compile gate; check-default remains the default-feature gate. Serializing these jobs behind a separate check-all-features job would duplicate compilation without sharing its job-scoped target/ cache.

Critical-path sequencing beats more matrix fan-out

Separate check, clippy, MSRV, fuzz, benchmark, and Docker jobs duplicate crate compilation when they do not share target state. The current ownership model executes applicable checks inside each crate job; run 27999009032 is evidence that direct crate execution beats a cold serialized cache-seeding gate.

The matrix should preserve attribution as well as speed: lint/check, deterministic package tests, integration-heavy runtime tests, Docker E2E, capsule, MSRV, validator, dependency policy, and audit stay distinguishable in GitHub checks. Archive fan-out is currently unsuitable because checkout-local tests fail from an archive, while prepared-target artifact download and extraction measured slower than semantic-cache restoration. Reconsider archives only after those tests are archive-safe and an equivalent run proves lower wall clock.

Combining packages would erase useful attribution. Graph selection instead removes unaffected crate jobs while keeping every selected crate independent; Docker E2E remains conditional work inside the jackin job and reuses its exact target state.

Main-to-preview admission control

The preview workflow starts from the exact successful CI workflow_run. It removes polling, preserves source-SHA and ancestry checks, and treats preview as lower-priority post-admission work.

Building preview archives inside the CI DAG would remove cross-workflow event latency, but it would also return release-profile compilation to the required critical path. Keep the validator as an independently sharded parity check and keep preview packaging in the post-CI workflow unless dedicated release capacity makes that contention impossible.

Preview and release build all targets from Linux with cargo-zigbuild plus a cached macOS SDK and shared target-scoped archive cache keys. The remaining durable cleanup is one composite/reusable "build signed archive" path, with release adding only tag/version and publishing gates.

The Linux cross-build path leaves a single native-macOS build-and-test in the scheduled hygiene lane for parity, so macOS-specific regressions remain visible off the critical path. Preview and release archive jobs share Swatinem/rust-cache keys by archive target.

Tool installation cache misses matter on cold revisions

The inspected logs show warm rustup caches, but cold mise.toml changes caused cargo-installed tools to build from source: cargo-audit, cargo-deny, cargo-shear, and codebook-lsp each paid cold-install cost in at least one job. The mise-action internal cache missed too because workflows pin the action SHA but did not pin the mise binary version input; some manual caches only covered ~/.local/share/mise/installs/cargo-*, so non-cargo tools such as zig, cosign, and syft relied on the action's internal cache. This was correct functionally, but noisy for a speed-critical pipeline.

Every Rust CLI tool in mise.toml uses mise's Cargo backend with cargo-binstall available; cargo-binstall falls back to source builds when no compatible runner asset exists, such as cargo-fuzz on linux-arm64. Repeated mise and rustup setup across roughly fifteen jobs supports a baked jackin-ci image as the deeper deduplication option. Roadmap owns whether and when that operational migration occurs.

Docker layer caching is good, and cache-mount risk is lower now

Docker Buildx registry and GHA caches are already used. Docker's docs note that cache-to mode=max exports more layers than mode=min, which matches the current construct cache choice. Because shellfirm is staged before Buildx, remaining Docker speed work concerns regular layer-cache behavior rather than Rust cache-mount preservation.

`sccache` should be adopted, with stats proving each lane

Mozilla sccache caches compiler outputs through RUSTC_WRAPPER and requires incremental compilation to be disabled for Rust cacheability. Hosted-runner GHA trials showed 0% hits and write errors; removing the wrapper then made Cargo target reuse slower despite successful cache restores. GitHub-hosted jobs therefore keep the wrapper for fingerprint compatibility with SCCACHE_GHA_ENABLED=off; Velnor uses local sccache, where persistent disk can provide useful hits.

The GitHub Actions cache backend for sccache is not part of the hosted baseline anymore: narrow pilots showed 0% hits and write errors, and the broader fan-out risk remains the same throttling class as Docker GHA cache. Prefer a backend that does not rate-limit -- local disk on a persistent runner, or S3/Redis -- which is also why sccache pairs best with the warm-runner lane, where an on-disk target/ already outperforms a remote compiler cache.

Incremental compilation is a targeted experiment, not a universal switch

Cargo's profile docs say incremental compilation stores reusable state in target, only applies to workspace/path dependencies, and can be overridden with CARGO_INCREMENTAL; dev/test defaults enable it, release defaults disable it. In this repo the CI jobs force it off to keep caches deterministic and sccache-compatible. Re-enabling it may help same-branch PR reruns if target caches are retained, but it increases cache size and conflicts with sccache. Treat it as an experiment for a narrow nextest lane, not for final release artifacts.

Persistent warm runners are the only instant path for Rust changes

A cold GitHub-hosted runner starts with an empty target/ and recompiles the changed crate's dependency closure; cache restore still pays archive transfer, extraction, and post-restore compilation. A persistent runner keeps target/, ~/.cargo, and Docker layers warm and compiles only the affected closure. The velnor lane remains an explicit opt-in accelerator through the lanes workflow input. GitHub-hosted runners stay the default required trust boundary and cold-run parity gate; Roadmap owns remaining Velnor capability work.

Dual-runner parity is a hard constraint; velnor speedups ride on top

Every lane must be runnable on both GitHub-hosted runners and the self-hosted velnor lane (tailrocks/velnor) when a maintainer explicitly selects lanes: both, and a GitHub-hosted run must stay the default and required parity gate -- a green warm-lane run has to imply a green cold-lane run, the same PR/main parity rule the repo already enforces. Velnor may carry heavy optimizations that hosted runners cannot (a warm target/, warm ~/.cargo, warm Docker layers, a local-disk sccache backend), but only as an opt-in accelerator on top of a baseline that still passes on hosted runners. The rule for every speedup is therefore: when a capability is missing, improve it in velnor itself so the capability exists on that runner, then verify the change still runs on both lanes before it lands. Never fork the pipeline into velnor-only behavior that hosted runners cannot reproduce, and never drop a job to hosted-only just to avoid teaching velnor the capability -- both leave the two lanes out of parity.

The scaffolding already exists: matrix-setup emits a configs array and compile and per-crate test jobs fan out with runs-on: ${{ fromJSON(matrix.config.runner) }}, so lanes: both runs the same contracts on both. The jackin crate job builds a changed construct image in its own lane before Docker E2E, while preview and release archive artifacts remain lane-scoped so manual lanes: both rehearsals can require both lanes and mutation jobs consume only GitHub-hosted artifacts. The current gaps to close before Velnor is useful as an optional parity lane are concrete: Docker E2E and construct builds assume a working Docker daemon; the construct workflow builds arm64 natively on ubuntu-24.04-arm, so Velnor needs an arm path or that leg stays hosted-only; and persistent state on Velnor must be protected from cross-run poisoning. Each gap is a fix to make Velnor compatible, not a reason to make it default.

Non-critical workflows compete for the runner pool on every push

Every push to main fires CI, Construct Image, Docs, and Renovate concurrently, then Publish Homebrew Preview after CI. Renovate took 3m30s on the analyzed run and is not a branch-protection check, yet it consumes Actions concurrency on every push; when the account's concurrent-runner pool is saturated, critical-path jobs queue behind work that does not gate anything. Moving Renovate to a cron schedule instead of a push trigger frees that capacity for the jobs that actually gate merge and publish. Queue time does not show up in per-job durations but is real in wall clock.

Path filters over-trigger on shared tool config

Small classifiers route mise.toml changes: Rust CI turns on only for Rust-relevant entries (zig or cargo tooling aliases), and preview turns on only for release-build entries (zig, cargo-zigbuild, cosign, or syft).

Guardrails

Keep one stable aggregator per workflow for branch protection, but allow separate early-signal jobs to finish before the full aggregator.
Do not remove a check unless it is moved to an equal or stronger gate. "Faster" must not mean "main learns later that release cannot build" without an intentional publish gate.
Keep preview and release publish steps hard-gated to main or tag/manual-release rules.
Keep tool installation through mise or a first-party wrapper that is documented in mise.toml; do not add ad hoc language setup actions to workflow files.
Keep cache keys observable. Every new cache must have a documented owner, invalidation input, and expected fallback behavior.
Set timeout-minutes on every job. Explicit caps across CI, Docs, construct, preview, release, reusable nextest, Renovate, and scheduled hygiene jobs prevent a hung network call or wedged process from reaching the multi-hour default.
Do not gate the cheap deterministic jobs in front of the heavy ones to "fail first". fmt, actionlint, and schema-check already run in parallel and return in seconds; serializing the compile-heavy jobs behind them would add their latency to the green path for no green-path benefit. Fast-fail is a red-path optimization -- keep it off the happy path.
Keep non-critical workflows off the per-push runner pool. Schedule Renovate rather than triggering it on every push so it cannot queue ahead of the jobs that gate merge and publish.
Dual-runner parity capability is mandatory. Every job must stay runnable on both GitHub-hosted runners and the self-hosted velnor lane when a maintainer explicitly selects lanes: both, with the GitHub-hosted run as the default and required parity gate. Velnor may carry heavier optimizations than hosted runners can (warm caches, a local compiler cache), but only as an opt-in accelerator over a baseline that still passes on hosted runners. When Velnor cannot do something a job needs, improve Velnor itself and re-verify both lanes -- do not fork the pipeline or quietly drop the job to hosted-only.

Sources

GitHub Actions cache searches the current branch first, then restore-key prefixes, then the default branch, which is why broad restore keys can deliberately warm PR branches from main; cache storage can also become read-only when budgets/limits are exhausted. GitHub dependency caching docs
Docker Buildx supports cache-to mode=max to export more layers than mode=min, and Docker documents both registry and GitHub Actions cache backends. Docker also documents that BuildKit cache mounts are not preserved in GHA cache by default. Docker cache backends, Docker GHA cache backend, Docker cache mounts in Actions
Mozilla sccache is a compiler wrapper cache with local and cloud/GHA-style storage backends; Rust usage is through RUSTC_WRAPPER, and Rust compiler caching requires incremental compilation to be disabled. sccache, sccache action Rust notes
Bun documents bun ci as equivalent to bun install --frozen-lockfile for reproducible CI installs from committed bun.lock. Bun install docs
mise's CI docs recommend pinned tool versions for reproducible CI environments, and jdx/mise-action supports install arguments and caching. mise CI docs, jdx/mise-action
mise's Cargo backend uses cargo-binstall when it is installed; cargo-binstall uses compatible prebuilt binaries when available and owns fallback to source builds. Direct GitHub release matching must not be used for Rust CLI tools unless every supported runner architecture has a matching upstream asset. mise cargo backend
Swatinem/rust-cache supports shared-key, cache-workspace-crates, and rust-environment hashing, which match the current direct nextest package/Docker fan-out design. rust-cache README
cargo-nextest supports build archives, but this repository rejects archive fan-out and test partitioning because every affected crate must remain one attributable job. nextest archiving
Cargo's profile docs describe incremental compilation, CARGO_INCREMENTAL, and default codegen-unit differences; use those as the basis for any incremental-compilation experiment. Cargo profiles
The self-hosted fast lane is powered by the velnor project; missing runner capabilities are added there so both lanes stay at parity rather than forking pipeline behavior. tailrocks/velnor

Code touchpoints

/research/engineering/ci/rust-tooling/ -- dependency hygiene, Codebook, coverage, and release-time tooling.
Shared CI compiler cache -- delivery ownership for compiler-cache and persistent-lane work.
/roadmap/workspace-registry-cache/ -- local pull-through Docker registry ideas for runtime workloads.

CI/CD speed evidence and design constraints

On this page