# CI/CD speed evidence and design constraints (https://jackin.tailrocks.com/research/engineering/ci/performance/ci-speed-roadmap/) **Research state:** Reference ## Summary [#summary] Current CI uses one job per affected crate, selected from the Cargo metadata reverse-dependency closure. Each job owns its applicable tests, checks, lint, benchmarks, powerset, fuzzing, and Docker coverage. The measured baseline supports affected-crate selection, direct crate execution, post-admission preview builds, and an optional warm persistent-runner path; current workflow state belongs in CI configuration and contributor references. Optimization work should remove repeated setup and compilation while preserving GitHub/Velnor parity and explicit cache evidence. [Shared CI compiler cache](/roadmap/shared-ci-compiler-cache/) and the relevant workflow roadmap items own delivery state and sequencing; this page owns evidence and design constraints. ## Research question [#research-question] PR feedback, main-branch confidence, preview publishing, and tagged releases all need faster wall-clock results without dropping coverage. jackināÆ already has many good CI primitives -- path filters, PR cancellation, aggregator jobs, `Swatinem/rust-cache`, `cargo-nextest`, Docker Buildx layer caches, Bun download caching, pinned mise tools, and split workflows -- but the run triggered by commit [`4c8b94bd05f84a62a04f9f235e2d846f14d04366`](https://github.com/jackin-project/jackin/commit/4c8b94bd05f84a62a04f9f235e2d846f14d04366) shows that broad Rust/runtime changes still produce a long post-merge feedback loop: roughly 12.5 minutes to green `CI`, then roughly 8.5 more minutes before preview artifacts publish. The governing design is staged signal: cheap deterministic checks report quickly, expensive checks start as early as their real prerequisites allow, full coverage still runs before merge or publishing, and every cache has measured hit/miss behavior rather than folklore. "Almost instant" has two distinct ceilings, and they need different work. A cold GitHub-hosted runner always pays a floor of toolchain install, cache restore, and compilation of whatever the change touched, so the realistic best case for a real Rust change on hosted runners is a few minutes, not seconds. The cases that can become near-instant on hosted runners are the ones where change-aware routing skips the Rust surface entirely: docs-only, workflow-only, or construct-only changes should finish in well under a minute. Rust changes only approach instant on a persistent runner whose `target/`, cargo registry, and Docker layers stay warm between runs, because nothing the GitHub Actions cache can do is as fast as the build directory already being on disk. The program therefore runs on two tracks: tighten change-aware routing so non-Rust changes are near-instant on hosted runners, and stand up a warm persistent lane so incremental Rust changes recompile only the edited crate. Change-aware CI means docs-only changes do not build Rust binaries, workflow-only changes do not run unrelated Docker E2E, and construct-image changes do not force unrelated docs deployment work. Rust changes run the affected test/build surface plus cross-cutting gates they can invalidate. An uncertain dependency graph selects the broader safe set. ## Current change-impact routing [#current-change-impact-routing] The pipeline derives a compact run plan from changed paths and workflow inputs: | Change class | Required work | | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | Docs/prose only | repo links, docs build/link check when published docs changed, Codebook docs/prose checks, deploy/live-link checks only on main docs deploy paths | | GitHub workflow/tooling only | `actionlint`/`shellcheck`, affected workflow dry-run or targeted job, plus docs checks if docs changed | | Rust crate-local change | fmt, schema when config/schema inputs changed, cargo check/clippy, affected package tests, dependency/audit policy when lock/tooling changed | | Runtime/launch/capsule handoff change | Rust gates plus nextest Docker E2E lane because `dind_e2e` covers real Docker/runtime/capsule behavior | | Construct image change | construct image build/publish path plus Docker E2E using the rebuilt construct artifact | | Preview/release build logic change | preview/release archive build rehearsal, signing/SBOM/attestation checks, publish mutation only after CI gates | | Unknown or cross-cutting change | broader safe set, then refine classifiers once the run shows which work was actually needed | ## Current evidence [#current-evidence] Commit [`4c8b94bd05f84a62a04f9f235e2d846f14d04366`](https://github.com/jackin-project/jackin/commit/4c8b94bd05f84a62a04f9f235e2d846f14d04366) supplies the broad-change baseline below: Rust runtime/image/launch code, docs, workflow files, and `docker/construct` inputs activated every mainline path filter. All measured runs succeeded. | Workflow | Run | Trigger | Wall time | Long pole | | -------------------------- | ---------------------------------------------------------------------------------- | ------------------------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `CI` | [`27937532691`](https://github.com/jackin-project/jackin/actions/runs/27937532691) | `push` to `main` | 12m 34s | `cargo nextest prepare` 4m 33s, `cargo build validator` 6m 00s, then Docker E2E waited for the full package matrix and ran 3m 07s | | `Docs` | [`27937532567`](https://github.com/jackin-project/jackin/actions/runs/27937532567) | `push` to `main` | 5m 06s | cold `codebook-lsp` installs in `spell-check-docs` and `spell-check-source` at about 2.5m each, docs link/build path 2m 26s, deploy live-link verification 1m 21s | | `Construct Image` | [`27937532647`](https://github.com/jackin-project/jackin/actions/runs/27937532647) | `push` to `main` | 4m 12s | arm64 image publish 2m 59s, amd64 image publish 1m 53s, manifest publish 39s | | `Publish Homebrew Preview` | [`27938150807`](https://github.com/jackin-project/jackin/actions/runs/27938150807) | `workflow_run` after `CI` | 8m 31s | four release-profile `cargo zigbuild` jobs at 6m 47s to 7m 26s, then publish 47s | | `Renovate` | [`27937532547`](https://github.com/jackin-project/jackin/actions/runs/27937532547) | `push` to `main` | 3m 40s | self-hosted Renovate 3m 30s; not part of branch protection but consumes Actions capacity after every main push | | `Renovate Validate` | [`27937532557`](https://github.com/jackin-project/jackin/actions/runs/27937532557) | `push` to `main` | 11s | no meaningful speed issue | Within `CI`, the fastest checks already return early: `changes` 5s, `actionlint` 13s, `fmt` 24s, `schema-check` 33s. The slow path is structural. Docker E2E belongs in the reusable nextest workflow without a serialized cache-seeding gate, allowing it to run beside crate jobs once its construct image is available. The inherited CI matrix split item also called out the real Docker boundary directly: crates/jackin/tests/dind\_e2e.rs is now 1323 LOC and exercises real `docker run`, PTY, runtime launch, and `jackin-capsule` handoff behavior. That should stay a named `docker-e2e` failure surface instead of being buried inside a general package-test lane. It should still belong to the nextest test system, though: the current command is already `cargo nextest run -p jackin --features e2e --profile docker-e2e`, so the better architecture is a dedicated nextest Docker lane inside the reusable nextest workflow, not an unrelated top-level CI job. ## Current controls [#current-controls] * Path filters in .github/workflows/ci.yml, .github/workflows/docs.yml, .github/workflows/construct.yml, and .github/workflows/preview\.yml prevent unrelated workflows from doing full work. * PR workflow concurrency cancels stale PR runs while preserving non-cancelled release serialization. * Rust jobs restore `~/.rustup`, mise-managed tool caches, and `Swatinem/rust-cache` target/cargo caches; the reusable nextest workflow centralizes the expensive test-binary build and shares it with package jobs. * Construct builds use Buildx with registry cache for published main builds and GitHub Actions cache scopes for PR/rehearsal builds. * Construct builds stage the pinned `shellfirm` binary before Docker Buildx runs, so the construct Dockerfile no longer carries a Rust toolchain or from-source `shellfirm` compile stage. * Docs jobs cache Bun's download cache and lychee's link cache, and they separate repo-link checks from full site build/link checks. * `CARGO_INCREMENTAL=0` is already set on the main compile-heavy CI/preview/release paths, which is compatible with compiler-output caching via `sccache`. ## Findings [#findings] ### Independent critical lanes set the floor [#independent-critical-lanes-set-the-floor] The measured baseline had two roughly twelve-minute critical lanes, proving that optimizing only lint or only tests cannot reduce total wall clock. The current workflow expands the affected-crate and reverse-dependent set into one job per crate, preserving attribution without testing unaffected crates. ### Lint and check jobs serialize behind a cache they do not share [#lint-and-check-jobs-serialize-behind-a-cache-they-do-not-share] `cargo clippy --workspace --all-targets --all-features` performs the full type and borrow check before linting, so it is the all-features compile gate; `check-default` remains the default-feature gate. Serializing these jobs behind a separate `check-all-features` job would duplicate compilation without sharing its job-scoped `target/` cache. ### Critical-path sequencing beats more matrix fan-out [#critical-path-sequencing-beats-more-matrix-fan-out] Separate check, clippy, MSRV, fuzz, benchmark, and Docker jobs duplicate crate compilation when they do not share target state. The current ownership model executes applicable checks inside each crate job; run `27999009032` is evidence that direct crate execution beats a cold serialized cache-seeding gate. The matrix should preserve attribution as well as speed: lint/check, deterministic package tests, integration-heavy runtime tests, Docker E2E, capsule, MSRV, validator, dependency policy, and audit stay distinguishable in GitHub checks. Archive fan-out is currently unsuitable because checkout-local tests fail from an archive, while prepared-target artifact download and extraction measured slower than semantic-cache restoration. Reconsider archives only after those tests are archive-safe and an equivalent run proves lower wall clock. Combining packages would erase useful attribution. Graph selection instead removes unaffected crate jobs while keeping every selected crate independent; Docker E2E remains conditional work inside the `jackin` job and reuses its exact target state. ### Main-to-preview admission control [#main-to-preview-admission-control] The preview workflow starts from the exact successful `CI` `workflow_run`. It removes polling, preserves source-SHA and ancestry checks, and treats preview as lower-priority post-admission work. Building preview archives inside the `CI` DAG would remove cross-workflow event latency, but it would also return release-profile compilation to the required critical path. Keep the validator as an independently sharded parity check and keep preview packaging in the post-CI workflow unless dedicated release capacity makes that contention impossible. ### Release and preview should share one build implementation [#release-and-preview-should-share-one-build-implementation] Preview and release build all targets from Linux with `cargo-zigbuild` plus a cached macOS SDK and shared target-scoped archive cache keys. The remaining durable cleanup is one composite/reusable "build signed archive" path, with release adding only tag/version and publishing gates. The Linux cross-build path leaves a single native-macOS build-and-test in the scheduled hygiene lane for parity, so macOS-specific regressions remain visible off the critical path. Preview and release archive jobs share `Swatinem/rust-cache` keys by archive target. ### Tool installation cache misses matter on cold revisions [#tool-installation-cache-misses-matter-on-cold-revisions] The inspected logs show warm `rustup` caches, but cold mise.toml changes caused cargo-installed tools to build from source: `cargo-audit`, `cargo-deny`, `cargo-shear`, and `codebook-lsp` each paid cold-install cost in at least one job. The `mise-action` internal cache missed too because workflows pin the action SHA but did not pin the mise binary version input; some manual caches only covered `~/.local/share/mise/installs/cargo-*`, so non-cargo tools such as `zig`, `cosign`, and `syft` relied on the action's internal cache. This was correct functionally, but noisy for a speed-critical pipeline. Every Rust CLI tool in mise.toml uses mise's Cargo backend with `cargo-binstall` available; cargo-binstall falls back to source builds when no compatible runner asset exists, such as `cargo-fuzz` on linux-arm64. Repeated mise and rustup setup across roughly fifteen jobs supports a baked `jackin-ci` image as the deeper deduplication option. Roadmap owns whether and when that operational migration occurs. ### Docker layer caching is good, and cache-mount risk is lower now [#docker-layer-caching-is-good-and-cache-mount-risk-is-lower-now] Docker Buildx registry and GHA caches are already used. Docker's docs note that `cache-to mode=max` exports more layers than `mode=min`, which matches the current construct cache choice. Because `shellfirm` is staged before Buildx, remaining Docker speed work concerns regular layer-cache behavior rather than Rust cache-mount preservation. ### `sccache` should be adopted, with stats proving each lane [#sccache-should-be-adopted-with-stats-proving-each-lane] Mozilla `sccache` caches compiler outputs through `RUSTC_WRAPPER` and requires incremental compilation to be disabled for Rust cacheability. Hosted-runner GHA trials showed 0% hits and write errors; removing the wrapper then made Cargo target reuse slower despite successful cache restores. GitHub-hosted jobs therefore keep the wrapper for fingerprint compatibility with `SCCACHE_GHA_ENABLED=off`; Velnor uses local `sccache`, where persistent disk can provide useful hits. The GitHub Actions cache backend for `sccache` is not part of the hosted baseline anymore: narrow pilots showed 0% hits and write errors, and the broader fan-out risk remains the same throttling class as Docker GHA cache. Prefer a backend that does not rate-limit -- local disk on a persistent runner, or S3/Redis -- which is also why `sccache` pairs best with the warm-runner lane, where an on-disk `target/` already outperforms a remote compiler cache. ### Incremental compilation is a targeted experiment, not a universal switch [#incremental-compilation-is-a-targeted-experiment-not-a-universal-switch] Cargo's profile docs say incremental compilation stores reusable state in `target`, only applies to workspace/path dependencies, and can be overridden with `CARGO_INCREMENTAL`; dev/test defaults enable it, release defaults disable it. In this repo the CI jobs force it off to keep caches deterministic and sccache-compatible. Re-enabling it may help same-branch PR reruns if target caches are retained, but it increases cache size and conflicts with sccache. Treat it as an experiment for a narrow nextest lane, not for final release artifacts. ### Persistent warm runners are the only instant path for Rust changes [#persistent-warm-runners-are-the-only-instant-path-for-rust-changes] A cold GitHub-hosted runner starts with an empty `target/` and recompiles the changed crate's dependency closure; cache restore still pays archive transfer, extraction, and post-restore compilation. A persistent runner keeps `target/`, `~/.cargo`, and Docker layers warm and compiles only the affected closure. The `velnor` lane remains an explicit opt-in accelerator through the `lanes` workflow input. GitHub-hosted runners stay the default required trust boundary and cold-run parity gate; Roadmap owns remaining Velnor capability work. ### Dual-runner parity is a hard constraint; velnor speedups ride on top [#dual-runner-parity-is-a-hard-constraint-velnor-speedups-ride-on-top] Every lane must be runnable on both GitHub-hosted runners and the self-hosted `velnor` lane ([tailrocks/velnor](https://github.com/tailrocks/velnor)) when a maintainer explicitly selects `lanes: both`, and a GitHub-hosted run must stay the default and required parity gate -- a green warm-lane run has to imply a green cold-lane run, the same PR/main parity rule the repo already enforces. Velnor may carry heavy optimizations that hosted runners cannot (a warm `target/`, warm `~/.cargo`, warm Docker layers, a local-disk `sccache` backend), but only as an opt-in accelerator on top of a baseline that still passes on hosted runners. The rule for every speedup is therefore: when a capability is missing, improve it in `velnor` itself so the capability exists on that runner, then verify the change still runs on both lanes before it lands. Never fork the pipeline into velnor-only behavior that hosted runners cannot reproduce, and never drop a job to hosted-only just to avoid teaching velnor the capability -- both leave the two lanes out of parity. The scaffolding already exists: `matrix-setup` emits a `configs` array and compile and per-crate test jobs fan out with `runs-on: ${{ fromJSON(matrix.config.runner) }}`, so `lanes: both` runs the same contracts on both. The `jackin` crate job builds a changed construct image in its own lane before Docker E2E, while preview and release archive artifacts remain lane-scoped so manual `lanes: both` rehearsals can require both lanes and mutation jobs consume only GitHub-hosted artifacts. The current gaps to close before Velnor is useful as an optional parity lane are concrete: Docker E2E and construct builds assume a working Docker daemon; the construct workflow builds arm64 natively on `ubuntu-24.04-arm`, so Velnor needs an arm path or that leg stays hosted-only; and persistent state on Velnor must be protected from cross-run poisoning. Each gap is a fix to make Velnor compatible, not a reason to make it default. ### Non-critical workflows compete for the runner pool on every push [#non-critical-workflows-compete-for-the-runner-pool-on-every-push] Every push to `main` fires `CI`, `Construct Image`, `Docs`, and `Renovate` concurrently, then `Publish Homebrew Preview` after `CI`. `Renovate` took 3m30s on the analyzed run and is not a branch-protection check, yet it consumes Actions concurrency on every push; when the account's concurrent-runner pool is saturated, critical-path jobs queue behind work that does not gate anything. Moving `Renovate` to a cron schedule instead of a push trigger frees that capacity for the jobs that actually gate merge and publish. Queue time does not show up in per-job durations but is real in wall clock. ### Path filters over-trigger on shared tool config [#path-filters-over-trigger-on-shared-tool-config] Small classifiers route changes: Rust CI turns on only for Rust-relevant entries (`zig` or cargo tooling aliases), and preview turns on only for release-build entries (`zig`, `cargo-zigbuild`, `cosign`, or `syft`). ## Guardrails [#guardrails] * Keep one stable aggregator per workflow for branch protection, but allow separate early-signal jobs to finish before the full aggregator. * Do not remove a check unless it is moved to an equal or stronger gate. "Faster" must not mean "main learns later that release cannot build" without an intentional publish gate. * Keep preview and release publish steps hard-gated to `main` or tag/manual-release rules. * Keep tool installation through mise or a first-party wrapper that is documented in mise.toml; do not add ad hoc language setup actions to workflow files. * Keep cache keys observable. Every new cache must have a documented owner, invalidation input, and expected fallback behavior. * Set `timeout-minutes` on every job. Explicit caps across CI, Docs, construct, preview, release, reusable nextest, Renovate, and scheduled hygiene jobs prevent a hung network call or wedged process from reaching the multi-hour default. * Do not gate the cheap deterministic jobs in front of the heavy ones to "fail first". `fmt`, `actionlint`, and `schema-check` already run in parallel and return in seconds; serializing the compile-heavy jobs behind them would add their latency to the green path for no green-path benefit. Fast-fail is a red-path optimization -- keep it off the happy path. * Keep non-critical workflows off the per-push runner pool. Schedule `Renovate` rather than triggering it on every push so it cannot queue ahead of the jobs that gate merge and publish. * Dual-runner parity capability is mandatory. Every job must stay runnable on both GitHub-hosted runners and the self-hosted `velnor` lane when a maintainer explicitly selects `lanes: both`, with the GitHub-hosted run as the default and required parity gate. Velnor may carry heavier optimizations than hosted runners can (warm caches, a local compiler cache), but only as an opt-in accelerator over a baseline that still passes on hosted runners. When Velnor cannot do something a job needs, improve Velnor itself and re-verify both lanes -- do not fork the pipeline or quietly drop the job to hosted-only. ## Sources [#sources] * GitHub Actions cache searches the current branch first, then restore-key prefixes, then the default branch, which is why broad restore keys can deliberately warm PR branches from `main`; cache storage can also become read-only when budgets/limits are exhausted. [GitHub dependency caching docs](https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching) * Docker Buildx supports `cache-to mode=max` to export more layers than `mode=min`, and Docker documents both registry and GitHub Actions cache backends. Docker also documents that BuildKit cache mounts are not preserved in GHA cache by default. [Docker cache backends](https://docs.docker.com/build/cache/backends/), [Docker GHA cache backend](https://docs.docker.com/build/cache/backends/gha/), [Docker cache mounts in Actions](https://docs.docker.com/build/ci/github-actions/cache/#cache-mounts) * Mozilla `sccache` is a compiler wrapper cache with local and cloud/GHA-style storage backends; Rust usage is through `RUSTC_WRAPPER`, and Rust compiler caching requires incremental compilation to be disabled. [sccache](https://github.com/mozilla/sccache), [sccache action Rust notes](https://github.com/mozilla-actions/sccache-action) * Bun documents `bun ci` as equivalent to `bun install --frozen-lockfile` for reproducible CI installs from committed `bun.lock`. [Bun install docs](https://bun.com/docs/pm/cli/install) * mise's CI docs recommend pinned tool versions for reproducible CI environments, and `jdx/mise-action` supports install arguments and caching. [mise CI docs](https://mise.jdx.dev/continuous-integration.html), [jdx/mise-action](https://github.com/jdx/mise-action) * mise's Cargo backend uses `cargo-binstall` when it is installed; cargo-binstall uses compatible prebuilt binaries when available and owns fallback to source builds. Direct GitHub release matching must not be used for Rust CLI tools unless every supported runner architecture has a matching upstream asset. [mise cargo backend](https://mise.jdx.dev/dev-tools/backends/cargo.html) * `Swatinem/rust-cache` supports `shared-key`, `cache-workspace-crates`, and rust-environment hashing, which match the current direct nextest package/Docker fan-out design. [rust-cache README](https://github.com/Swatinem/rust-cache) * cargo-nextest supports build archives, but this repository rejects archive fan-out and test partitioning because every affected crate must remain one attributable job. [nextest archiving](https://nexte.st/docs/ci-features/archiving/) * Cargo's profile docs describe incremental compilation, `CARGO_INCREMENTAL`, and default codegen-unit differences; use those as the basis for any incremental-compilation experiment. [Cargo profiles](https://doc.rust-lang.org/cargo/reference/profiles.html) * The self-hosted fast lane is powered by the `velnor` project; missing runner capabilities are added there so both lanes stay at parity rather than forking pipeline behavior. [tailrocks/velnor](https://github.com/tailrocks/velnor) ## Related work [#related-work] * .github/workflows/ci.yml * .github/workflows/rust-nextest.yml * .github/workflows/docs.yml * .github/workflows/construct.yml * .github/workflows/preview.yml * .github/workflows/release.yml * .github/workflows/jackin-dev.yml * docker/construct/Dockerfile * .config/nextest.toml * crates/jackin/tests/dind_e2e.rs * crates/jackin-capsule/Cargo.toml ## Code touchpoints [#code-touchpoints] * [/research/engineering/ci/rust-tooling/](/research/engineering/ci/rust-tooling/) -- dependency hygiene, Codebook, coverage, and release-time tooling. * [Shared CI compiler cache](/roadmap/shared-ci-compiler-cache/) -- delivery ownership for compiler-cache and persistent-lane work. * [/roadmap/workspace-registry-cache/](/roadmap/workspace-registry-cache/) -- local pull-through Docker registry ideas for runtime workloads.