CI/CD performance: prioritized speedup backlog

Status: Open — analysis and prioritized backlog. This item is the measured follow-up companion to the CI/CD speed roadmap, which tracks the shipped baseline. It ranks the remaining high-leverage levers from highly likely to less likely, grounded in job-level and step-level evidence from the merged PR #641 check suite (June 2026) rather than opinion.

Problem

The PR #641 check suite was slow for two separate reasons, and a deeper analysis of where time actually goes points at one dominant cost the existing speed work has not yet eliminated.

1. jackin-dev was the overall long pole (~11m53s) because build aarch64-apple-darwin (GitHub) spent 10m40s inside jdx/mise-action before the Rust build even started. The compile itself took 24s.
2. CI has a structural serialization where, when docker/construct/** changes, all pure-Rust nextest package shards wait behind the E2E construct-image build even though only the Docker E2E shard uses that image — adding ~3 minutes of idle time to unit-test shards.

The single fact that should drive prioritization: the dominant CI cost is not running tests — it is compiling the same ~741-crate dependency graph from cold, roughly ten times in parallel, on most runs. Measured on run 28256244365:

• The jackin nextest shard spent 3m02 compiling and ~12s running its 719 tests (each ≈0.01s). Compilation is ~94% of the job.
• The same cold full compile recurs in clippy (2m29), check-default (2m16), msrv (2m19), bench-build, fuzz, each of the five nextest shards, docker-e2e, and inside the construct image — and none of these jobs share compiled objects, because every Rust job sets SCCACHE_GHA_ENABLED: "off".
• The cache goes cold because Swatinem/rust-cache's key includes the Cargo.lock hash, and Cargo.lock changed in ~48% of recent commits (PR #641 included). docker/construct/** changed in ~43% of recent commits, so the E2E serialization hits nearly half of all PRs.

So the biggest levers either compile fewer times or make the compiler cache survive Cargo.lock churn. Everything else is secondary. Do not pursue any of this by removing PR/main parity: the workflow-authoring rules in .github/AGENTS.md require every push-to-main failure invariant to be evaluated at PR time against the same inputs.

Relationship to the CI/CD speed roadmap

The CI/CD speed roadmap already shipped the baseline (per-lane cargo-registry-warmup, semantic per-shard rust-cache keys, the reusable nextest workflow with a path-routed Docker E2E lane, the dual-runner GitHub/Velnor parity proof, and timing/cache summaries). It also already recorded outcomes that constrain the recommendations below — this item does not re-propose them naively:

• Hosted GHA sccache was measured and rejected (0% hits with cache write errors). The corrected baseline keeps RUSTC_WRAPPER=sccache for Cargo fingerprint/target-cache compatibility but disables the GHA backend with SCCACHE_GHA_ENABLED=off. So "turn sccache on" below means a real remote backend (S3/R2/WebDAV/Depot Cache), never the GHA backend again.
• A cargo nextest archive fan-out experiment built successfully but failed an existing checkout-dependent test and was reverted. So the build-once/test-many lever below is gated on first making tests relocatable (rely on runtime CARGO_MANIFEST_DIR / nextest binary env vars, not baked absolute paths).
• Swatinem/rust-cache does not dependably cache workspace-crate outputs across jobs — confirmed when the Docker E2E lane restored the shared exact key yet still recompiled jackin-capsule (~7s) and the docker-e2e test graph (~32s). This is why the archive handoff, not another target-cache family, is the remaining candidate for removing duplicate workspace compilation.

This item also intersects Rust CI tooling and dependency hygiene (which is adopting cargo-llvm-cov, the prerequisite for dynamic test selection) and lists cargo-hakari and cargo-chef as open.

Master priority: highly likely to less likely

Ordered by (impact × confidence × breadth ÷ risk). Apply top-down; stop once a warm run is under ~4 minutes and cold outliers are gone, because past that complexity outweighs the marginal second saved.

Tier 1 — highly likely, biggest result, do first

Items 2 and 3 are complementary: 2 makes each compile cheaper; 3 makes there be only one of them.

Tier 2 — likely, strong result, do next

Tier 3 — plausible, moderate result, measure while applying

Tier 4 — lower likelihood, diminishing, or measure-first

Sequencing rationale

Do 1 first (largest single number, isolated to one job). Do 2 + 3 together next — they are the structural fix for the "compile ten times cold" root cause and benefit every subsequent run. 4 + 5 are cheap, high-frequency wins layered on top. Only then evaluate 6–8 with real before/after numbers, and treat 9–14 as experiments gated on measurement.

Evidence

Workflow-level timing (PR #641 final suite)

jackin-dev aarch64 macOS setup stall

The target matrix builds four release artifacts in parallel. Three finished tool setup in 14–16s; one took 10m40s.

This is not compile time — the compile was fast. The time disappeared in tool setup, which points at an intermittent setup/cache/network/cache-save contention problem, not inherent build complexity. All four matrix jobs install the same set (cargo-binstall rust zig cargo:cargo-zigbuild cosign syft cargo:sccache) with cache_key_prefix: "mise-v2".

Levers: instrument mise-action (per-tool grouping or log_level: debug) so the next stall names the exact tool; set an explicit cache_key and cache_save: false on matrix jobs after a single tool-prewarm job so the four targets never race to save the same tool cache; reassess whether PR jobs need cosign/syft on every target (keep them only where artifact attestation is a required main invariant); optionally run release-artifact builds on the velnor lane. Expected win: jackin-dev drops from ~11m53s to roughly ~1.5–2 min on warm setup.

CI critical path (warm reference run 28256244365, ~8m26s)

changes(8s) ─▶ construct-e2e-image (3m11 job) ─▶ test reusable workflow (3m59 job)
                  build image 1m58 + upload 23s     ├ nextest jackin   3m59 job, idle ~3m first
                                                     ├ Docker E2E smoke 3m59 job, needs image
                                                     └ runtime/capsule/tui/small, idle ~3m first

clippy / check / fuzz / bench / msrv run in parallel and are NOT on the CI critical path.

All package shards started ~18:08:07, right after construct E2E image finished — pure unit tests should not wait for that Docker image.

Root cause of the unit/E2E coupling

In .github/workflows/ci.yml the single test job depends on construct-e2e-image and calls the reusable workflow in .github/workflows/rust-nextest.yml, which contains six jobs: five pure-Rust shards (jackin, jackin-capsule, jackin-runtime, jackin-tui, small-crates) and one docker-e2e. Only docker-e2e downloads the construct image, but GitHub Actions treats the reusable-workflow call as a single dependency-graph node, so all five package shards inherit the caller's needs: construct-e2e-image gate.

Cold-cache variance (run 28254775151, ~14m02s)

In that run Docker E2E smoke spent 1m59s building jackin-capsule and 4m09s running tests, because it uses its own target-cache namespace (ci-docker-e2e-dev-workspace-v1) separate from the package shards (ci-all-features-dev-workspace-v2-package-<group>). When that namespace is cold, Docker E2E becomes the long pole.

Detailed analysis per lever

Decouple package nextest from the Docker E2E image (Tier 2 #4)

Split the reusable workflow so the package shards depend only on [changes, cargo-registry-warmup], while a separate docker-e2e entrypoint keeps needs: construct-e2e-image. Unit tests then start immediately instead of after the ~3-minute image build.

Collapse the construct→E2E handoff (Tier 2 #5)

The image handoff costs ~45s in docker save → upload-artifact (23s) → download (11s) → docker load (11s). Either build the construct image inside the docker-e2e job (no inter-job gate, no artifact round-trip) or push to a registry / type=gha cache and pull. As a cheap interim step, set compression-level: 0 on the tar upload (the tar is already poorly compressible). Unify or seed the docker-e2e Rust cache with the package namespace so a cold namespace stops causing the 7m+ outlier.

nextest archive: build once, test many (Tier 1 #3)

Compile all workspace test binaries once with cargo nextest archive --workspace --all-features --archive-file nextest-archive.tar.zst, upload it, and have N partitioned run jobs do cargo nextest run --archive-file ... --partition count:i/N with no toolchain and no compilation. The archive bundles test binaries, dynamic libs, non-test binaries used by integration tests (so jackin-capsule is carried for the e2e lane), and one-level OUT_DIRs. This collapses 5–6 redundant compiles into one and structurally removes the cold per-shard build. Blocked first on making tests relocatable, because the earlier archive experiment failed a checkout-dependent capsule test.

sccache is half-configured (Tier 1 #2)

Every Rust job sets RUSTC_WRAPPER: sccache and SCCACHE_GHA_ENABLED: "off", so sccache only writes a local disk cache that rust-cache then archives — duplicating rust-cache and adding wrapper overhead, while the documented caveat that bin/proc-macro/cdylib crates cannot be cached still applies. Pick one coherent shape: enable a real remote backend (S3/R2/WebDAV/Depot Cache — not the rejected GHA backend) so the compiled-object cache survives Cargo.lock churn across all jobs and branches; or drop sccache entirely and rely on rust-cache. With the archive approach this only matters in the single build job. Also: sccache can slow release builds by up to 50%, so never wrap the release/publish path.

Feature-unification waste (Tier 3 #6)

Every Rust job runs --all-features. The jackin nextest shard therefore compiles dind_e2e.rs (1348 lines) and the whole e2e graph via --all-features even though the default nextest profile excludes those tests at runtime; docker-e2e then compiles them again. Use targeted feature sets per shard. Separately, per-crate -p shards get divergent transitive feature unification, so the shared cache cannot be reused cleanly across shards — the concrete need that would justify cargo-hakari.

Static test-impact analysis (Tier 3 #7)

Cargo has no native test-impact analysis. The repo already runs dorny/paths-filter in the changes job; extend it to per-crate-shard granularity using the cargo metadata reverse-dependency closure: a PR touching only crates/jackin-tui/ runs the jackin-tui shard plus its dependents and skips the rest. Fail safe — changes to Cargo.lock, root Cargo.toml, .cargo/, .github/, rust-toolchain.toml, mise.toml, or a shared root crate (jackin-core, jackin-config, jackin-protocol) force the full matrix. Reverse-dependency reality from cargo metadata today: a jackin-core change reaches almost the whole workspace (little to gain); jackin-term reaches only jackin-term + jackin-capsule (good candidate); leaf binaries (jackin-dev, jackin-pr-trailers, jackin-xtask, jackin-tui-lookbook) are strong candidates. Use it first as an early fast signal with full CI as the authoritative gate; once #3 lands, its compile-savings shrink and the residual value is skipping the archive build entirely on non-Rust PRs.

Research appendix

Real-world data: Depot's Zed benchmark

Depot benchmarked optimizations on Zed (a very large Rust codebase). Ranked by measured effect, this recalibrates priorities:

Two caveats from the same source: sccache can slow release builds up to 50% (wrap test/check only), and raw core scaling is dominated by cache warmth (a cold 16-core run lost to a warm 8-core one).

Test-impact analysis tiers

• Tier 1 — static (path filters + crate reverse-deps): cheapest, half-built via the changes job, deterministic. Recommended.
• Tier 2 — dynamic (cargo-difftests, per-test LLVM-coverage diff selection): becomes cheap once cargo-llvm-cov lands (already on the Rust CI tooling roadmap); reserve for expensive suites such as a future cargo-mutants gate.
• Tier 3 — fine-grained build system (Buck2/Bazel): gold standard but migration tax unjustified at 20 crates; note both disable rustc incremental compilation for hermetic remote builds.

Other knobs

--locked and CARGO_INCREMENTAL=0 are already set everywhere (good). [profile.dev] debug = 1 is already line-tables level; dropping test-binary debuginfo to 0 would shrink the nextest archive and speed linking at the cost of test backtraces. cargo-chef does not apply — docker/construct/Dockerfile installs system/mise packages and the only Rust compile (shellfirm) is already prebuilt and cached. Keep cargo-hack feature powersets on scheduled hygiene, not the PR hot path. Use nextest run recording (JUnit, per-test duration, priority, threads-required, test groups) to schedule slow/failing tests first for faster red-path signal; do not use --rerun as the green correctness gate, since it is not test-impact analysis.

Host-side effects

None on the operator's machine. All work is CI configuration plus root tool/config files such as .github/workflows/ci.yml, .github/workflows/rust-nextest.yml, and .cargo/config.toml. Any remote sccache backend or third-party runner is an opt-in infrastructure decision with its own credentials.