# CI/CD performance: prioritized speedup backlog (https://jackin.tailrocks.com/reference/research/ci/performance/ci-performance-analysis/)
**Status**: Open — analysis and prioritized backlog. This item is the measured follow-up companion to the [CI/CD speed roadmap](/reference/research/ci/performance/ci-speed-roadmap/), which tracks the shipped baseline. It ranks the remaining high-leverage levers from highly likely to less likely, grounded in job-level and step-level evidence from the merged PR #641 check suite (June 2026) rather than opinion.
## Problem [#problem]
The PR #641 check suite was slow for two separate reasons, and a deeper analysis of where time actually goes points at one dominant cost the existing speed work has not yet eliminated.
1. **`jackin-dev` was the overall long pole** (\~11m53s) because `build aarch64-apple-darwin (GitHub)` spent **10m40s inside `jdx/mise-action`** before the Rust build even started. The compile itself took 24s.
2. **`CI` has a structural serialization** where, when `docker/construct/**` changes, all pure-Rust nextest package shards wait behind the E2E construct-image build even though only the Docker E2E shard uses that image — adding \~3 minutes of idle time to unit-test shards.
The single fact that should drive prioritization: **the dominant CI cost is not running tests — it is compiling the same \~741-crate dependency graph from cold, roughly ten times in parallel, on most runs.** Measured on run [28256244365](https://github.com/jackin-project/jackin/actions/runs/28256244365):
* The `jackin` nextest shard spent **3m02 compiling and \~12s running** its 719 tests (each ≈0.01s). Compilation is \~94% of the job.
* The same cold full compile recurs in `clippy` (2m29), `check-default` (2m16), `msrv` (2m19), `bench-build`, `fuzz`, each of the five nextest shards, `docker-e2e`, and inside the construct image — and **none of these jobs share compiled objects**, because every Rust job sets `SCCACHE_GHA_ENABLED: "off"`.
* The cache goes cold because `Swatinem/rust-cache`'s key includes the hash, and changed in \~48% of recent commits (PR #641 included). `docker/construct/**` changed in \~43% of recent commits, so the E2E serialization hits nearly half of all PRs.
So the biggest levers either **compile fewer times** or **make the compiler cache survive churn**. Everything else is secondary. Do not pursue any of this by removing PR/main parity: the workflow-authoring rules in .github/AGENTS.md require every push-to-main failure invariant to be evaluated at PR time against the same inputs.
## Relationship to the CI/CD speed roadmap [#relationship-to-the-cicd-speed-roadmap]
The [CI/CD speed roadmap](/reference/research/ci/performance/ci-speed-roadmap/) already shipped the baseline (per-lane `cargo-registry-warmup`, semantic per-shard `rust-cache` keys, the reusable nextest workflow with a path-routed Docker E2E lane, the dual-runner GitHub/Velnor parity proof, and timing/cache summaries). It also already recorded outcomes that constrain the recommendations below — this item does not re-propose them naively:
* **Hosted GHA `sccache` was measured and rejected** (0% hits with cache write errors). The corrected baseline keeps `RUSTC_WRAPPER=sccache` for Cargo fingerprint/target-cache compatibility but disables the GHA backend with `SCCACHE_GHA_ENABLED=off`. So "turn sccache on" below means a real remote backend (S3/R2/WebDAV/Depot Cache), never the GHA backend again.
* **A `cargo nextest archive` fan-out experiment built successfully but failed an existing checkout-dependent test** and was reverted. So the build-once/test-many lever below is gated on first making tests relocatable (rely on runtime `CARGO_MANIFEST_DIR` / nextest binary env vars, not baked absolute paths).
* **`Swatinem/rust-cache` does not dependably cache workspace-crate outputs across jobs** — confirmed when the Docker E2E lane restored the shared exact key yet still recompiled `jackin-capsule` (\~7s) and the `docker-e2e` test graph (\~32s). This is why the archive handoff, not another target-cache family, is the remaining candidate for removing duplicate workspace compilation.
This item also intersects [Rust CI tooling and dependency hygiene](/reference/research/ci/rust-tooling/rust-ci-tooling/) (which is adopting `cargo-llvm-cov`, the prerequisite for dynamic test selection) and lists `cargo-hakari` and `cargo-chef` as open.
## Master priority: highly likely to less likely [#master-priority-highly-likely-to-less-likely]
Ordered by **(impact × confidence × breadth ÷ risk)**. Apply top-down; stop once a warm run is under \~4 minutes and cold outliers are gone, because past that complexity outweighs the marginal second saved.
### Tier 1 — highly likely, biggest result, do first [#tier-1--highly-likely-biggest-result-do-first]
| # | Change | Why it wins | Impact | Confidence | Effort |
| - | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- | --------------------------------------- | ------- |
| 1 | Fix the `jackin-dev` aarch64-macOS `mise-action` stall | 10m40s in tool install before the build starts; it is the overall suite long pole (\~11m53s) | up to **−10 min** on affected runs | High | Low–Med |
| 2 | A real shared compiler cache that survives churn (remote `sccache` S3/R2/WebDAV, not the rejected GHA backend) so a two-crate lock bump recompiles two crates, not 741 | Attacks the root cause across all \~10 cold-compiling jobs at once, including the lint wave | **−1.5 to −2.5 min on every cold job**, broadest win | Med | Med |
| 3 | nextest archive (build-once / test-many) — after tests are made relocatable | Collapses 5–6 redundant cold full compiles (\~3m each) into one; run shards drop to seconds | **−3 to −4 min** wall on the test wave; removes the cold 7m38 spike | Med–High (blocked on relocatable tests) | Med |
Items 2 and 3 are complementary: 2 makes each compile cheaper; 3 makes there be only one of them.
### Tier 2 — likely, strong result, do next [#tier-2--likely-strong-result-do-next]
| # | Change | Why | Impact | Confidence | Effort |
| - | --------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- | ---------- | ------- |
| 4 | Decouple unit nextest from the construct-E2E image | Only `docker-e2e` needs the image, but all five unit shards inherit `needs: construct-e2e-image`; hits \~43% of PRs | **−3 min** on construct-touching PRs | High | Low |
| 5 | Collapse the construct→E2E artifact handoff and unify/seed the `docker-e2e` cache | Removes save→upload→download→load (\~45s) and the separate cold cache that causes \~14-min outliers | **−45s steady**; eliminates \~14-min cold outliers | Med–High | Low–Med |
### Tier 3 — plausible, moderate result, measure while applying [#tier-3--plausible-moderate-result-measure-while-applying]
| # | Change | Why | Impact | Confidence | Effort |
| - | ----------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ---------- | --------------- |
| 6 | Drop blanket `--all-features`; use targeted per-shard features | The non-e2e `jackin` shard compiles `dind_e2e.rs` (1348 lines) and the whole `e2e` graph under `--all-features`, then `docker-e2e` compiles it again | **−20–40s/shard**, cleaner caches | Med | Low–Med |
| 7 | Static test-impact analysis: skip untouched crate shards; skip the archive build on non-Rust PRs | Extends the existing `changes` / `paths-filter` job to per-shard granularity with safe fallbacks | **−1 full compile per skipped shard** — but largely superseded by #3; residual value is skipping the archive build on non-Rust PRs | Med | Med |
| 8 | Faster runners for compile-bound jobs (free `ubuntu-24.04-arm`, the existing `velnor` lane, or third-party) | Compilation is CPU-bound; Zed data shows test exec −50% on 16 cores | **−20–40%** on compile-bound jobs | Med | Low–Med (infra) |
### Tier 4 — lower likelihood, diminishing, or measure-first [#tier-4--lower-likelihood-diminishing-or-measure-first]
| # | Change | Verdict |
| -- | -------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 9 | `cargo-hakari` (workspace-hack) | Only if feature-duplication measurement proves cross-shard cache divergence; partly mooted once #3 lands. Already open on the Rust CI tooling roadmap. |
| 10 | `mold` linker (x86\_64-scoped) | Depot's Zed benchmark showed only **−0.7%** — not link-bound. Measure before investing; do not assume a win. |
| 11 | Nightly parallel frontend (`-Z threads`, `-Z share-generics`) | −22.7% build on Zed, but adds toolchain risk; run as a measured CI-only experiment, never on the ship path. |
| 12 | Micro-opts: `upload-artifact` `compression-level: 0` for tarballs, per-platform Buildx cache exports, slow-test-first scheduling | Small and safe; do opportunistically. |
| 13 | Cranelift codegen backend | Failed to compile Zed (inline asm). Avoid as a drop-in. |
| 14 | Bazel / Buck2 / Nix | True target-level test-impact analysis plus remote cache, but the migration tax is unjustified at 20 crates with a solo maintainer. Reconsider only if Cargo tuning plateaus (\~100+ crates). |
### Sequencing rationale [#sequencing-rationale]
Do **1** first (largest single number, isolated to one job). Do **2 + 3** together next — they are the structural fix for the "compile ten times cold" root cause and benefit every subsequent run. **4 + 5** are cheap, high-frequency wins layered on top. Only then evaluate **6–8** with real before/after numbers, and treat **9–14** as experiments gated on measurement.
## Evidence [#evidence]
### Workflow-level timing (PR #641 final suite) [#workflow-level-timing-pr-641-final-suite]
| Workflow | Run | Wall time | Long pole | Notes |
| ----------------- | -------------------------------------------------------------------------------- | --------- | --------------------------------------------------- | -------------------------------------------------------------- |
| `jackin-dev` | [28257109705](https://github.com/jackin-project/jackin/actions/runs/28257109705) | \~11m53s | `build aarch64-apple-darwin` 11m40s | Overall suite long pole. Actual build only 24s; setup stalled. |
| `CI` | [28257109946](https://github.com/jackin-project/jackin/actions/runs/28257109946) | \~7m54s | `Docker E2E smoke` 4m23s after construct-image gate | Unit shards started only after the construct image finished. |
| `Construct Image` | [28257109672](https://github.com/jackin-project/jackin/actions/runs/28257109672) | \~4m43s | `arm64 (GitHub)` 3m40s | Normal PR rehearsal path. |
| `Docs` | [28257109767](https://github.com/jackin-project/jackin/actions/runs/28257109767) | \~3m00s | `docs-link-check` 2m42s | Not the main problem. |
### `jackin-dev` aarch64 macOS setup stall [#jackin-dev-aarch64-macos-setup-stall]
The target matrix builds four release artifacts in parallel. Three finished tool setup in 14–16s; one took 10m40s.
| Job | Wall time | Dominant step |
| ------------------------------------------ | --------- | ------------------------------------ |
| `build aarch64-apple-darwin (GitHub)` | 11m40s | `jdx/mise-action`: 10m40s |
| `build x86_64-apple-darwin (GitHub)` | 1m13s | `jdx/mise-action`: 15s, `Build`: 18s |
| `build x86_64-unknown-linux-gnu (GitHub)` | 1m00s | `jdx/mise-action`: 16s, `Build`: 16s |
| `build aarch64-unknown-linux-gnu (GitHub)` | 59s | `jdx/mise-action`: 14s, `Build`: 18s |
This is not compile time — the compile was fast. The time disappeared in tool setup, which points at an intermittent setup/cache/network/cache-save contention problem, not inherent build complexity. All four matrix jobs install the same set (`cargo-binstall rust zig cargo:cargo-zigbuild cosign syft cargo:sccache`) with `cache_key_prefix: "mise-v2"`.
Levers: instrument `mise-action` (per-tool grouping or `log_level: debug`) so the next stall names the exact tool; set an explicit `cache_key` and `cache_save: false` on matrix jobs after a single tool-prewarm job so the four targets never race to save the same tool cache; reassess whether PR jobs need `cosign`/`syft` on every target (keep them only where artifact attestation is a required main invariant); optionally run release-artifact builds on the `velnor` lane. Expected win: `jackin-dev` drops from \~11m53s to roughly \~1.5–2 min on warm setup.
### CI critical path (warm reference run [28256244365](https://github.com/jackin-project/jackin/actions/runs/28256244365), \~8m26s) [#ci-critical-path-warm-reference-run-28256244365-8m26s]
```text
changes(8s) ─▶ construct-e2e-image (3m11 job) ─▶ test reusable workflow (3m59 job)
build image 1m58 + upload 23s ├ nextest jackin 3m59 job, idle ~3m first
├ Docker E2E smoke 3m59 job, needs image
└ runtime/capsule/tui/small, idle ~3m first
clippy / check / fuzz / bench / msrv run in parallel and are NOT on the CI critical path.
```
| Time (UTC) | Job | Duration |
| ----------------- | ------------------------------------- | -------- |
| 18:04:22–18:04:30 | `changes` | 8s |
| 18:04:51–18:07:37 | `cargo check default` | 2m46s |
| 18:04:51–18:07:51 | `cargo clippy` | 3m00s |
| 18:04:52–18:09:02 | `cargo bench build` | 4m10s |
| 18:04:53–18:08:04 | `construct E2E image` | 3m11s |
| 18:08:07–18:12:06 | `test / Docker E2E smoke` | 3m59s |
| 18:08:07–18:12:06 | `test / cargo nextest jackin` | 3m59s |
| 18:08:07–18:11:12 | `test / cargo nextest jackin-capsule` | 3m05s |
| 18:08:07–18:11:06 | `test / cargo nextest jackin-runtime` | 2m59s |
| 18:08:06–18:09:23 | `test / cargo nextest small-crates` | 1m17s |
| 18:08:07–18:08:51 | `test / cargo nextest jackin-tui` | 44s |
All package shards started \~18:08:07, right after `construct E2E image` finished — pure unit tests should not wait for that Docker image.
### Root cause of the unit/E2E coupling [#root-cause-of-the-unite2e-coupling]
In .github/workflows/ci.yml the single `test` job depends on `construct-e2e-image` and calls the reusable workflow in .github/workflows/rust-nextest.yml, which contains six jobs: five pure-Rust shards (`jackin`, `jackin-capsule`, `jackin-runtime`, `jackin-tui`, `small-crates`) and one `docker-e2e`. Only `docker-e2e` downloads the construct image, but GitHub Actions treats the reusable-workflow call as a single dependency-graph node, so all five package shards inherit the caller's `needs: construct-e2e-image` gate.
### Cold-cache variance (run [28254775151](https://github.com/jackin-project/jackin/actions/runs/28254775151), \~14m02s) [#cold-cache-variance-run-28254775151-14m02s]
| Job | Duration |
| ----------------------------- | -------- |
| `construct E2E image` | 1m33s |
| `test / Docker E2E smoke` | 7m38s |
| `test / cargo nextest jackin` | 4m21s |
In that run `Docker E2E smoke` spent 1m59s building `jackin-capsule` and 4m09s running tests, because it uses its own target-cache namespace (`ci-docker-e2e-dev-workspace-v1`) separate from the package shards (`ci-all-features-dev-workspace-v2-package-`). When that namespace is cold, Docker E2E becomes the long pole.
## Detailed analysis per lever [#detailed-analysis-per-lever]
### Decouple package nextest from the Docker E2E image (Tier 2 #4) [#decouple-package-nextest-from-the-docker-e2e-image-tier-2-4]
Split the reusable workflow so the package shards depend only on `[changes, cargo-registry-warmup]`, while a separate `docker-e2e` entrypoint keeps `needs: construct-e2e-image`. Unit tests then start immediately instead of after the \~3-minute image build.
### Collapse the construct→E2E handoff (Tier 2 #5) [#collapse-the-constructe2e-handoff-tier-2-5]
The image handoff costs \~45s in `docker save` → `upload-artifact` (23s) → `download` (11s) → `docker load` (11s). Either build the construct image inside the `docker-e2e` job (no inter-job gate, no artifact round-trip) or push to a registry / `type=gha` cache and pull. As a cheap interim step, set `compression-level: 0` on the tar upload (the tar is already poorly compressible). Unify or seed the `docker-e2e` Rust cache with the package namespace so a cold namespace stops causing the 7m+ outlier.
### nextest archive: build once, test many (Tier 1 #3) [#nextest-archive-build-once-test-many-tier-1-3]
Compile all workspace test binaries once with `cargo nextest archive --workspace --all-features --archive-file nextest-archive.tar.zst`, upload it, and have N partitioned run jobs do `cargo nextest run --archive-file ... --partition count:i/N` with no toolchain and no compilation. The archive bundles test binaries, dynamic libs, non-test binaries used by integration tests (so `jackin-capsule` is carried for the e2e lane), and one-level `OUT_DIR`s. This collapses 5–6 redundant compiles into one and structurally removes the cold per-shard build. Blocked first on making tests relocatable, because the earlier archive experiment failed a checkout-dependent capsule test.
### sccache is half-configured (Tier 1 #2) [#sccache-is-half-configured-tier-1-2]
Every Rust job sets `RUSTC_WRAPPER: sccache` and `SCCACHE_GHA_ENABLED: "off"`, so sccache only writes a local disk cache that `rust-cache` then archives — duplicating `rust-cache` and adding wrapper overhead, while the documented caveat that bin/proc-macro/cdylib crates cannot be cached still applies. Pick one coherent shape: enable a real remote backend (S3/R2/WebDAV/Depot Cache — not the rejected GHA backend) so the compiled-object cache survives churn across all jobs and branches; or drop sccache entirely and rely on `rust-cache`. With the archive approach this only matters in the single build job. Also: sccache can slow release builds by up to 50%, so never wrap the release/publish path.
### Feature-unification waste (Tier 3 #6) [#feature-unification-waste-tier-3-6]
Every Rust job runs `--all-features`. The `jackin` nextest shard therefore compiles `dind_e2e.rs` (1348 lines) and the whole `e2e` graph via `--all-features` even though the default nextest profile excludes those tests at runtime; `docker-e2e` then compiles them again. Use targeted feature sets per shard. Separately, per-crate `-p` shards get divergent transitive feature unification, so the shared cache cannot be reused cleanly across shards — the concrete need that would justify `cargo-hakari`.
### Static test-impact analysis (Tier 3 #7) [#static-test-impact-analysis-tier-3-7]
Cargo has no native test-impact analysis. The repo already runs `dorny/paths-filter` in the `changes` job; extend it to per-crate-shard granularity using the `cargo metadata` reverse-dependency closure: a PR touching only `crates/jackin-tui/` runs the `jackin-tui` shard plus its dependents and skips the rest. Fail safe — changes to , root , `.cargo/`, `.github/`, `rust-toolchain.toml`, , or a shared root crate (`jackin-core`, `jackin-config`, `jackin-protocol`) force the full matrix. Reverse-dependency reality from `cargo metadata` today: a `jackin-core` change reaches almost the whole workspace (little to gain); `jackin-term` reaches only `jackin-term` + `jackin-capsule` (good candidate); leaf binaries (`jackin-dev`, `jackin-pr-trailers`, `jackin-xtask`, `jackin-tui-lookbook`) are strong candidates. Use it first as an early fast signal with full CI as the authoritative gate; once #3 lands, its compile-savings shrink and the residual value is skipping the archive build entirely on non-Rust PRs.
## Research appendix [#research-appendix]
### Real-world data: Depot's Zed benchmark [#real-world-data-depots-zed-benchmark]
Depot benchmarked optimizations on Zed (a very large Rust codebase). Ranked by measured effect, this recalibrates priorities:
| Optimization | Measured effect on Zed | Takeaway |
| ---------------------------------------------- | --------------------------------------------- | -------------------------------------------------------------- |
| `cargo-nextest` + warm sccache | −35% (biggest single win) | Make the cache actually warm and shared |
| Nightly `-Z threads=8` + `-Z share-generics=y` | −7.3% total, −22.7% build | A CI-only nightly build toolchain is a real lever; ship stable |
| More cores (8→16) | test exec −50%, build slower (cache variance) | Cores help the test phase; pair with archive split |
| Warm Cargo registry cache | −8% | Already shipped (`cargo-registry-warmup`) |
| `mold` linker | −0.7% (negligible) | Not a guaranteed win; measure before investing |
| Cranelift codegen backend | failed to compile (inline asm) | Avoid as a drop-in |
Two caveats from the same source: sccache can slow release builds up to 50% (wrap test/check only), and raw core scaling is dominated by cache warmth (a cold 16-core run lost to a warm 8-core one).
### Test-impact analysis tiers [#test-impact-analysis-tiers]
* **Tier 1 — static** (path filters + crate reverse-deps): cheapest, half-built via the `changes` job, deterministic. Recommended.
* **Tier 2 — dynamic** ([cargo-difftests](https://github.com/dnbln/cargo-difftests), per-test LLVM-coverage diff selection): becomes cheap once `cargo-llvm-cov` lands (already on the Rust CI tooling roadmap); reserve for expensive suites such as a future `cargo-mutants` gate.
* **Tier 3 — fine-grained build system** (Buck2/Bazel): gold standard but migration tax unjustified at 20 crates; note both disable rustc incremental compilation for hermetic remote builds.
### Other knobs [#other-knobs]
`--locked` and `CARGO_INCREMENTAL=0` are already set everywhere (good). `[profile.dev] debug = 1` is already line-tables level; dropping test-binary debuginfo to `0` would shrink the nextest archive and speed linking at the cost of test backtraces. `cargo-chef` does not apply — docker/construct/Dockerfile installs system/mise packages and the only Rust compile (`shellfirm`) is already prebuilt and cached. Keep `cargo-hack` feature powersets on scheduled hygiene, not the PR hot path. Use nextest run recording (JUnit, per-test duration, priority, `threads-required`, test groups) to schedule slow/failing tests first for faster red-path signal; do not use `--rerun` as the green correctness gate, since it is not test-impact analysis.
## Host-side effects [#host-side-effects]
None on the operator's machine. All work is CI configuration plus root tool/config files such as .github/workflows/ci.yml, .github/workflows/rust-nextest.yml, and .cargo/config.toml. Any remote `sccache` backend or third-party runner is an opt-in infrastructure decision with its own credentials.