ResearchTesting

Test infrastructure: shared test-support crate, behavioral specs, and parser fuzzing

Design rationale and phased proposal for three related testing gaps that block the codebase readability program: duplicated test helpers across mega-test-files, missing behavioral specs for major components, and no property/fuzz testing for parser surfaces. The actionable remainder of this work is tracked at Test infrastructure & behavioral specs; this page owns the "why" and the design alternatives.

Problem

Three testing gaps increase the risk of silent regressions and block the Phase 2 source-file splits.

Test file duplication mirrors source duplication. crates/jackin/tests/manager_flow.rs and crates/jackin/tests/dind_e2e.rs are each 1,500+ lines. Mock runners are independently defined in at least three places (crates/jackin/tests/common.rs's FakeRunner, crates/jackin-runtime/src/runtime/test_support.rs's FakeRunner, and crates/jackin/tests/per_mount_isolation_e2e.rs's ScriptedRunner), a config_with_agents-family helper is redefined per-crate (jackin-console's editor view tests, jackin-console's workspace tests, and jackin's console editor tests each carry their own copy), and ResolvedWorkspace { ... } is constructed inline across at least nine files spanning jackin-console, jackin, jackin-runtime, and jackin-config. This blocks the Phase 2 source-file splits: splitting a large source file means rewriting the large test file that depends on it, and every duplicated helper is a place the split has to touch twice.

Behavioral specs are incomplete. Specs exist for runtime/launch.rs and the console op_picker, but not for jackin-capsule's daemon (crates/jackin-capsule/src/daemon.rs — the in-container PID 1 control plane) or for the operator console's TUI state machine. Behavioral specs should be a general practice for every major component, not a one-off exercise per refactor.

No property-based or fuzz testing for parser surfaces. crates/jackin-manifest/src/validate.rs, crates/jackin-env/src/env_resolver.rs, and the migration chains (crates/jackin-config/src/migrations.rs, crates/jackin-manifest/src/migrations.rs) all parse operator- or role-author-supplied input, and are tested only by example-based unit tests. An input the developer didn't think of isn't covered. cargo-fuzz already exists for jackin-term (the damage_grid_process target, scheduled nightly in .github/workflows/hygiene.yml); no equivalent exists for any parser crate, and proptest is not part of the dependency graph at all.

Proposal

1. Test file split + mock consolidation

Rust projects that maintain large integration test suites (tokio, bevy, cargo) commonly extract a dedicated test-support crate rather than let helpers proliferate per test file. Apply the same pattern here:

  • Phase 1 — extract jackin-test-support. No such crate exists yet. Promote the duplicated FakeRunner/ScriptedRunner mocks, the config_with_agents family, ResolvedWorkspace builders, and role-seeding helpers into one crate under crates/jackin-test-support/, then update every test file to import from it instead of redefining. Expected outcome is a net deletion of duplicated code, not new functionality.
  • Phase 2 — split the mega-test-files. tests/manager_flow.rs into several files grouped by scenario family; tests/dind_e2e.rs into smaller end-to-end groups. This is a companion to each source-file Phase 2 split in the codebase readability program, and should follow the helper extraction so the split files don't re-duplicate mocks.
  • Phase 3 — snapshot/golden-file infrastructure. insta is already a dev-dependency in several crates (jackin-image, jackin-console, jackin, jackin-runtime, jackin-capsule) but is not standardized through a shared harness. Route it through the test-support crate so JSON output, derived Dockerfiles, and friendly error blocks can all be snapshotted with one helper, with a shared determinism harness (fixed size/theme/clock, redaction). The styled-SVG visual goldens for TUI screens and CLI output build on this crate but are designed separately in Visual snapshot testing (CLI & TUI): that item owns the artifact format, this item owns the shared jackin-test-support crate it would live in.

2. Behavioral specs for major components

Behavioral specs already exist for runtime/launch.rs (published at runtime/launch.rs Behavioral Spec) and for the console op_picker (published at op_picker Behavioral Spec), plus a differently-scoped spec for alternate agent credential/config folder sync (Auth Source-Folder Sync). The gap is the capsule daemon and the operator console:

  1. Capsule daemon (crates/jackin-capsule/src/daemon.rs) — PID 1 contract, single-attach-client invariant, control channel dispatch, session lifecycle, PTY mutex poison recovery, attach framing, OSC passthrough, mode-state restore, sessions persistence and reattach.
  2. Operator console — the TUI state machine: workspace selection to role selection to agent selection to instance lifecycle to session management, keybinding dispatch, dialog stack.

Each spec should list invariants, give the failure mode for each violation, and link to the source line where the invariant is enforced — the same INV-format shape used by the existing published specs under /reference/developer-reference/specs/.

3. Property and fuzz tests for parser surfaces

Two testing layers, neither of which exists today for the parser crates:

proptest for invariant testing:

  • env_resolver: ${env.VAR} interpolation is associative, idempotent on no-op substitutions, escapes correctly for every Unicode input.
  • manifest::validate: a manifest that parses successfully always round-trips through serialize to parse unchanged.
  • Migration chain: any input that parses at version N migrates successfully to N+1 and the resulting output parses at N+1.

cargo-fuzz targets, run alongside the existing jackin-term nightly job:

  • manifest::validate (TOML input, expect no panic).
  • env_resolver::resolve (any Unicode env string).
  • Each migration step (any input that parses at version N).

Non-goals

  • Do not block the readability program's Phase 1 work on the test-support crate; Phase 1 splits don't need it yet.
  • Do not add mutation testing yet. Re-evaluate once cargo-llvm-cov (from Rust CI tooling & dependency hygiene) shows where coverage is weakest.
  • Do not fuzz the TUI rendering layer or the Docker client — those surfaces change too fast for fuzzing to be cost-effective.

Phasing

  1. Audit every tests/*.rs file and inventory helpers and mock structs; research how tokio/bevy/cargo structure their test-support crates; create crates/jackin-test-support/ with dev-only dependencies from consumers' perspective (publish = false); promote one helper family at a time, mounts first; split tests/manager_flow.rs last, after helpers are extracted.
  2. Write the capsule daemon behavioral spec, then the console behavioral spec.
  3. Add proptest to dev-dependencies; add a fuzz/ directory with one cargo-fuzz target per parser; extend the existing nightly fuzz schedule to cover them.

Open design questions

  1. Which exact Rust projects should be studied for test-support crate patterns? (tokio, bevy, cargo, and clap are candidates.)
  2. Should the test-support crate live in crates/ or alongside the test files it serves?
  3. Should behavioral specs live in the roadmap or in contributor-facing reference docs? (The existing precedent — runtime-launch, op-picker, auth-source-folder-sync — is contributor-facing reference docs under /reference/developer-reference/specs/, with a thin roadmap item tracking the link-back task.)
  4. What is the right nightly CI schedule and time budget for the new parser fuzz targets, relative to the existing jackin-term target?

Visual snapshot testing (CLI & TUI)

Previous Page

Token-Optimization Techniques — Research Dossier

Next Page

On this page

ProblemProposal1. Test file split + mock consolidation2. Behavioral specs for major components3. Property and fuzz tests for parser surfacesNon-goalsPhasingOpen design questionsRelated work