01 — Goals and non-goals

Summary

The context engine should reduce tokens automatically at ingestion and retrieval boundaries while keeping every lossy transform reversible, measurable, and cache-safe.

Question and scope

Which goals, exclusions, and evidence bars must constrain a jackin❯ context engine?

Method

Evidence bars (every feature, no exceptions)

• Adoption bar (from techniques 16 — Validation Harness: the No-Quality-Loss Proof Protocol + tools hub 07 — Evidence, benchmarks, and the claim graveyard): task/test success ≥ baseline (non-inferiority, 5pp margin, n=12 screening / n=30 confirmation); cache_read continuity preserved; command re-run / bounce rate not worse; total tokens per solved task ≤ baseline by ≥20% net of the feature's own overhead (schema rent, injected instructions, retrieval round-trips, handles).
• Net-of-injection accounting: any standing prefix the engine injects is a first-class cost in its own ledger. lean-ctx's current documentation reports roughly 2.1K tokens of fixed session overhead; jackin❯ must measure and gate its own current footprint rather than inherit that number as a benchmark result.
• Canary gate: the six compression-failure canaries (negation, ordering, numeric precision, don't-do-X, multi-step, caveat retention — techniques 16 — Validation Harness: the No-Quality-Loss Proof Protocol §3) run in both arms of every A/B; one Arm-B-only failure vetoes the feature regardless of savings.
• Quality-risk features run behind holdouts: aggressive registers and effort downgrades ship with a headroom-style holdout control group and an online sampled canary (techniques 25 — Meta layer: the cost of optimizing, budget governance, and online quality guarding: alarm, don't block), so the "does register compression degrade agentic success" open question finally gets an answer on live traffic.

Findings

Goals

1. Out-of-box token optimization for every capsule. A freshly launched jackin❯ container is born with the negative-cost set active (tool-search deferral intact, Edit-over-Write guards, observation filtering, subagent model/effort pins) — zero operator discipline required. This is the dossier's K16 "jackin-baked optimization pack" (ESTIMATE 20–35% integrated) made concrete.
2. Cover both interception points. Write-time compression must reach shell output (RTK's slot) and native reads (headroom/lean-ctx's slot), because the measured Bash share of observation tokens ranged 16.5% (docs workload) to dominant (test-heavy workloads). One engine, both boundaries, exactly one shell-rewrite path per capsule.
3. Reversible by default. Every lossy transform leaves a content-addressed verbatim copy retrievable by handle on all exit codes — fixing RTK's success-path gap and adopting headroom-CCR/lean-ctx-archive semantics. Recovery hints ride on every compressed payload (lean-ctx #625 doctrine: agents fight irreversible compression; undiscoverable escape hatches are the root cause of "too compressed" failures).
4. Deterministic core, no ML in the hot path. Typed deterministic transforms (filters, crushers, outlines, extractive prose scoring) only. This is now the converged industry position: headroom itself is demoting its ModernBERT stage after timeout cascades, and CompressionAttack (arXiv 2510.22963, ≤80% ASR) turned hot-path ML compressors into a security finding, not just a latency cost.
5. Cache-safety by construction. All input-side work happens at content-ingestion time (append-only), never by rewriting history or the cached prefix. The engine ships the prefix-stability regression tests headroom pioneered and surfaces cache_read continuity as a monitored invariant.
6. Retrieval before reading. Bounded, task-shaped retrieval (outline → section/symbol → span) replaces whole-file reads only when a current implementation-specific benchmark shows a net win; small files remain whole. Current alternatives are tracked in the rising and retrieval-tools comparison.
7. Fleet-aware economics. Spawn staggering, prefix pre-warm, excludeDynamicSections, batch-lane routing, effort tiering, and advisor escalation are orchestrator-enforced. Dual cost model: dollars and tasks-per-cap, selected by the capsule's auth mode.
8. Prove it or lose it. Every layer reports through one bounce-netted, Claude-tokenizer-honest ledger; features that fail the acceptance bar (below) get disabled per role, automatically. The engine is its own validation harness.
9. Agent-agnostic by layer. ~80% of the stack ports as discipline across Claude Code/Codex/other runtimes (techniques 23 — Cross-agent / cross-provider portability matrix); the engine abstracts effort tier, routing, rules-file compilation, output caps, and cache-stability per Agent adapter, and degrades gracefully where a runtime lacks a hook surface (Codex: instructions-only).

Implications for jackin❯

These findings define the goals and non-goals contract for jackin❯; any implementation must preserve the acceptance gates and boundaries above.

Limitations and unknowns

Non-goals (the graveyard, enforced)

Each entry below was killed by the research with evidence; the engine treats these as tripwires in review, not options:

Sources

Evidence and repository references are cited inline beside the claims they support.

Related work

• Context engine dossier
• Token-optimization techniques
• Token-optimization tools