# 01 — Goals and non-goals (https://jackin.tailrocks.com/research/context/engine/01-goals-and-non-goals/) ## Summary [#summary] The context engine should reduce tokens automatically at ingestion and retrieval boundaries while keeping every lossy transform reversible, measurable, and cache-safe. ## Question and scope [#question-and-scope] Which goals, exclusions, and evidence bars must constrain a jackin❯ context engine? ## Method [#method] ### Evidence bars (every feature, no exceptions) [#evidence-bars-every-feature-no-exceptions] * **Adoption bar** (from [techniques 16 — Validation Harness: the No-Quality-Loss Proof Protocol](/research/context/techniques/16-validation-harness/) + [tools hub 07 — Evidence, benchmarks, and the claim graveyard](/research/context/tools/07-evidence-and-claims/)): task/test success ≥ baseline (non-inferiority, 5pp margin, n=12 screening / n=30 confirmation); `cache_read` continuity preserved; command re-run / bounce rate not worse; **total tokens per solved task ≤ baseline by ≥20% net of the feature's own overhead** (schema rent, injected instructions, retrieval round-trips, handles). * **Net-of-injection accounting**: any standing prefix the engine injects is a first-class cost in its own ledger. lean-ctx's current documentation reports roughly 2.1K tokens of fixed session overhead; jackin❯ must measure and gate its own current footprint rather than inherit that number as a benchmark result. * **Canary gate**: the six compression-failure canaries (negation, ordering, numeric precision, don't-do-X, multi-step, caveat retention — [techniques 16 — Validation Harness: the No-Quality-Loss Proof Protocol](/research/context/techniques/16-validation-harness/) §3) run in both arms of every A/B; one Arm-B-only failure vetoes the feature regardless of savings. * **Quality-risk features run behind holdouts**: aggressive registers and effort downgrades ship with a headroom-style holdout control group and an online sampled canary ([techniques 25 — Meta layer: the cost of optimizing, budget governance, and online quality guarding](/research/context/techniques/25-meta-cost-governance-and-online-quality/): alarm, don't block), so the "does register compression degrade agentic success" open question finally gets an answer on live traffic. ## Findings [#findings] ### Goals [#goals] 1. **Out-of-box token optimization for every capsule.** A freshly launched jackin❯ container is born with the negative-cost set active (tool-search deferral intact, Edit-over-Write guards, observation filtering, subagent model/effort pins) — zero operator discipline required. This is the dossier's K16 "jackin-baked optimization pack" (ESTIMATE 20–35% integrated) made concrete. 2. **Cover both interception points.** Write-time compression must reach shell output (RTK's slot) *and* native reads (headroom/lean-ctx's slot), because the measured Bash share of observation tokens ranged 16.5% (docs workload) to dominant (test-heavy workloads). One engine, both boundaries, exactly one shell-rewrite path per capsule. 3. **Reversible by default.** Every lossy transform leaves a content-addressed verbatim copy retrievable by handle on *all* exit codes — fixing RTK's success-path gap and adopting headroom-CCR/lean-ctx-archive semantics. Recovery hints ride on every compressed payload (lean-ctx #625 doctrine: agents fight irreversible compression; undiscoverable escape hatches are the root cause of "too compressed" failures). 4. **Deterministic core, no ML in the hot path.** Typed deterministic transforms (filters, crushers, outlines, extractive prose scoring) only. This is now the *converged* industry position: headroom itself is demoting its ModernBERT stage after timeout cascades, and CompressionAttack (arXiv 2510.22963, ≤80% ASR) turned hot-path ML compressors into a security finding, not just a latency cost. 5. **Cache-safety by construction.** All input-side work happens at content-ingestion time (append-only), never by rewriting history or the cached prefix. The engine ships the prefix-stability regression tests headroom pioneered and surfaces `cache_read` continuity as a monitored invariant. 6. **Retrieval before reading.** Bounded, task-shaped retrieval (outline → section/symbol → span) replaces whole-file reads only when a current implementation-specific benchmark shows a net win; small files remain whole. Current alternatives are tracked in the [rising and retrieval-tools comparison](/research/context/tools/06-combining/). 7. **Fleet-aware economics.** Spawn staggering, prefix pre-warm, `excludeDynamicSections`, batch-lane routing, effort tiering, and advisor escalation are orchestrator-enforced. Dual cost model: dollars *and* tasks-per-cap, selected by the capsule's auth mode. 8. **Prove it or lose it.** Every layer reports through one bounce-netted, Claude-tokenizer-honest ledger; features that fail the acceptance bar (below) get disabled per role, automatically. The engine is its own validation harness. 9. **Agent-agnostic by layer.** \~80% of the stack ports as discipline across Claude Code/Codex/other runtimes ([techniques 23 — Cross-agent / cross-provider portability matrix](/research/context/techniques/23-cross-agent-portability/)); the engine abstracts effort tier, routing, rules-file compilation, output caps, and cache-stability per `Agent` adapter, and degrades gracefully where a runtime lacks a hook surface (Codex: instructions-only). ## Implications for jackin❯ [#implications-for-jackin] These findings define the goals and non-goals contract for jackin❯; any implementation must preserve the acceptance gates and boundaries above. ## Limitations and unknowns [#limitations-and-unknowns] ### Non-goals (the graveyard, enforced) [#non-goals-the-graveyard-enforced] Each entry below was killed by the research with evidence; the engine treats these as tripwires in review, not options: | Non-goal | Killed by | | -------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Whole-prompt / mid-stream proxy compression in front of an agent | It rewrites cacheable history and expands the trusted path. Any exception must pass the current bake-off and cache-continuity gates ([current comparison](/research/context/tools/05-head-to-head/)). | | Cache-aligned text-to-image re-encoding as a **default** | Model-specific legibility, lossy exact-text handling, and provider-dependent image accounting make pxpipe-class representation suitable only for an allowlisted current-model experiment ([current rising-tools comparison](/research/context/tools/06-combining/)). | | ML compression model in the hot path | CompressionAttack ≤80% ASR; headroom P99 4.17 s + executor-saturation outages; deterministic TextCrusher-class reaches \~94% retention (13-reuse ch.) | | API gateway that re-serializes requests | Custom `ANTHROPIC_BASE_URL` disables tool-search deferral; one nondeterministic serializer = broken cache = \~3.8× session cost ([techniques 06 — Context architecture — what enters the window at all](/research/context/techniques/06-context-architecture/), 13) | | Semantic response caching for coding traffic | Zero published coding hit rates; repetition lives in the prefix already priced 0.1×; 3–7% confidently-wrong on code ([techniques 08 — Retrieval, memory, and state offloading](/research/context/techniques/08-retrieval-memory-and-state/), 13) | | base64/gzip/glyph-DSL/wenyan "compression" | base64 = 4.33× MORE tokens; gzip+base64 = 2.68× MORE; glyphs 3.9–4.9 tok each with 26% instruction-fidelity; wenyan char/token illusion ([techniques 04 — Style and Language Compression — Beyond Caveman](/research/context/techniques/04-style-and-language-compression/), 05, 13) | | Cache-keepalive pinger on the interactive main loop | Subscription main loop already 1h-TTL; 1h break-even \~19 h idle ([techniques 14 — Frontier — unrealistic but maybe real](/research/context/techniques/14-frontier-ideas/) K13) — only SDK/API fleet lanes with sub-40-min gaps qualify | | Vector DB / embeddings for **code navigation** as default | Current retrieval alternatives still lack a release-specific result clearing jackin❯ acceptance gates over exact symbol/lexical tools ([current retrieval-tools comparison](/research/context/tools/06-combining/)). | | A second memory store, a second shell-rewrite path, a second output policy | Doubling rules from the combining page: pure overhead, double-compression risk ([tools hub 06 — Combining them: is there one product?](/research/context/tools/06-combining/)) | | Renaming existing identifiers for token shape | Cache bust + blame destruction; design-time-only lever ([techniques 05 — Tokenizer arbitrage](/research/context/techniques/05-tokenizer-arbitrage/)) | | KV-cache eviction/quantization, soft prompts, speculative decoding | Self-host-only; no hosted API channel ([techniques 24 — Fresh literature & market delta (clean-room re-sweep)](/research/context/techniques/24-fresh-literature-and-market-delta/), 14 K1/K2) | | Trusting per-payload percentages as savings | Per-payload ≠ whole-bill: the correction that killed every vendor headline ([tools hub 07 — Evidence, benchmarks, and the claim graveyard](/research/context/tools/07-evidence-and-claims/)) | | `max_tokens` as a spend governor | Truncates `tool_use`, bills the attempt, forces retry at higher cap ([techniques 09 — Output discipline and structured generation](/research/context/techniques/09-output-discipline/), 25) | ## Sources [#sources] Evidence and repository references are cited inline beside the claims they support. ## Related work [#related-work] * [Context engine dossier](/research/context/engine/) * [Token-optimization techniques](/research/context/techniques/) * [Token-optimization tools](/research/context/tools/)