jackin❯ Context Engine — design spec

This folder is the product specification distilled from two research corpora — the token-optimization-techniques dossier (Volumes I–III, 160+ technique records, per-file verification ledgers) and the token-optimization tools hub (source-level teardowns of caveman, headroom, RTK, lean-ctx) — plus a dedicated reuse-surface sweep of seven external tools (sem, fff, rtk, lean-ctx, codedb, mq, headroom) recorded in 13 — Reuse surfaces and deltas and 25 — sem and mq, first run 2026-07-03 and re-run 2026-07-05. The 2026-07-05 re-research also feeds the spec: the latest-versions refresh (per-tool delta ledger across the whole hub), the pxpipe teardown (the one new tool that cleared the coverage bar), and the techniques drift delta (provider/docs drift against live primary sources). It specifies one engine, embedded in jackin, that takes the best-evidenced mechanism from each tool and drops every cost those tools pay for not owning the runtime.

The one-paragraph thesis

Every external token-optimization tool fights the same three battles: it must install itself into an agent environment it does not own (hooks, settings.json rewrites, rule files, self-healing drift checkers — lean-ctx spends an entire subsystem and 19 per-agent installers on this; RTK spends a 6,894-line installer; caveman collides with RTK over ~/.claude), it must guess what the agent is doing from the outside (RTK cannot see native Read; headroom's proxy must reverse-engineer cache breakpoints), and it must prove its own savings with someone else's tokenizer (all four teardown tools count Claude traffic with GPT tokenizers or chars/4). jackin❯ owns the capsule, generates the agent configuration, supervises the process, and already parses per-session usage JSONL — so an embedded engine gets each tool's core mechanism without its delivery tax: RTK's write-time shell compression without hook installers, headroom's live-zone and reversible-store designs without a proxy in front of Claude Code, lean-ctx's read-stub cache and honest ledger without a 19-installer monolith, sem's entity layer and attention-ledger idea generalized across all tool traffic, fff's resident search as a crate instead of a per-capsule binary, codedb's fused query verbs without a 5.2k-token MCP schema, mq's structural markdown retrieval for the prose-read bucket nothing else touches, and caveman's output register as role-generated policy instead of a plugin.

What the research says the engine must target

The dossier's measured invariants bound the design (all sourced in 01 — economics, 02 — baseline audit, 10 — first-party measurements):

• Dollar split (measured heavy session): cache reads 32% / cache writes 29% / thinking ~20% / visible output ~17% / uncached 2%. Token volume is 94% cache reads. 1 output token = 5 uncached input = 50 cache-read tokens.
• The defaults already bank ~4–5× (prompt caching −86.3% input-side, MCP schema deferral, Edit-diffs). The engine's job is the residual: protect the banked wins, then compress the write-time observation stream, then route.
• Honest ceiling: ≈2.5× defensible at zero quality loss; ≈5–6.2× if the model-routing flip passes the harness (Claude Sonnet 5 uses the new Opus-4.7+ tokenizer, so the compounding tokenizer discount in the downtier arithmetic applies only when pinning legacy Sonnet 4.6 / Haiku 4.5); no honest 10× — thinking (20% of dollars) is reachable only by effort/routing, and the cache-read floor of genuinely-used context is real.
• Workload split decides the lever: on this repo's docs-heavy transcripts, 76.2% of observation tokens came through native Read (RTK-blind) and only 16.5% through Bash. A code-heavy session inverts this. The engine must cover both interception points and measure its own split per capsule.
• For subscribers the metric flips to tasks-per-cap (cache reads ≈0.1× against the cap, T3): prefix stability and request volume outrank everything; style compression is demoted. The engine meters both objectives.

The six layers

        L5  MEMORY & STATE          one reversible store; handoff files; net-accounted injection
        L4  ROUTING & FLEET         effort tiers, model pinning, advisor, stagger/pre-warm,
                                    batch lane, quota/cap metering, auth-mode policy
        L3  OUTPUT & PROMPT         register pack, guard lines, report registers, codebook
        L2  OBSERVATION PIPELINE    write-time compression at the tool boundary:
                                    shims + filters + crushers + outlines + handles + CCR archive
        L1  RETRIEVAL               resident search (fff), entity/structure (sem-core, tree-sitter),
                                    markdown sections (mq), fused query verbs (codedb-shaped)
        L0  MEASUREMENT             usage ledger, cache-bust detector, thinking estimator,
                                    harness runner, canaries, CI token-budget linter

L0 is the foundation, not an afterthought: the single loudest finding across both research corpora is that nobody's numbers survive contact with a controlled harness (caveman 75%→58.5% — upstream itself retired the 75% headline for a "65% average" in v1.9.1, headroom 60–95%→median 4.8% whole-session, RTK 60–90%→never measured whole-session, lean-ctx 99%→code-reads-only, and lean-ctx's first independent benchmark found its default config taxing the workload). The engine therefore ships measurement first and treats every compression feature as unproven until its own ledger — bounce-netted, counted against Claude tokenizers via count_tokens, net of the engine's own injection overhead — says otherwise.

Provenance: what the engine takes from each tool

Structural advantages no external tool can copy

1. No host writes, no installers, no drift. The engine's hooks, shims, MCP registration, and instruction files are composed at capsule build/launch inside /jackin/ — the entire installer/self-healing subsystem class disappears, and the host-write ban is satisfied by construction.
2. Enforcement instead of discipline. The adoption roadmap's "automatic beats disciplined" split becomes total: effort tiers, subagent model pins, report registers, and session-boundary routing are generated into agent config per role, not remembered by operators.
3. Whole-boundary visibility. The capsule supervisor sees every tool result (native reads via MCP tools it registers, shell via shims, web via its own fetchers), so the attention ledger, bounce netting, and reach measurement cover 100% of observation traffic — the number RTK/headroom/lean-ctx each only estimate for their slice.
4. Fleet-level economics. Staggered spawn, pre-warm, excludeDynamicSections, workspace-pinned cache lanes, batch-lane scheduling, and per-capsule cap attribution are orchestrator features — none of the seven tools has an orchestrator.
5. Honest self-measurement built in. crates/jackin-usage already parses per-agent session JSONL into token-class totals; the engine extends this into the harness the research defined but no tool ships.

How to read this folder