# 08 — Measurement and verification (L0) (https://jackin.tailrocks.com/research/context/engine/08-measurement-and-verification/)



## Summary [#summary]

Measurement must ship before compression, attribute every token class and observation source, and reject savings that fail paired quality and cache-continuity gates.
The engine's differentiator is not a cleverer compressor — the market has plenty — it is being the first tool whose numbers are **produced the way the research says numbers must be produced**. L0 ships before any compression does, and every other layer reports through it.

## Question and scope [#question-and-scope]

How should jackin❯ prove context savings while preserving task quality, latency, and failure recovery?

## Method [#method]

The verification model pairs baseline and candidate runs on identical tasks, reconciles provider usage with quality and latency outcomes, and requires explicit rollback thresholds before adoption.

## Findings [#findings]

### The ledger [#the-ledger]

Per-capsule, turso-backed, extending the existing `jackin-usage` monitors:

* **Token classes, exact**: per-call `usage` fields from session JSONL (`message.id`-deduplicated — the \~3× overcount trap), split uncached / cache-write (5m vs 1h TTL) / cache-read / output, per agent, per subagent (`task:` scope), per tool.
* **Observation attribution**: tool-result sizes attributed to producing tool (the measurement that produced the 76.2%-native-reads / 16.5%-Bash split) — continuously, per capsule, so RTK-slot vs read-slot coverage is a live number, not a one-off study.
* **Transform accounting**: per-transform tokens\_in/out counted with **Claude tokenizers via the free `count_tokens` endpoint** (batched, own RPM pool, never inline) — not `chars/4` (RTK), not GPT BPE (lean-ctx/caveman/headroom). Family-correct: counts against the capsule's actual model; never ported across tokenizer families.
* **Bounce netting**: a compressed read followed by a full re-read of the same content within a short window writes a *negative* ledger event; per-extension bounce rates above threshold auto-upgrade that class to fuller modes (lean-ctx's `adjusted_total_saved` + auto-pin, adopted wholesale). Command re-runs after filtered output are the shell-side bounce equivalent.
* **Injection overhead as cost**: every standing token the engine adds (instructions, schemas, codebook, `[related:]` hints, recovery footers) is debited — savings are reported **net-of-injection** (the tokbench lesson), and `ctx doctor --gate` fails CI when the standing footprint exceeds the role budget (INV-6).
* Optional hash-chained event log for tamper-evidence (lean-ctx's signed ledger) — off by default, available for fleet billing contexts.

### The estimators nobody ships [#the-estimators-nobody-ships]

* **Thinking estimator**: JSONL redacts thinking; the engine computes `output_tokens − count_tokens(visible blocks replayed)` per message (±5%), closing the blind spot every JSONL dashboard has and bounding the register pack's honest ceiling per workload.
* **Cache-bust detector**: `cache_creation_input_tokens` spike detection with cause classification (model/effort switch, tool-set change, MCP flap, compaction, fleet misalignment) and per-bust cost ($2.30–3.80 at 200k history class) — the invalidator list from [techniques 07 — Caching exploitation](/research/context/techniques/07-caching-exploitation/) as running diagnostics, surfaced via `clog!` and the usage dialog.
* **Counterfactual repricing**: what the measured workload would have cost uncached / unrouted / unfiltered, reconciled against provider usage before any A/B is trusted. Current representation tools expose useful counterfactual patterns, but jackin❯ validates the exact method independently ([current rising-tools comparison](/research/context/tools/06-combining/)).
* **Cap prober** (subscription capsules): fits per-account cap weights from captured `unified-*` headers (L4), converting tasks-per-cap from folklore to a fitted model.

### The harness ([techniques 16 — Validation Harness: the No-Quality-Loss Proof Protocol](/research/context/techniques/16-validation-harness/), runnable per role) [#the-harness-techniques-16--validation-harness-the-no-quality-loss-proof-protocol-runnable-per-role]

`jackin ctx bench` drives the paired-task A/B protocol: headless agent runs (`claude -p --output-format json` class), fixture repo reset between tasks, arms differing in exactly one engine feature (or the composed pack validated as a unit), n=12 screening / n=30 confirmation, non-inferiority margin 5pp on objective checkers, cost compared only between quality-indistinguishable arms; batch-lane execution (50% off + cache stacking makes confirmation runs cheap). Standard arms: measure-only baseline / +shims / +read-verbs / +register / +routing / full pack — plus external-tool comparison arms (RTK binary, headroom-MCP, lean-ctx faithful-arm profile) so the engine's claim of subsuming the stack is a measured statement, closing the "first controlled cross-tool numbers" gap the tools hub named its highest-value open deliverable.

### Estimator calibration [#estimator-calibration]

* **Conversation-stable holdout with online variance**: behavioral features (registers, effort shaping) measure themselves against a control cohort keyed by `blake3(system ‖ first user msg) mod 10,000` — same conversation, same arm, across turns and machines (prefix-cache-safe) — with sum-of-squares online variance giving a CI without storing per-turn data (headroom/lean-ctx convergent design). The engine adds what both upstreams lack: an **auto-disable gate when the CI lower bound crosses zero**.
* **Footprint self-ablation**: each element the engine injects (rules block, tool schemas, codebook, briefing) gets periodically A/B-ablated for pass-rate delta, emitting `prune_recommended` per element (lean-ctx #959) — "does my own injection earn its tokens" as a running answer, not an assumption.
* **Failed-experiments ledger**: negative results are archived with their metrics next to the harness results (codedb's `failed.md` practice) — prevents re-trying dead ends and keeps the tuning history honest.

### Canaries and online quality [#canaries-and-online-quality]

The six compression-failure canaries (negation, ordering, numeric precision, don't-do-X, multi-step dependency, caveat retention) run in every screening/confirmation round with exact string-level assertions; any Arm-B-only failure vetoes. On live traffic: sampled async reference-free judge (1–10% high-volume, alarm-don't-block) watching register-active roles for dropped caveats, wired to per-role auto-demotion (V7). Regression tripwires seeded with known-bad traces.

### CI token-budget linter (K14) [#ci-token-budget-linter-k14]

`cargo xtask` gains a context-budget check: `count_tokens` over every always-loaded file (root instruction chain, role files, engine-injected pack) with per-file budgets; PRs fail on over-budget growth — \~30 lines of tooling that prevents the \~70k-tokens-in-5-days prefix-drift class and the "bloated CLAUDE.md causes instruction-ignoring" quality failure. Budgets counted against the worst-case (Fable-family) tokenizer.

### Surfaces [#surfaces]

Capsule TUI usage dialog and status bar gain the engine panel (4-class token bars with 5m/1h write split, cache hit rate + bust events, net savings, bounce rate, cap %); host console aggregates per-fleet; everything also lands in `clog!`/JSONL diagnostics for offline analysis. No web dashboard, no external telemetry (INV-10).

## Implications for jackin❯ [#implications-for-jackin]

These findings define the measurement and verification (l0) contract for jackin❯; any implementation must preserve the acceptance gates and boundaries above.

## Limitations and unknowns [#limitations-and-unknowns]

The design remains bounded by the evidence and acceptance gates above; volatile dependencies and provider behavior require revalidation before implementation.

## Sources [#sources]

Evidence and repository references are cited inline beside the claims they support.

## Related work [#related-work]

* [Context engine dossier](/research/context/engine/)
* [Token-optimization techniques](/research/context/techniques/)
* [Token-optimization tools](/research/context/tools/)