# 08 — Measurement and verification (L0) (https://jackin.tailrocks.com/reference/research/jackin-context-engine/08-measurement-and-verification/)



# 08 — Measurement and verification (L0) [#08--measurement-and-verification-l0]

The engine's differentiator is not a cleverer compressor — the market has plenty — it is being the first tool whose numbers are **produced the way the research says numbers must be produced**. L0 ships before any compression does, and every other layer reports through it.

## The ledger [#the-ledger]

Per-capsule, turso-backed, extending the existing `jackin-usage` monitors:

* **Token classes, exact**: per-call `usage` fields from session JSONL (`message.id`-deduplicated — the \~3× overcount trap), split uncached / cache-write (5m vs 1h TTL) / cache-read / output, per agent, per subagent (`task:` scope), per tool.
* **Observation attribution**: tool-result sizes attributed to producing tool (the measurement that produced the 76.2%-native-reads / 16.5%-Bash split) — continuously, per capsule, so RTK-slot vs read-slot coverage is a live number, not a one-off study.
* **Transform accounting**: per-transform tokens\_in/out counted with **Claude tokenizers via the free `count_tokens` endpoint** (batched, own RPM pool, never inline) — not `chars/4` (RTK), not GPT BPE (lean-ctx/caveman/headroom). Family-correct: counts against the capsule's actual model; never ported across tokenizer families.
* **Bounce netting**: a compressed read followed by a full re-read of the same content within a short window writes a *negative* ledger event; per-extension bounce rates above threshold auto-upgrade that class to fuller modes (lean-ctx's `adjusted_total_saved` + auto-pin, adopted wholesale). Command re-runs after filtered output are the shell-side bounce equivalent.
* **Injection overhead as cost**: every standing token the engine adds (instructions, schemas, codebook, `[related:]` hints, recovery footers) is debited — savings are reported **net-of-injection** (the tokbench lesson), and `ctx doctor --gate` fails CI when the standing footprint exceeds the role budget (INV-6).
* Optional hash-chained event log for tamper-evidence (lean-ctx's signed ledger) — off by default, available for fleet billing contexts.

## The estimators nobody ships [#the-estimators-nobody-ships]

* **Thinking estimator**: JSONL redacts thinking; the engine computes `output_tokens − count_tokens(visible blocks replayed)` per message (±5%), closing the blind spot every JSONL dashboard has and bounding the register pack's honest ceiling per workload.
* **Cache-bust detector**: `cache_creation_input_tokens` spike detection with cause classification (model/effort switch, tool-set change, MCP flap, compaction, fleet misalignment) and per-bust cost ($2.30–3.80 at 200k history class) — the invalidator list from [techniques 07 — Caching exploitation](/reference/research/token-optimization-techniques/07-caching-exploitation/) as running diagnostics, surfaced via `clog!` and the usage dialog.
* **Counterfactual repricing**: what this session would have cost uncached / unrouted / unfiltered — the honest "saved" number, reconciled against `session_cost.py`-style ground truth within 5% before any A/B is trusted. Convergent prior art: [pxpipe](/reference/research/token-optimization-tools/17-pxpipe-design/) ships per-request `count_tokens` counterfactual accounting with a same-cache-state baseline and unfloored negative rows — the strongest self-measurement in the surveyed tool field, and the same design this ledger specifies.
* **Cap prober** (subscription capsules): fits per-account cap weights from captured `unified-*` headers (L4), converting tasks-per-cap from folklore to a fitted model.

## The harness ([techniques 16 — Validation Harness: the No-Quality-Loss Proof Protocol](/reference/research/token-optimization-techniques/16-validation-harness/), runnable per role) [#the-harness-techniques-16--validation-harness-the-no-quality-loss-proof-protocol-runnable-per-role]

`jackin ctx bench` drives the paired-task A/B protocol: headless agent runs (`claude -p --output-format json` class), fixture repo reset between tasks, arms differing in exactly one engine feature (or the composed pack validated as a unit), n=12 screening / n=30 confirmation, non-inferiority margin 5pp on objective checkers, cost compared only between quality-indistinguishable arms; batch-lane execution (50% off + cache stacking makes confirmation runs cheap). Standard arms: measure-only baseline / +shims / +read-verbs / +register / +routing / full pack — plus external-tool comparison arms (RTK binary, headroom-MCP, lean-ctx faithful-arm profile) so the engine's claim of subsuming the stack is a measured statement, closing the "first controlled cross-tool numbers" gap the tools hub named its highest-value open deliverable.

## Round-2 estimator upgrades [#round-2-estimator-upgrades]

* **Conversation-stable holdout with online variance**: behavioral features (registers, effort shaping) measure themselves against a control cohort keyed by `blake3(system ‖ first user msg) mod 10,000` — same conversation, same arm, across turns and machines (prefix-cache-safe) — with sum-of-squares online variance giving a CI without storing per-turn data (headroom/lean-ctx convergent design). The engine adds what both upstreams lack: an **auto-disable gate when the CI lower bound crosses zero**.
* **Footprint self-ablation**: each element the engine injects (rules block, tool schemas, codebook, briefing) gets periodically A/B-ablated for pass-rate delta, emitting `prune_recommended` per element (lean-ctx #959) — "does my own injection earn its tokens" as a running answer, not an assumption.
* **Failed-experiments ledger**: negative results are archived with their metrics next to the harness results (codedb's `failed.md` practice) — prevents re-trying dead ends and keeps the tuning history honest.

## Canaries and online quality [#canaries-and-online-quality]

The six compression-failure canaries (negation, ordering, numeric precision, don't-do-X, multi-step dependency, caveat retention) run in every screening/confirmation round with exact string-level assertions; any Arm-B-only failure vetoes. On live traffic: sampled async reference-free judge (1–10% high-volume, alarm-don't-block) watching register-active roles for dropped caveats, wired to per-role auto-demotion (V7). Regression tripwires seeded with known-bad traces.

## CI token-budget linter (K14) [#ci-token-budget-linter-k14]

`cargo xtask` gains a context-budget check: `count_tokens` over every always-loaded file (root instruction chain, role files, engine-injected pack) with per-file budgets; PRs fail on over-budget growth — \~30 lines of tooling that prevents the \~70k-tokens-in-5-days prefix-drift class and the "bloated CLAUDE.md causes instruction-ignoring" quality failure. Budgets counted against the worst-case (Fable-family) tokenizer.

## Surfaces [#surfaces]

Capsule TUI usage dialog and status bar gain the engine panel (4-class token bars with 5m/1h write split, cache hit rate + bust events, net savings, bounce rate, cap %); host console aggregates per-fleet; everything also lands in `clog!`/JSONL diagnostics for offline analysis. No web dashboard, no external telemetry (INV-10).