16 — Validation Harness: the No-Quality-Loss Proof Protocol

Summary

This file is a runnable protocol, not a literature review. Any technique or stack in this dossier earns "no quality loss" only by passing this harness.

• Paired-task A/B: same ≥12 tasks per arm (baseline vs technique), headless claude -p runs, objective pass/fail checks, usage read from the JSON result envelope.
• Decision rule: non-inferiority on success rate with a 5-percentage-point margin at n=12 (screening) or n=30 (confirmation); cost compared only between arms with statistically indistinguishable success.
• Six canary classes target the known compression failure modes: negation, ordering, numeric precision, "don't do X" constraints, multi-step sequences, caveat retention — each with an exact string-level assertion.
• Tokens by class come from usage in claude -p --output-format json (plus OTel or session JSONL for per-call decomposition); dollars are computed with the live price table in 01 — Token Economics and Measurement.
• A technique is NEGATIVE-COST only if it wins cost and its success rate is ≥ baseline at n≥30 — the bar folklore numbers never clear.

Question and scope

What paired-task protocol can prove lower token use with non-inferior quality, bounded latency, and reliable rollback?

Method

The protocol runs paired baseline and candidate tasks from identical repository state, records provider usage and latency, applies objective acceptance checks, and rejects any quality regression.

Findings

1. Design

Two arms, identical except for the technique under test:

• Arm A (baseline): default Claude Code settings, no style compression, default context habits, fixed model + effort.
• Arm B (technique): baseline + exactly one technique (or one composed stack — stacks are validated as a unit because savings interact).

Run all tasks in both arms, fresh session per task (no cross-task contamination), same repo fixture state per task (reset via git between runs). Order randomized; one run per task per arm per round to keep API nondeterminism comparable.

Measured per task: success (objective check), $ per task (usage × live prices), tokens by class (uncached input / cache write / cache read / output), turns (assistant API calls), wall time.

2. Task suite (n = 12, objective checks)

Tasks are repo-generic; instantiate against any pinned fixture repo (for this repo, pin a commit and use a worktree). Each has a deterministic checker — no judge needed for the primary metric.

Success = checker exit 0. Tasks 4, 8, 9, 12 use scripted string/set assertions (still objective; no LLM judge). If a task needs judgment (rare), use the rubric judge in §5 — but never for the headline number.

3. Canary tasks (compression failure modes)

Run in both arms; any Arm-B-only failure is a quality regression regardless of the task suite score. Exact assertions:

Rationale: these are precisely what register compression (caveman/wenyan), terse-output prompts, summarization, and aggressive pruning are most likely to silently drop — short connectives, negations, and qualifiers carry disproportionate meaning per token (04 — Style and Language Compression — Beyond Caveman).

4. Runner (concrete)

Headless driver; one invocation per task per arm. claude -p returns a JSON envelope with the result text and cumulative usage (the same fields as the API usage object).

#!/usr/bin/env bash
# run_arm.sh <arm-name> <settings-dir> <tasks.tsv> <out.jsonl>
set -euo pipefail
ARM=$1; SETTINGS=$2; TASKS=$3; OUT=$4
while IFS=$'\t' read -r ID PROMPT CHECK; do
 git -C fixture reset --hard "$FIXTURE_SHA" -q
 R=$(CLAUDE_CONFIG_DIR="$SETTINGS" claude -p "$PROMPT" \
 --output-format json --max-turns 40 --cwd fixture 2>/dev/null)
 PASS=0; (cd fixture && bash -c "$CHECK") && PASS=1
 jq -nc --arg id "$ID" --arg arm "$ARM" --argjson pass "$PASS" \
 --argjson r "$R" \
 '{id:$id, arm:$arm, pass:$pass,
 usage:$r.usage, cost_usd:$r.total_cost_usd,
 turns:$r.num_turns, duration_ms:$r.duration_ms}' >> "$OUT"
done < "$TASKS"

Technique arms are realized as a separate CLAUDE_CONFIG_DIR (own settings.json, hooks, CLAUDE.md, plugins) so arms cannot leak into each other. For techniques that are session habits rather than config (e.g. "/clear between tasks"), encode the habit in the driver, not the prompt.

Dollars: prefer total_cost_usd from the envelope; recompute from usage × the price table (01 — Token Economics and Measurement §1) as a cross-check. Per-call class decomposition when needed: enable OTel (claude_code.token.usage by type) or parse the session JSONL deduplicating by message.id (trap documented in 02 — Baseline audit of the research environment §1).

5. Judging

• Primary: objective checkers only (above). Tests pass, diff correct, string assertions.
• Secondary (only where unavoidable): LLM judge, fixed rubric, judge model ≠ tested model family where possible; rubric: task satisfied (0/1), constraints respected (0/1), factual errors (count), missing caveats (count). Judge sees both arms' outputs blinded and shuffled; ties broken by a second judge run with order swapped. Judge verdicts never override an objective checker.

6. The statistical bar for "no quality loss"

Success rates p_A, p_B over n tasks/arm. Declare no quality loss when Arm B is non-inferior with margin δ = 5 percentage points:

• Screening (n = 12): require successes_B ≥ successes_A − 1 AND zero canary regressions. Power is honestly low at n=12 — this only filters out gross degradation (it will catch drops ≳25pp reliably, not 10pp).
• Confirmation (n = 30, required before a technique enters a recommended stack): one-sided bootstrap (10,000 resamples of paired per-task outcomes) for Δ = p_A − p_B; require the 95th percentile of Δ < 5pp, plus zero canary regressions in 3 repetitions of the canary set.
• Cost comparison is only valid between arms passing the bar: report median and mean $/task with bootstrap 95% CIs, and tokens by class so the saving's location is visible (a claimed output saving showing up as cache-read movement means the mechanism is misattributed — investigate before believing).

Paired bootstrap (tasks are paired across arms) in ~15 lines:

import json, random, sys
rows = [json.loads(l) for l in open(sys.argv[1])]
A = {r['id']: r for r in rows if r['arm']=='A'}
B = {r['id']: r for r in rows if r['arm']=='B'}
ids = sorted(set(A) & set(B)); n = len(ids)
deltas = []
for _ in range(10_000):
 s = [random.choice(ids) for _ in range(n)]
 deltas.append(sum(A[i]['pass'] for i in s)/n - sum(B[i]['pass'] for i in s)/n)
deltas.sort
print(f"Δ success (A−B) p95 = {deltas[int(.95*10_000)]:.3f} (non-inferior if < 0.05)")
print(f"cost/task A median {sorted(A[i]['cost_usd'] for i in ids)[n//2]:.3f} "
 f"B median {sorted(B[i]['cost_usd'] for i in ids)[n//2]:.3f}")

7. Per-technique falsification experiments

Every technique record in files 04 — Style and Language Compression — Beyond Caveman through 14 — Frontier — unrealistic but maybe real names its falsification experiment; they all reduce to an instantiation of this harness:

• Style layers (caveman/wenyan): arms differ only in the style hook; canaries C1–C6 are the decisive part (register compression's failure mode is dropped qualifiers, not wrong code).
• Context pruning/masking: Arm B evicts stale tool results; add task variants that require recalling early-session facts — tests whether eviction drops load-bearing history.
• Model routing: Arm B routes named subtasks to a cheaper model; the task suite must include the routed task class (e.g. tasks 4, 5, 8 for Haiku exploration).
• Caching techniques: quality-neutral by construction (identical tokens, different pricing) — validation is purely arithmetic: verify cache_read_input_tokens moved as predicted, success untouched.
• Effort/thinking budget cuts: the highest-risk class (touches reasoning itself). Confirmation n=30 mandatory; include tasks 1, 9, 11, 12 (the reasoning-heavy ones).

8. Where to read the numbers (recap)

Implications for jackin❯

For jackin❯, the validation harness: the no-quality-loss proof protocol evidence identifies which mechanisms are safe to adopt directly and which still require workload-specific validation.

Limitations and unknowns

Quantitative conclusions are workload- and model-specific. Re-measure volatile provider inputs and validate transfers on representative jackin❯ tasks.

Sources

Related work

• Token-optimization techniques dossier
• Economics and measurement
• Context engine