03 — Observation pipeline (L2)

Design for classifying, filtering, compressing, and preserving agent observations before they enter cached context.

Summary

Observations should be classified and reduced once, before first cache admission, with a content-addressed original available for recovery on every outcome. The write-time layer shrinks an observation once, at ingestion, before it is first cached. Current design references are Headroom v0.34.0, RTK v0.45.0, lean-ctx v3.9.18, and Caveman v2.0.0.

Question and scope

Which runtime observations should the engine collect, normalize, filter, and expose without leaking secrets or flooding context?

Method

The design traces current runtime signals from collection through normalization and filtering, using repository interfaces and the dossier’s measurement rules to bound payload size, freshness, and secret exposure.

Findings

Pipeline

 shim / MCP read / web fetch                    (ingestion point)
   │
   ▼
 CLASSIFY   current reviewed RTK-derived classifier OR content detector
   │        (magika ONNX tier optional → unidiff parse-oracle → aho-corasick keywords)
   ▼
 POLICY     verbatim-class gate: build/lint errors, test failures, security warnings,
   │        explicitly-requested raw reads → PASS THROUGH untouched (INV-7)
   ▼
 TRANSFORM  exactly one typed transform, deterministic:
   │          reviewed declarative + native streaming shell filters
   │          json/tabular/yaml crushers (stats-compaction, sampled arrays, anomalies kept)
   │          code outline (tree-sitter signatures; never regex+brace-counting)
   │          search-result shaping (file:line:content, dedup, import-line de-noise)
   │          log dedup/grouping (template clustering, counts)
   │          markdown section slicing (mq; see L1)
   │          extractive prose scoring (BM25+recency+salience; TextCrusher-class, no ML)
   ▼
 GUARD      never_worse: estimated tokens(out) ≤ tokens(raw) or emit raw (INV-3)
   │        ParseResult tiers: Full / Degraded(+warnings) / Passthrough with marker
   ▼
 ARCHIVE    verbatim original → CCR store (blake3, TTL ≥30 min), all exit codes
   │        emission carries recovery footer:  [full output: ctx expand <<ccr:HASH>>]
   ▼
 LEDGER     tokens_in/out, transform id, per-block manifest → measure/ (L0)

Shell boundary (shims)

Coverage: review RTK v0.45.0's current declarative corpus and port only high-traffic filters proven by the local reach audit. The declarative pipeline is the primary extension form; project-local filters are content-hash trusted before loading.
Classifier: review and port RTK v0.45.0's current classifier table + shell lexer: split on &&/||/; and rewrite each segment; anything with backticks, $( ), or file redirects is unattestable → passthrough. Ignore-list for shell built-ins and one-liners.
Safety: exit codes bit-exact (INV-4); stderr filtered separately; signal exits propagate 128+n; package-manager sniffing before JS tool rewrites.
What shims buy over hooks: identical behavior for every runtime (Claude Code, Codex, Amp, Kimi, Opencode, Grok) with zero per-agent integration code, no settings.json surface, no permission-prompt mediation (the agent's permission system sees the original command; the shim is transparent downstream of approval).

Native-read boundary (MCP verbs + optional PostToolUse dedup)

ctx_read with full / signatures / outline / lines:N-M / section:<heading> modes. Compressed modes remain experimental until current local task evidence establishes a size gate and non-inferiority.
Read-stub cache: content-addressed re-read suppression for unchanged files; changed files return a bounded delta or full content. Scoping is per conversation and subagent, with purge on compaction. Exact output sizes and fallback thresholds are benchmark outputs, not inherited constants.
Steering, honestly: role instructions advertise the verbs and when to prefer them; native tools stay available and (where the runtime supports PostToolUse output mutation) get transparent read-dedup so even un-steered reads benefit. No deny-hooks, no "always use X" absolutism — codedb's blocking installer hook and lean-ctx's abandoned "NEVER use native tools" rulebook are the documented anti-patterns.

Code outlining

tree-sitter only; no regex-and-brace approximation. The grammar set is per role so the image links only required languages. Output uses a declaration skeleton, explicit omission count, and an escalation footer naming the exact follow-up read.

Structured-data crushers

Deterministic, byte-stable, never-inflate: JSON schema/statistics with disclosed sampling, CSV/TSV sampling with counts, and YAML/TOML minification only where round-trip lossless. Current token effects are measured with the target provider tokenizer; no percentage is inherited from another tool or format benchmark.

Media policy (techniques 20 — Multimodal token economics: images, screenshots, PDFs)

Text-first enforcement at the boundary: born-digital PDFs transcode to text via pdftotext baked in the construct image (~2× PDF tax avoided; 25-page PDF ≈ 78.8k tokens as pages vs ~40k as text); screenshots of textual content are rejected with a pointer to the text source when one exists (2–6× cost); images client-side downsampled to the model family's pixel cap before send (up to −73% on Opus-family caps); vision work routes to a Sonnet/Haiku-tier subagent returning text (−67%/frame, the ~3.0–3.1× per-image cap divergence). Text-to-image re-encoding of textual context — current rising-tools comparison — inverts this rule; the engine keeps text-first as the default and treats modality re-encoding as a harness-gated experiment only, because byte-exact content must never live only in pixels (silent-confabulation failure shape) and the economics depend on undocumented provider vision pricing.

Prose (the residual)

No ML in the default path. Any extractive prose mode must be CCR-recoverable and pass a current buried-answer/non-inferiority test. Web/HTML first converts to structured Markdown, then selects sections instead of stripping tags heuristically.

Guard contracts adopted from the current source verification

The pipeline requires these contracts; each port is revalidated against current upstream source and local tests:

Anti-confusion truncation: exactly one [N more lines] end marker per truncated payload, never interleaved omission markers (agents parse those as code and retry-loop); the kept+N==total invariant is property-tested (RTK).
unless anti-swallow guards: any short-circuit summary rule ("N tests passed") carries an unless regex so error-bearing output is never swallowed by the happy-path summary (RTK TOML DSL).
Per-filter panic isolation: every filter runs under catch_unwind; panic → warn + raw passthrough — fail-open in both the crash and quality dimensions when combined with the never-worse guard (RTK).
transparent_prefixes: wrapper commands (docker exec c, poetry run, direnv exec .) are stripped before classification and re-prepended after, so wrappers never defeat filtering (RTK).
Content-sniffing pipe mode: cmd | ctx pipe fingerprints the first 1 KiB (banner-anchored to avoid misroutes) and auto-selects a filter — zero-flag usage for role scripts (RTK).
Trust-before-load for role/project filter files: SHA-256 content pinning, single read returning the verified bytes (TOCTOU-safe), fail-closed on canonicalization errors, CI-only env override (RTK trust.rs — the threat model is an attacker-committed filter shaping what the model sees).
Lossless-compaction tier with a runtime round-trip self-check: format-native reversible transforms (run-collapse with exact counts, grep-result reshaping, ANSI strip) verify inverse(compressed) == original at runtime and emit the original on any mismatch — a store-free tier below CCR (headroom --lossless).
Kolmogorov gate: a gzip-ratio probe classifies payload compressibility before attempting aggressive modes — never spend effort on incompressible content (lean-ctx).
Per-tool line budgets as defense-in-depth behind typed filters, with thresholds derived from the current local workload.
anchored read mode: verbatim content with per-line hash anchors, making compressed-read → patch workflows drift-safe (lean-ctx; excluded from bounce accounting as lossless).
Preserved-segment span classification for any prose transform: fences, inline code, URLs, paths, versions, numbers, and identifiers pass byte-exact through every lossy path.