# 03 — Observation pipeline (L2) (https://jackin.tailrocks.com/research/context/engine/03-observation-pipeline/)



## Summary [#summary]

Observations should be classified and reduced once, before first cache admission, with a content-addressed original available for recovery on every outcome.
The write-time layer shrinks an observation **once, at ingestion, before it is first cached**. Current design references are [Headroom v0.34.0](/research/context/tools/02-headroom-design/), [RTK v0.45.0](/research/context/tools/03-rtk-design/), [lean-ctx v3.9.18](/research/context/tools/04-leanctx-design/), and [Caveman v2.0.0](/research/context/tools/01-caveman-design/).

## Question and scope [#question-and-scope]

Which runtime observations should the engine collect, normalize, filter, and expose without leaking secrets or flooding context?

## Method [#method]

The design traces current runtime signals from collection through normalization and filtering, using repository interfaces and the dossier’s measurement rules to bound payload size, freshness, and secret exposure.

## Findings [#findings]

### Pipeline [#pipeline]

```text
 shim / MCP read / web fetch                    (ingestion point)
   │
   ▼
 CLASSIFY   current reviewed RTK-derived classifier OR content detector
   │        (magika ONNX tier optional → unidiff parse-oracle → aho-corasick keywords)
   ▼
 POLICY     verbatim-class gate: build/lint errors, test failures, security warnings,
   │        explicitly-requested raw reads → PASS THROUGH untouched (INV-7)
   ▼
 TRANSFORM  exactly one typed transform, deterministic:
   │          reviewed declarative + native streaming shell filters
   │          json/tabular/yaml crushers (stats-compaction, sampled arrays, anomalies kept)
   │          code outline (tree-sitter signatures; never regex+brace-counting)
   │          search-result shaping (file:line:content, dedup, import-line de-noise)
   │          log dedup/grouping (template clustering, counts)
   │          markdown section slicing (mq; see L1)
   │          extractive prose scoring (BM25+recency+salience; TextCrusher-class, no ML)
   ▼
 GUARD      never_worse: estimated tokens(out) ≤ tokens(raw) or emit raw (INV-3)
   │        ParseResult tiers: Full / Degraded(+warnings) / Passthrough with marker
   ▼
 ARCHIVE    verbatim original → CCR store (blake3, TTL ≥30 min), all exit codes
   │        emission carries recovery footer:  [full output: ctx expand <<ccr:HASH>>]
   ▼
 LEDGER     tokens_in/out, transform id, per-block manifest → measure/ (L0)
```

### Shell boundary (shims) [#shell-boundary-shims]

* **Coverage**: review RTK v0.45.0's current declarative corpus and port only high-traffic filters proven by the local reach audit. The declarative pipeline is the primary extension form; project-local filters are content-hash trusted before loading.
* **Classifier**: review and port RTK v0.45.0's current classifier table + shell lexer: split on `&&`/`||`/`;` and rewrite each segment; anything with backticks, `$( )`, or file redirects is unattestable → passthrough. Ignore-list for shell built-ins and one-liners.
* **Safety**: exit codes bit-exact (INV-4); stderr filtered separately; signal exits propagate 128+n; package-manager sniffing before JS tool rewrites.
* **What shims buy over hooks**: identical behavior for every runtime (Claude Code, Codex, Amp, Kimi, Opencode, Grok) with zero per-agent integration code, no settings.json surface, no permission-prompt mediation (the agent's permission system sees the original command; the shim is transparent downstream of approval).

### Native-read boundary (MCP verbs + optional PostToolUse dedup) [#native-read-boundary-mcp-verbs--optional-posttooluse-dedup]

* `ctx_read` with `full` / `signatures` / `outline` / `lines:N-M` / `section:<heading>` modes. Compressed modes remain experimental until current local task evidence establishes a size gate and non-inferiority.
* **Read-stub cache**: content-addressed re-read suppression for unchanged files; changed files return a bounded delta or full content. Scoping is per conversation and subagent, with purge on compaction. Exact output sizes and fallback thresholds are benchmark outputs, not inherited constants.
* **Steering, honestly**: role instructions advertise the verbs and when to prefer them; native tools stay available and (where the runtime supports PostToolUse output mutation) get transparent read-dedup so even un-steered reads benefit. No deny-hooks, no "always use X" absolutism — codedb's blocking installer hook and lean-ctx's abandoned "NEVER use native tools" rulebook are the documented anti-patterns.

### Code outlining [#code-outlining]

tree-sitter only; no regex-and-brace approximation. The grammar set is **per role** so the image links only required languages. Output uses a declaration skeleton, explicit omission count, and an escalation footer naming the exact follow-up read.

### Structured-data crushers [#structured-data-crushers]

Deterministic, byte-stable, never-inflate: JSON schema/statistics with disclosed sampling, CSV/TSV sampling with counts, and YAML/TOML minification only where round-trip lossless. Current token effects are measured with the target provider tokenizer; no percentage is inherited from another tool or format benchmark.

### Media policy ([techniques 20 — Multimodal token economics: images, screenshots, PDFs](/research/context/techniques/20-multimodal-token-economics/)) [#media-policy-techniques-20--multimodal-token-economics-images-screenshots-pdfs]

Text-first enforcement at the boundary: born-digital PDFs transcode to text via `pdftotext` baked in the construct image (\~2× PDF tax avoided; 25-page PDF ≈ 78.8k tokens as pages vs \~40k as text); screenshots of textual content are rejected with a pointer to the text source when one exists (2–6× cost); images client-side downsampled to the model family's pixel cap before send (up to −73% on Opus-family caps); vision work routes to a Sonnet/Haiku-tier subagent returning text (−67%/frame, the \~3.0–3.1× per-image cap divergence). Text-to-image re-encoding of textual context — [current rising-tools comparison](/research/context/tools/06-combining/) — inverts this rule; the engine keeps text-first as the default and treats modality re-encoding as a harness-gated experiment only, because byte-exact content must never live only in pixels (silent-confabulation failure shape) and the economics depend on undocumented provider vision pricing.

### Prose (the residual) [#prose-the-residual]

No ML in the default path. Any extractive prose mode must be CCR-recoverable and pass a current buried-answer/non-inferiority test. Web/HTML first converts to structured Markdown, then selects sections instead of stripping tags heuristically.

### Guard contracts adopted from the current source verification [#guard-contracts-adopted-from-the-current-source-verification]

The pipeline requires these contracts; each port is revalidated against current upstream source and local tests:

* **Anti-confusion truncation**: exactly one `[N more lines]` end marker per truncated payload, never interleaved omission markers (agents parse those as code and retry-loop); the kept+N==total invariant is property-tested (RTK).
* **`unless` anti-swallow guards**: any short-circuit summary rule ("N tests passed") carries an `unless` regex so error-bearing output is never swallowed by the happy-path summary (RTK TOML DSL).
* **Per-filter panic isolation**: every filter runs under `catch_unwind`; panic → warn + raw passthrough — fail-open in both the crash and quality dimensions when combined with the never-worse guard (RTK).
* **`transparent_prefixes`**: wrapper commands (`docker exec c`, `poetry run`, `direnv exec .`) are stripped before classification and re-prepended after, so wrappers never defeat filtering (RTK).
* **Content-sniffing pipe mode**: `cmd | ctx pipe` fingerprints the first 1 KiB (banner-anchored to avoid misroutes) and auto-selects a filter — zero-flag usage for role scripts (RTK).
* **Trust-before-load for role/project filter files**: SHA-256 content pinning, single read returning the verified bytes (TOCTOU-safe), fail-closed on canonicalization errors, CI-only env override (RTK `trust.rs` — the threat model is an attacker-committed filter shaping what the model sees).
* **Lossless-compaction tier with a runtime round-trip self-check**: format-native reversible transforms (run-collapse with exact counts, grep-result reshaping, ANSI strip) verify `inverse(compressed) == original` at runtime and emit the original on any mismatch — a store-free tier below CCR (headroom `--lossless`).
* **Kolmogorov gate**: a gzip-ratio probe classifies payload compressibility before attempting aggressive modes — never spend effort on incompressible content (lean-ctx).
* **Per-tool line budgets** as defense-in-depth behind typed filters, with thresholds derived from the current local workload.
* **`anchored` read mode**: verbatim content with per-line hash anchors, making compressed-read → patch workflows drift-safe (lean-ctx; excluded from bounce accounting as lossless).
* **Preserved-segment span classification** for any prose transform: fences, inline code, URLs, paths, versions, numbers, and identifiers pass byte-exact through every lossy path.

## Implications for jackin❯ [#implications-for-jackin]

These findings define the observation pipeline (l2) contract for jackin❯; any implementation must preserve the acceptance gates and boundaries above.

## Limitations and unknowns [#limitations-and-unknowns]

### Explicit exclusions [#explicit-exclusions]

No whole-prompt rewriting, history pruning, or conversation compaction in the default engine. It measures incremental benefit over current native agent features, not over a naive baseline ([current objective comparison](/research/context/tools/05-head-to-head/)).

## Sources [#sources]

Evidence and repository references are cited inline beside the claims they support.

## Related work [#related-work]

* [Context engine dossier](/research/context/engine/)
* [Token-optimization techniques](/research/context/techniques/)
* [Token-optimization tools](/research/context/tools/)