04 — Retrieval layer (L1)

Summary

Bounded lexical, structural, semantic, and documentation retrieval should replace whole-file reads only when measured task evidence shows a net quality-preserving win. Prevention beats compression: tokens never loaded cost nothing and improve quality (context-rot literature; the dossier's negative-cost set is dominated by input-architecture levers). L1 gives the agent bounded, task-shaped retrieval so it stops reading whole files — the lever measured locally at −91% (outline vs read) and −98% (symbol search vs read), and 12–53× for markdown sections.

Question and scope

How should jackin❯ retrieve the smallest sufficient repository and memory context for each task?

Method

The analysis compares lexical, structural, and semantic retrieval against the repository’s scoped-rule and memory surfaces, then assigns each source a trigger, budget, provenance record, and fallback.

Findings

The four retrieval modalities and their engines

Compiler-grade truth stays with the existing role stack: rust-analyzer (LSP) and ast-grep (CLI) remain the routing targets for definitions/references and structural rewrites — the engine's verbs complement, never replace, them (the architect role's routing decision, inherited).

Query-verb surface (codedb-shaped, MCP + CLI)

A small lazy-advertised set (lean-ctx's 5-tool lesson: default-advertise few, keep the rest callable through one invoker; schema rent is CI-gated by INV-6):

• ctx_search — lexical content/path search (fff underneath): paths_only, compact, offset pagination with a more results … offset=N tail line, weak-match flagging, import-line de-noise.
• ctx_read — mode-gated read (L2 §native-read): signatures default for code, section: for markdown, unchanged:HASH short-circuit via if_hash.
• ctx_doc — markdown verbs: --outline (TOC), --section <pattern>, --code [lang], --links, --skeleton (headings+code); size-gated (files under ~2 KB return whole — two-step retrieval only pays above that).
• ctx_impact — entity-level blast radius (sem): deps/dependents/transitive + affected-test folding ("covered by N tests" one-liners).
• ctx_context — the task-shaped composer: NL task → identifier extraction → one budget-packed block (symbols + ranked files + callers + snippets) with value-ordered packing under a token budget and one-line markers for omitted sections (codedb's handleContext two-step pack — the best single design in that codebase).
• ctx_expand — dereference any handle (CCR hash, read-stub, section handle, omitted-tail marker).

Output-shaping idioms applied uniformly (the grammar of "spend tokens only when forced"): bounded result counts, file:line + enclosing-symbol tags, disclosed omissions (OmittedTail, +N more), escalation footers naming the exact next verb, frecency rendered as words (hot/warm), compact per-line trees instead of JSON (sem measured 3.7× fewer tokens than pretty JSON; techniques 11 — Multi-agent protocols and subagent compression: labeled compact lines beat JSON by 35%).

Ranking

Deterministic, additive-never-filter, every signal kill-switchable (codedb's eval-tuned stack): BM25 core + eponymy/stem boost + path-class priors (tests/vendored down-weighted) + tanh size prior + lexical-saturation penalty + graph-distance boost + git co-change boost (via gix, no shell-out) + fff frecency fusion. No learned ranker.

The doctrine file

Retrieval only saves tokens if the agent uses it correctly, so the role instruction file carries a compiled routing section (one source, per-runtime paths — the architect role's consolidation pattern): fff for files/literals, ctx_impact/rust-analyzer for relationships, ast-grep for syntax shapes, rg for plain text ("the common case — do not force a structural tool on it"), ctx_doc before reading any markdown file over the size gate, "after two searches, read", "never dump trees/snapshots unbounded". The engine's ledger measures adherence (native-read share, search-then-read chains, bounce rate) so doctrine failures are visible, not vibes.

Additional verified mechanisms

• Config-fingerprinted cache keys: every ranking kill-switch and mode default is hashed into result-cache keys, so a config toggle can never serve stale-config results (codedb) — a structural fix for a whole bug class.
• ConvergenceGovernor: an 8-slot ring of recent call-signature hashes per session; three identical calls inside the window inject a one-line nudge naming a better verb — ~20 lines that break agent retry-loops at the tool layer (codedb).
• Warmup replay through the exact handler path: top-N repeated queries from a bounded log tail replayed through the same code path with the same defaults at index start, so cache keys match live traffic (codedb; motivated by its measured "62% of calls are exact repeats").
• Mode-default table with a no-lossy-override guard: markdown/docs/config are never compressed by default; learned/auto modes may pick a more faithful mode than the static default but never a lossier one (lean-ctx — encodes the instructions-must-survive-verbatim failure mode).
• Typed mode enum with byte-stable Display round-trip and legacy-predicate-pinned tests — removes the stringly-mode bug class (lean-ctx #528 pattern).

Index lifecycle

Index once at capsule launch (parallel build over the workspace mounts), then incremental: notify-based watcher patches fff and sem indexes; corpus-signature freshness (content hash over the file set) rather than TTLs decides rebuilds; snapshot to /jackin/state/context/index/ (rkyv/postcard + zstd) for instant warm restarts of the capsule daemon. Memory budget per capsule enforced via content-cache budgets (fff ContentCacheBudget; mmap + CLOCK-cache borrowing rather than whole-content RAM residency — codedb's 1.7 GB steady-state is the anti-lesson).

Implications for jackin❯

These findings define the retrieval layer (l1) contract for jackin❯; any implementation must preserve the acceptance gates and boundaries above.

Limitations and unknowns

The design remains bounded by the evidence and acceptance gates above; volatile dependencies and provider behavior require revalidation before implementation.

Sources

Evidence and repository references are cited inline beside the claims they support.

Related work

• Context engine dossier
• Token-optimization techniques
• Token-optimization tools