# 04 — Retrieval layer (L1) (https://jackin.tailrocks.com/research/context/engine/04-retrieval-layer/) ## Summary [#summary] Bounded lexical, structural, semantic, and documentation retrieval should replace whole-file reads only when measured task evidence shows a net quality-preserving win. Prevention beats compression: tokens never loaded cost nothing and improve quality (context-rot literature; the dossier's negative-cost set is dominated by input-architecture levers). L1 gives the agent bounded, task-shaped retrieval so it stops reading whole files — the lever measured locally at −91% (outline vs read) and −98% (symbol search vs read), and 12–53× for markdown sections. ## Question and scope [#question-and-scope] How should jackin❯ retrieve the smallest sufficient repository and memory context for each task? ## Method [#method] The analysis compares lexical, structural, and semantic retrieval against the repository’s scoped-rule and memory surfaces, then assigns each source a trigger, budget, provenance record, and fallback. ## Findings [#findings] ### The four retrieval modalities and their engines [#the-four-retrieval-modalities-and-their-engines] | Modality | Engine | Form | Question it owns | | ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Lexical (files, content, literals) | **fff** — `fff-search` + `fff-query-parser` crates, one `SharedFilePicker` per workspace mount, frecency LMDB under `/jackin/state/context/index/` | current design candidate `=0.10.3`, MIT; API/dependency review required | "find the file / the string" — typo-tolerant, frecency+git ranked | | Structural (entities, diffs, impact) | **sem-core** crate — entity graph, structural-hash diff, bounded context surfaces | current design candidate `=0.21.0`, MIT OR Apache-2.0; API/dependency review required | "what is this function / what changed / what breaks / entity-level blame" | | Markdown/docs (sections, TOC, code blocks) | **mq** — `mq-lang` (+`sync`, no `http-import`) compiled canned programs + `mq-markdown` for chunking | current design candidate `=0.8.2`, MIT; API/dependency review required | "which section of this large doc answers the question" — select structure before sending full prose | | Semantic (fuzzy concept recall) | **none by default** | — | Deliberately absent: embeddings for code navigation never met the ≥20% bar ([current retrieval-tools comparison](/research/context/tools/06-combining/)); optional docs-corpus hybrid index stays a role-level experiment | Compiler-grade truth stays with the existing role stack: `rust-analyzer` (LSP) and `ast-grep` (CLI) remain the routing targets for definitions/references and structural rewrites — the engine's verbs complement, never replace, them (the architect role's routing decision, inherited). ### Query-verb surface (codedb-shaped, MCP + CLI) [#query-verb-surface-codedb-shaped-mcp--cli] A small lazy-advertised set (lean-ctx's 5-tool lesson: default-advertise few, keep the rest callable through one invoker; schema rent is CI-gated by INV-6): * `ctx_search` — lexical content/path search (fff underneath): `paths_only`, `compact`, offset pagination with a `more results … offset=N` tail line, weak-match flagging, import-line de-noise. * `ctx_read` — mode-gated read (L2 §native-read): `signatures` default for code, `section:` for markdown, `unchanged:HASH` short-circuit via `if_hash`. * `ctx_doc` — markdown verbs: `--outline` (TOC), `--section `, `--code [lang]`, `--links`, `--skeleton` (headings+code); size-gated (files under \~2 KB return whole — two-step retrieval only pays above that). * `ctx_impact` — entity-level blast radius (sem): deps/dependents/transitive + affected-test folding ("covered by N tests" one-liners). * `ctx_context` — the task-shaped composer: NL task → identifier extraction → one budget-packed block (symbols + ranked files + callers + snippets) with value-ordered packing under a token budget and one-line markers for omitted sections (codedb's `handleContext` two-step pack — the best single design in that codebase). * `ctx_expand` — dereference any handle (CCR hash, read-stub, section handle, omitted-tail marker). Output-shaping idioms applied uniformly (the grammar of "spend tokens only when forced"): bounded result counts, `file:line` + enclosing-symbol tags, disclosed omissions (`OmittedTail`, `+N more`), escalation footers naming the exact next verb, frecency rendered as words (`hot`/`warm`), compact per-line trees instead of JSON (sem measured 3.7× fewer tokens than pretty JSON; [techniques 11 — Multi-agent protocols and subagent compression](/research/context/techniques/11-multi-agent-protocols/): labeled compact lines beat JSON by 35%). ### Ranking [#ranking] Deterministic, additive-never-filter, every signal kill-switchable (codedb's eval-tuned stack): BM25 core + eponymy/stem boost + path-class priors (tests/vendored down-weighted) + tanh size prior + lexical-saturation penalty + graph-distance boost + git co-change boost (via `gix`, no shell-out) + fff frecency fusion. No learned ranker. ### The doctrine file [#the-doctrine-file] Retrieval only saves tokens if the agent uses it correctly, so the role instruction file carries a compiled routing section (one source, per-runtime paths — the architect role's consolidation pattern): fff for files/literals, `ctx_impact`/`rust-analyzer` for relationships, `ast-grep` for syntax shapes, `rg` for plain text ("the common case — do not force a structural tool on it"), `ctx_doc` before reading any markdown file over the size gate, "after two searches, read", "never dump trees/snapshots unbounded". The engine's ledger measures adherence (native-read share, search-then-read chains, bounce rate) so doctrine failures are visible, not vibes. ### Additional verified mechanisms [#additional-verified-mechanisms] * **Config-fingerprinted cache keys**: every ranking kill-switch and mode default is hashed into result-cache keys, so a config toggle can never serve stale-config results (codedb) — a structural fix for a whole bug class. * **ConvergenceGovernor**: an 8-slot ring of recent call-signature hashes per session; three identical calls inside the window inject a one-line nudge naming a better verb — \~20 lines that break agent retry-loops at the tool layer (codedb). * **Warmup replay through the exact handler path**: top-N repeated queries from a bounded log tail replayed through the same code path with the same defaults at index start, so cache keys match live traffic (codedb; motivated by its measured "62% of calls are exact repeats"). * **Mode-default table with a no-lossy-override guard**: markdown/docs/config are **never compressed by default**; learned/auto modes may pick a *more* faithful mode than the static default but never a lossier one (lean-ctx — encodes the instructions-must-survive-verbatim failure mode). * **Typed mode enum** with byte-stable `Display` round-trip and legacy-predicate-pinned tests — removes the stringly-mode bug class (lean-ctx #528 pattern). ### Index lifecycle [#index-lifecycle] Index once at capsule launch (parallel build over the workspace mounts), then incremental: `notify`-based watcher patches fff and sem indexes; corpus-signature freshness (content hash over the file set) rather than TTLs decides rebuilds; snapshot to `/jackin/state/context/index/` (rkyv/postcard + zstd) for instant warm restarts of the capsule daemon. Memory budget per capsule enforced via content-cache budgets (fff `ContentCacheBudget`; mmap + CLOCK-cache borrowing rather than whole-content RAM residency — codedb's 1.7 GB steady-state is the anti-lesson). ## Implications for jackin❯ [#implications-for-jackin] These findings define the retrieval layer (l1) contract for jackin❯; any implementation must preserve the acceptance gates and boundaries above. ## Limitations and unknowns [#limitations-and-unknowns] The design remains bounded by the evidence and acceptance gates above; volatile dependencies and provider behavior require revalidation before implementation. ## Sources [#sources] Evidence and repository references are cited inline beside the claims they support. ## Related work [#related-work] * [Context engine dossier](/research/context/engine/) * [Token-optimization techniques](/research/context/techniques/) * [Token-optimization tools](/research/context/tools/)