02 — Architecture

Crate boundaries, processing layers, data flow, policies, and failure behavior for the jackin❯ context engine.

Summary

One shared jackin-context library should own measurement, observation filtering, retrieval, output policy, routing, and memory so Capsule and host tooling cannot diverge.

Question and scope

How should the context engine divide retrieval, observation, output, routing, memory, and verification without duplicating runtime authority?

Method

The architecture maps the dossier’s validated mechanisms onto existing jackin❯ runtime boundaries, then checks each layer for one owner, explicit data flow, failure isolation, and an acceptance gate.

One new workspace library crate, crates/jackin-context, following the jackin-<noun> subsystem convention (jackin-usage, jackin-isolation). Consumed by jackin-capsule (in-container execution — the primary host) and jackin (host CLI reporting). Internal module map mirrors the layers:

crates/jackin-context/src/
  measure/     L0: ledger, usage ingestion, bust detector, thinking estimator, harness
  observe/     L2: classifier, filter engine (TOML corpus), crushers, code outline, media
  retrieve/    L1: fff embed, sem entities, mq sections, query verbs, packer
  prompt/      L3: register pack compiler, guard lines, codebook, report registers
  route/       L4: effort/model policy, spawn choreography, lanes, quota model
  store/       archive (CCR), read-stub index, handles, session state
  policy/      role config resolution, kill switches, auth-mode policy

No daemon of its own: the engine runs inside the existing jackin-capsule process (PID 1 control plane) — the warm-index economics codedb/fff prove (sub-ms queries, index-once) without a new long-lived service, dashboard, or socket surface. Retrieval indexes live in-process behind the existing daemon state tick; heavy index builds run on the existing tokio runtime with rayon under spawn_blocking.

Interception points (all inside the capsule, zero host writes)

Boundary	Mechanism	Replaces
Shell commands	PATH shims at `/jackin/shims/{git,cargo,npm,pytest,go,…}` ahead of `/jackin/runtime` in PATH (image already prepends PATH in `crates/jackin-image/src/derived_image.rs`), each exec-ing `jackin-capsule ctx run -- <cmd>`	RTK's PreToolUse hook + 14 per-agent integrations; works identically for every runtime incl. hook-less ones (Codex, Amp)
Native `Read`/`Grep`/`Glob`	Engine-owned MCP tools (`ctx_read`, `ctx_search`, `ctx_doc`, …) registered via the existing `claude mcp add jackin-exec`-style path in `crates/jackin-capsule/src/runtime_setup.rs`, plus steering text in the role instruction file; optionally a PostToolUse read-dedup hook via the existing merge-safe `crates/jackin-capsule/src/agent_status/hook_installer.rs` machinery (12 events already wired, drift-repaired every launch)	headroom-MCP / lean-ctx `ctx_read`; the lean-ctx `HookCovered` retreat says steer honestly, never "NEVER use native tools"
Agent configuration	Generated at capsule setup: model/effort flags in `build_agent_command()` (`crates/jackin-capsule/src/session.rs`), subagent frontmatter pins, guard lines and register pack into the role instruction file, `[context]` manifest table plumbed host→capsule via `CapsuleConfig` (`crates/jackin-protocol/src/lib.rs`)	Current Caveman, lean-ctx, and RTK user-scope installer/hook classes
Transcripts / usage	Extends the existing JSONL monitors in `crates/jackin-usage/src/token_monitor.rs` (Claude/Codex/Kimi/Amp already parsed) with `message.id` dedup, per-tool observation attribution, and cache-class deltas	token-optimizer / ccusage / `rtk gain`

Storage

All under /jackin/state/context/ (container-only, honoring the /jackin/ layout rule), using turso (already in-workspace via crates/jackin-usage/src/usage_snapshot_store.rs, schema-versioned):

archive.db — CCR store: ccr_entries(hash TEXT PRIMARY KEY, original BLOB, created_at, ttl_seconds) + FTS index; blake3 keys, first 24 hex chars; wire marker <<ccr:HASH>> kept byte-compatible with headroom so mixed environments interoperate.
ledger.db — savings events (per-transform, per-tool, per-session), bounce events (negative entries), injection-overhead entries; optional hash-chaining for tamper evidence (lean-ctx pattern) off by default.
stubs.json — read-stub bookkeeping only (never content), LRU-capped, atomic tmp+rename, purged on compaction events, scoped session / task:<id> for subagents (lean-ctx #954–#956 semantics).
index/ — fff frecency LMDB, sem-core entity cache, BM25/tantivy segments (if enabled), mq-chunked doc index; all rebuildable, corpus-signature freshness (not TTL).

Cache-safety invariants

The INV table below is the engine's contract; each is verifiable from session JSONL and enforced by regression tests (headroom's prefix_cache_benchmark pattern):

INV	Invariant	Verify by
INV-1	The engine never mutates content already sent: all transforms run at first ingestion (shim output, MCP read result), append-only	Diff request bodies with/without engine on fixture sessions; byte-identical prefixes
INV-2	`cache_read_input_tokens > 0` from call 2 of every session; session cache-read share ≥85% unless the workload explains it	Ledger assertion on live JSONL; alert on regression
INV-3	Filtered output ≤ raw output in estimated tokens, or raw is emitted (`never_worse`, revalidated against RTK v0.45.0)	Property test over the reviewed current filter corpus
INV-4	Exit codes preserved bit-exact through shims; a filter failure emits raw output, never an error (`ParseResult::Passthrough` tier)	Shim integration tests; CI-safe guarantee
INV-5	Every lossy emission carries a recovery handle; `ctx expand <handle>` returns byte-exact originals for entries within retention (all exit codes, not just failures)	Round-trip tests `retrieve(id) == content`
INV-6	Engine-injected standing context (instructions, tool schemas, codebook) stays under a CI-gated token budget counted with `count_tokens` against the capsule's actual model	`ctx doctor --gate` in CI (lean-ctx pattern)
INV-7	Never compress error/test-failure signal: policy classifier marks build/lint/test-failure lines verbatim-class before any filter runs	Canary fixtures per filter
INV-8	Exactly one shell-rewrite path, one output register, one memory store per capsule; engine refuses to activate if a conflicting third-party tool is detected in the role (e.g. RTK hook + engine shims)	Setup-time conflict probe
INV-9	Mid-session model/effort switches are engine-initiated only at task boundaries; the engine never toggles prompt-resident modes mid-session	Route-policy unit tests; JSONL audit
INV-10	No network egress from the engine (no telemetry, no update checks, no cloud sync); `count_tokens` is the only permitted API call and runs on the capsule's own credential within its rate pool	Firewall allowlist already enforced by `crates/jackin-capsule/src/firewall.rs`

Kill switches and config

Every subsystem carries an env kill switch following the existing JACKIN_DISABLE_TIRITH/JACKIN_DISABLE_SHELLFIRM convention: JACKIN_CTX_DISABLE (master), JACKIN_CTX_NO_SHIMS, JACKIN_CTX_NO_MCP, JACKIN_CTX_NO_REGISTER, JACKIN_CTX_NO_ROUTING, JACKIN_CTX_MEASURE_ONLY. The last one matters most: measurement-on, mutation-off is the default posture for new roles (lean-ctx's "measurement before mutation" doctrine) — a role graduates to active compression only after its own baseline window exists in the ledger.

Role-facing configuration is a new versioned [context] table in jackin.role.toml (schema change per PRERELEASE rules — one version bump, 5 artifacts): see 10 — Rollout for the field sketch.

Implications for jackin❯

These findings define the architecture contract for jackin❯; any implementation must preserve the acceptance gates and boundaries above.

Limitations and unknowns

What the engine does NOT contain

No output-style rewriting of the agent's own settings files beyond jackin❯ existing merge-safe installers; no browser dashboard (surfaces render in the existing capsule TUI usage dialog and host console); no cross-container daemon (fleet coordination stays in the host orchestrator); no cloud component whatsoever.

Sources

Evidence and repository references are cited inline beside the claims they support.