# 25 — sem and mq: entity-level retrieval and structural markdown queries (https://jackin.tailrocks.com/reference/research/token-optimization-tools/25-sem-and-mq-structural-and-markdown-retrieval/)



# 25 — sem and mq: entity-level retrieval and structural markdown queries [#25--sem-and-mq-entity-level-retrieval-and-structural-markdown-queries]

Volume III addendum, researched 2026-07-03 (source clones + registry/GitHub API sweeps). Two tools the dossier had not covered: **sem** (Ataraxy-Labs) extends the code-intelligence class of [19 — Code-intelligence tools](/reference/research/token-optimization-tools/19-code-intelligence-tools/) with an entity/diff/history dimension and one genuinely novel idea; **mq** (harehare) opens a class no surveyed tool occupies — structural retrieval over *Markdown*, which is exactly the workload where [the first-party measurement](/reference/research/token-optimization-tools/10-first-party-measurements/) found 76.2% of observation tokens flowing through native reads of `.mdx` prose that every code compressor helps least on. Reuse-surface conclusions feed the [jackin❯ Context Engine spec](/reference/research/jackin-context-engine/); tool-hub deltas for the original four tools are in [tools hub 13](/reference/research/token-optimization-tools/13-reuse-surfaces-and-deltas/).

## TL;DR [#tldr]

* **sem is "semantic version control" for agents**: tree-sitter entities (function/class/method) as the unit of diff/blame/log/impact/context instead of lines/files. Rust workspace, MIT/Apache-2.0, **`sem-core` is a real crates.io library** (latest 0.18.0, released 2026-07-03 hours after the sweep's 0.16.2 clone) with disciplined per-language feature flags and proven external consumers. Created 2026-02; 3,032★ vs 6 watchers (hype-curve stars); 27 contributors; solo-founder startup velocity.
* **sem's stealable crown jewel is the attention ledger**: a per-session server-side record of what the agent's context already holds, fingerprinted by structural hash — re-asks of unchanged entities return a one-line `≡ unchanged` (measured 8,586 B → 139 B, 98.4% suppressed); changed entities return an entity-level **delta against the version that session saw**. No other surveyed tool models the context window as a cache tier. Its marketing ("2.3× more accurate agents") was a circular n=3 self-benchmark — cite the mechanism, not the number (T4). sem has since retired the 2.3×-accuracy and 75%-fewer-tokens hero claims; the benchmarks page now reports paired-run SWE-bench Verified A/B *speed* results — 50–65% faster code understanding, verify loop 2.90 s → 0.59 s/iter when edges resolve, bimodal with a disclosed 1.2× floor, token parity and unchanged solve rate stated plainly ([https://raw.githubusercontent.com/Ataraxy-Labs/sem/HEAD/CHANGELOG.md](https://raw.githubusercontent.com/Ataraxy-Labs/sem/HEAD/CHANGELOG.md)). The T4 caveat now attaches to those speed claims, still self-run.
* **mq is jq for Markdown**: a full query/transform language (46 selectors, \~300 builtins, section/TOC stdlib) over a flat markdown-rs node model. Rust, MIT, **`mq-lang` 0.6.4 + `mq-markdown` 0.6.4 on crates.io** (both 0.6.4 since 2026-07-03; 0.6.3 at the 07-03 sweep); `mq-markdown` is a *non-optional nushell dependency* (82k downloads; \~84k as of 2026-07-05) — the strongest third-party trust signal in either tool. Solo-maintained, weekly 0.x releases.
* **mq's token case measured locally by the sweep**: TOC extraction 15,202 B → 286 B (**53×**) on a README; named-section fetch 11,018 B → 254 B (**43×**); pattern = one \~50–100-token TOC call + one \~100–500-token section call replacing a 4–8k-token file read. This is the first surveyed lever that attacks the docs/prose read bucket — lean-ctx measured 7.5% on Markdown, headroom 0% on prose-RAG, RTK is Bash-only. Gate below \~2 KB files where two calls cost more than one read (T1-local, small-n; needs the harness like everything else).
* **Verdicts**: sem-core = DEPEND (entity layer) + reimplement the ledger natively at the runtime's tool boundary where it generalizes to *all* traffic; sem cloud/telemetry/self-update/git-hijack = strip. mq = DEPEND (`sync` feature, `http-import` off) for canned doc-retrieval programs + `mq-markdown` for heading-bounded chunking; ship the CLI as an agent escape hatch. Neither replaces fff (lexical) or the structural stack (rust-analyzer/ast-grep) — they add the *time/history* axis (sem) and the *prose/docs* axis (mq) the existing stack lacks.

## sem (Ataraxy-Labs/sem) [#sem-ataraxy-labssem]

**Identity.** "Semantic version control built on Git… Built for coding agents." Rust \~94% (75k LOC in `crates/`), dual MIT/Apache-2.0, created 2026-02-05, \~50 releases to v0.16.2 in 5 months (v0.18.0 / sem-core 0.18.0 released 2026-07-03), daily activity. Adoption honesty: 3,032★ / 6 watchers / 93 forks — trending-page stars; the 27 merged-PR contributors and the downstream tools (weave — semantic merge driver; inspect — entity-level review) are the legitimacy signals. Telemetry default-on (`SEM_NO_TELEMETRY=1` opt-out); optional cloud tier does **server-side clones** (`sem-cloud.fly.dev`) — hard exclusion for container use.

**Architecture (4-tier cache framing, `docs/attention-architecture.md`).** L3 optional cloud; L2 SQLite outside the repo (entities/edges/commits/entity\_changes; zstd content-store re-slicing entity bodies from byte spans, −20% cache size; a **semantic commit index** that pre-diffs every commit once → `sem log` 4.46 s → 0.03 s); L1 a warm resident graph with FS watcher + per-repo Unix-socket sidecar (\<10 ms answers; auto-spawned, self-exits after 30 min idle); **L0 = the agent's context window, tracked** (the attention ledger above, automatic on the MCP path, opt-in `SEM_SESSION` on CLI). Graph build is two-pass with scope-chain resolution ("compiler-like accuracy without a language server" — heuristic, not compiler truth: dynamic dispatch, re-exports, Rust traits/macros produce false/missing edges). Entity matching: exact ID → structural hash (xxh3 over normalized AST; whitespace/comment-insensitive → rename/move/cosmetic-vs-logic classification) → fuzzy >80% token overlap. 33 tree-sitter grammar crates behind per-language features; 32 code languages + 6 data formats.

**Token-relevant features.** Budget-packed `sem context` (priority tiers target → deps → dependents → transitive, full-body → signature → head-truncation fallbacks, test-entity folding "covered by 39 tests", explicit `OmittedTail` disclosure); compact per-line entity trees (measured 3.7× fewer tokens than JSON rendering); entity-addressed text search (\~15 tokens/hit); `sem impact` with affected-test classification in \<10 ms warm; `orient` structural search (term relevance × graph centrality); hotspots + co-change mining over history; jj (Jujutsu) support. Interception: CLI, 6 MCP tools, GitHub Action, Claude skill, optional PreToolUse guard that hard-denies Grep/Read on code files (philosophically interesting, operationally risky — not adopted), and `sem setup` rewiring `git diff` itself (the HN-controversial host write — not adopted). `sem setup`'s host-write surface now also edits `~/.claude/settings.json` — SessionStart resident-server + UserPromptSubmit context-injection hooks, unreleased on main — and sem is listed on the official MCP registry as `io.github.Ataraxy-Labs/sem` ([https://raw.githubusercontent.com/Ataraxy-Labs/sem/HEAD/CHANGELOG.md](https://raw.githubusercontent.com/Ataraxy-Labs/sem/HEAD/CHANGELOG.md)); the strip/not-adopted verdict is unchanged.

**Evidence quality.** Unusually good epistemic hygiene for a hype-stage repo (CHANGELOG states regressions plainly; token-estimator undercount found and fixed; dependency-accuracy benchmark against PyCG/stack-graphs with published ground truth) — but the headline "2.3× more accurate" was a 3-commit self-benchmark whose questions mirror sem's own JSON (an HN commenter: "Your benchmark doesn't prove that. Your tool is cool. Sell it for what it is."). No independent benchmarks. T1 for mechanisms (ledger suppression, latency, packer arithmetic — trivially reproducible), T4 for product claims. The 2.3× and 75%-fewer-tokens hero claims have since been retired in favor of paired-run SWE-bench Verified A/B speed numbers with the bimodal 1.2× floor disclosed — better hygiene, still first-party, so the T4 tier stands for the new claims too.

**Reuse verdict (jackin❯).** `sem-core = "=0.18.0"` (0.16.2 at the 07-03 sweep; two breaking 0.x minors since — 0.16.0's public `ContextResult.omitted` already broke struct-literal construction, and 0.17/0.18 add unique-method-name edges, `Parent::child` qualifiers, `orient --pack`) from crates.io with `grammar-core` + role languages + `parallel`: `ParserRegistry`, `EntityGraph`, `compute_semantic_diff`, `build_context_result_bounded`, `orient` — \~75% of a structural substrate for one dependency line. The attention-ledger machinery lives in bin-first `sem-mcp` → reimplement natively in the capsule supervisor, where it can fingerprint **every** tool result (not just sem fills) — the runtime owner's version is strictly more complete than sem's own. Strip cloud/telemetry/self-update/setup. Pin exact (0.x breaks in minors); watch the wedged cli/mcp crates.io publishing (unpublished `sem-cloud-client` path-dep) as a workspace-hygiene signal.

## mq (harehare/mq) [#mq-hareharemq]

**Identity.** jq-like query/transform **language + toolchain for Markdown** (site mqlang.org); README's first stated use case is LLM workflows. Rust \~127k LOC, MIT, created 2025-02, v0.6.4 (released 2026-07-03), 952★ (2026-07-03); 955★ (2026-07-05), effectively solo (4,706/\~5,590 commits by the author; dependabot next). Ships its own agents instruction file and llms.txt, a Claude skill (`skills/processing-markdown/`), and an MCP server (`mq-mcp` 0.1.21 as of 2026-07-05: `html_to_markdown`, `extract_markdown`, …) — an unusually agent-aware project. Quality signals: codecov, CodSpeed benches, cargo-deny, libFuzzer, proptest.

**Architecture.** Full language (pipes, functions, pattern matching, try/catch, macros, modules — including HTTP imports behind a default-off allowlisted feature that must stay off in capsules) evaluated per-node over a &#x2A;*flat `Vec<Node>`** parsed by markdown-rs (CommonMark/GFM/MDX + frontmatter + footnotes + math + Obsidian extensions behind features; source positions on every node). `Engine::compile()` → `eval_compiled()` gives compile-once/run-many; `sync&#x60; feature swaps Rc→Arc for Send/Sync embedding; optional lossless CST with incremental reparse. Workspace of 15+ crates; the two that matter: &#x2A;*`mq-lang`*&#x2A; (engine; \~291-crate tree — no format gating upstream, an obvious features PR) and &#x2A;*`mq-markdown`** (node model + md/html/text/json render + HTML→Markdown conversion; **19-crate minimal tree**; nushell's `from md` depends on it non-optionally).

**Token-relevant features.** 46 selectors with typed attribute access (`.h.level`, `.code("rust")`, table cells with `.row/.column`); a `section` stdlib module (sections/TOC/split/collect — named-section extraction is first-class); update-in-place (`|=` + `-U`) so an agent states a transformation in tens of tokens instead of regenerating a file in thousands; grep output mode with *node*-context (semantically complete snippets); multi-format input (html→md first); `-F json` with source positions for heading-bounded chunking; \~19 ms cold CLI start measured (negligible); rayon multi-file.

**Token case, measured (sweep, this repo's docs).** README.md 15,202 B → TOC 286 B (53×); `docs/books/src/start/example.md` 11,018 B → named section 254 B (43×), TOC 908 B (12×). The agent pattern — cheap TOC, then one section — replaces whole-file reads of large docs; the repo's docs corpus (\~234 MD/MDX files, ≈900k–1M tokens, per [20 — Qdrant and vector databases](/reference/research/token-optimization-tools/20-qdrant-and-vector-databases/)) is exactly the surface. Caveats: T1-local micro-measurements, n=2 files; benefit scales with doc size and section granularity (gate \<\~2 KB); stdlib 0.x edge-case bugs observed (a `section()` variant returned empty on the released binary where main was already fixed); tree-walking interpreter — fine for docs, wrong for GB corpora (that's the 0.1.x `mq-db` prototype's job — watch, don't adopt).

**Best-in-class niche.** vs mdq: mq is the programmable/embeddable one (engine API, modules, LSP/WASM/MCP). vs pandoc+Lua: unembeddable Haskell binary. vs comrak/markdown-rs ad-hoc: mq *is* the reusable query layer above markdown-rs. vs remark: Node-in-capsule. Within the surveyed set nothing else does structural markdown retrieval — complementary to everything.

**Reuse verdict (jackin❯).** `mq-lang = { version = "=0.6.4", features = ["sync"] }` (http-import off — including the new default-off `http-import-ureq` feature added in 0.6.4; 0.6.3 at the 07-03 sweep) for canned compiled programs (outline/section/skeleton/code-blocks); `mq-markdown = "=0.6.4"` alone where only chunking is needed (19 crates); ship the `mq` binary + skill in capsule images as the ad-hoc escape hatch. Pin exact; treat `mq-markdown` as the safe long-term dependency (nushell co-pressure), `mq-lang` as the forkable one.

## What this changes in the dossier's standing recommendations [#what-this-changes-in-the-dossiers-standing-recommendations]

1. The default code-intelligence stack recommendation ([19 — Code-intelligence tools](/reference/research/token-optimization-tools/19-code-intelligence-tools/)/20: `rust-analyzer + ast-grep + codedb + fff`) gains two axes, not replacements: **sem-core for the entity/diff/history axis** and **mq for the docs/prose axis**. codedb's role as structural-verb design donor stands ([tools hub 13](/reference/research/token-optimization-tools/13-reuse-surfaces-and-deltas/) records that its own shootout found plain FTS5 beating it end-to-end — output-shape discipline, not index sophistication, is the differentiator).
2. The "nobody ships anything for the prose-read bucket" white space ([03 — Prior art and market scan](/reference/research/token-optimization-techniques/03-prior-art-and-market-scan/)'s market-scan finding, sharpened by [10 — First-party measurements](/reference/research/token-optimization-tools/10-first-party-measurements/)'s 76.2% measurement) now has a concrete, embeddable, measured lever: markdown-structural retrieval. It joins the negative-cost input-architecture family (bounded retrieval instead of reads) rather than the compression family.
3. The attention-ledger idea (sem) is the first surveyed mechanism that suppresses **re-sends** rather than shrinking payloads — orthogonal to both compression and retrieval, and maximally effective precisely where cache reads dominate volume (94% measured). It slots into the engine spec as the read-stub/delta-fill layer alongside lean-ctx's `[unchanged]` stubs, with sem's structural hashes as the change-proof for code.

## Source ledger [#source-ledger]

* sem repo/docs/HN/crates.io: [github.com/Ataraxy-Labs/sem](https://github.com/Ataraxy-Labs/sem) · [sem-core](https://crates.io/crates/sem-core) · [HN 175-pt thread](https://news.ycombinator.com/item?id=48428475) · [attention architecture doc](https://github.com/Ataraxy-Labs/sem/blob/main/docs/attention-architecture.md) (accessed 2026-07-03; clone @ v0.16.2-era HEAD; v0.18.0 released 2026-07-03 after the clone — [https://github.com/Ataraxy-Labs/sem/releases/tag/v0.18.0](https://github.com/Ataraxy-Labs/sem/releases/tag/v0.18.0), accessed 2026-07-05)
* mq repo/site/crates.io: [github.com/harehare/mq](https://github.com/harehare/mq) · [mqlang.org](https://mqlang.org) · [mq-lang](https://crates.io/crates/mq-lang) · [mq-markdown](https://crates.io/crates/mq-markdown) · nushell dependency: `nu-command` ≥0.113 `from md` (accessed 2026-07-03; clone @ `bacbc3d`)
* Local measurements: byte counts via the sweep's clone-and-run on this repo's docs (method: `mq` v0.6.3 prebuilt aarch64-darwin; commands recorded in the sweep transcript); token conversions ≈ chars÷4 directional, same caveat as [10 — First-party measurements](/reference/research/token-optimization-tools/10-first-party-measurements/).