25 — sem and mq: entity-level retrieval and structural markdown queries

Volume III addendum, researched 2026-07-03 (source clones + registry/GitHub API sweeps). Two tools the dossier had not covered: sem (Ataraxy-Labs) extends the code-intelligence class of 19 — Code-intelligence tools with an entity/diff/history dimension and one genuinely novel idea; mq (harehare) opens a class no surveyed tool occupies — structural retrieval over Markdown, which is exactly the workload where the first-party measurement found 76.2% of observation tokens flowing through native reads of .mdx prose that every code compressor helps least on. Reuse-surface conclusions feed the jackin❯ Context Engine spec; tool-hub deltas for the original four tools are in tools hub 13.

TL;DR

• sem is "semantic version control" for agents: tree-sitter entities (function/class/method) as the unit of diff/blame/log/impact/context instead of lines/files. Rust workspace, MIT/Apache-2.0, sem-core is a real crates.io library (latest 0.18.0, released 2026-07-03 hours after the sweep's 0.16.2 clone) with disciplined per-language feature flags and proven external consumers. Created 2026-02; 3,032★ vs 6 watchers (hype-curve stars); 27 contributors; solo-founder startup velocity.
• sem's stealable crown jewel is the attention ledger: a per-session server-side record of what the agent's context already holds, fingerprinted by structural hash — re-asks of unchanged entities return a one-line ≡ unchanged (measured 8,586 B → 139 B, 98.4% suppressed); changed entities return an entity-level delta against the version that session saw. No other surveyed tool models the context window as a cache tier. Its marketing ("2.3× more accurate agents") was a circular n=3 self-benchmark — cite the mechanism, not the number (T4). sem has since retired the 2.3×-accuracy and 75%-fewer-tokens hero claims; the benchmarks page now reports paired-run SWE-bench Verified A/B speed results — 50–65% faster code understanding, verify loop 2.90 s → 0.59 s/iter when edges resolve, bimodal with a disclosed 1.2× floor, token parity and unchanged solve rate stated plainly (https://raw.githubusercontent.com/Ataraxy-Labs/sem/HEAD/CHANGELOG.md). The T4 caveat now attaches to those speed claims, still self-run.
• mq is jq for Markdown: a full query/transform language (46 selectors, ~300 builtins, section/TOC stdlib) over a flat markdown-rs node model. Rust, MIT, mq-lang 0.6.4 + mq-markdown 0.6.4 on crates.io (both 0.6.4 since 2026-07-03; 0.6.3 at the 07-03 sweep); mq-markdown is a non-optional nushell dependency (82k downloads; ~84k as of 2026-07-05) — the strongest third-party trust signal in either tool. Solo-maintained, weekly 0.x releases.
• mq's token case measured locally by the sweep: TOC extraction 15,202 B → 286 B (53×) on a README; named-section fetch 11,018 B → 254 B (43×); pattern = one ~50–100-token TOC call + one ~100–500-token section call replacing a 4–8k-token file read. This is the first surveyed lever that attacks the docs/prose read bucket — lean-ctx measured 7.5% on Markdown, headroom 0% on prose-RAG, RTK is Bash-only. Gate below ~2 KB files where two calls cost more than one read (T1-local, small-n; needs the harness like everything else).
• Verdicts: sem-core = DEPEND (entity layer) + reimplement the ledger natively at the runtime's tool boundary where it generalizes to all traffic; sem cloud/telemetry/self-update/git-hijack = strip. mq = DEPEND (sync feature, http-import off) for canned doc-retrieval programs + mq-markdown for heading-bounded chunking; ship the CLI as an agent escape hatch. Neither replaces fff (lexical) or the structural stack (rust-analyzer/ast-grep) — they add the time/history axis (sem) and the prose/docs axis (mq) the existing stack lacks.

sem (Ataraxy-Labs/sem)

Identity. "Semantic version control built on Git… Built for coding agents." Rust ~94% (75k LOC in crates/), dual MIT/Apache-2.0, created 2026-02-05, ~50 releases to v0.16.2 in 5 months (v0.18.0 / sem-core 0.18.0 released 2026-07-03), daily activity. Adoption honesty: 3,032★ / 6 watchers / 93 forks — trending-page stars; the 27 merged-PR contributors and the downstream tools (weave — semantic merge driver; inspect — entity-level review) are the legitimacy signals. Telemetry default-on (SEM_NO_TELEMETRY=1 opt-out); optional cloud tier does server-side clones (sem-cloud.fly.dev) — hard exclusion for container use.

Architecture (4-tier cache framing, docs/attention-architecture.md). L3 optional cloud; L2 SQLite outside the repo (entities/edges/commits/entity_changes; zstd content-store re-slicing entity bodies from byte spans, −20% cache size; a semantic commit index that pre-diffs every commit once → sem log 4.46 s → 0.03 s); L1 a warm resident graph with FS watcher + per-repo Unix-socket sidecar (<10 ms answers; auto-spawned, self-exits after 30 min idle); L0 = the agent's context window, tracked (the attention ledger above, automatic on the MCP path, opt-in SEM_SESSION on CLI). Graph build is two-pass with scope-chain resolution ("compiler-like accuracy without a language server" — heuristic, not compiler truth: dynamic dispatch, re-exports, Rust traits/macros produce false/missing edges). Entity matching: exact ID → structural hash (xxh3 over normalized AST; whitespace/comment-insensitive → rename/move/cosmetic-vs-logic classification) → fuzzy >80% token overlap. 33 tree-sitter grammar crates behind per-language features; 32 code languages + 6 data formats.

Token-relevant features. Budget-packed sem context (priority tiers target → deps → dependents → transitive, full-body → signature → head-truncation fallbacks, test-entity folding "covered by 39 tests", explicit OmittedTail disclosure); compact per-line entity trees (measured 3.7× fewer tokens than JSON rendering); entity-addressed text search (~15 tokens/hit); sem impact with affected-test classification in <10 ms warm; orient structural search (term relevance × graph centrality); hotspots + co-change mining over history; jj (Jujutsu) support. Interception: CLI, 6 MCP tools, GitHub Action, Claude skill, optional PreToolUse guard that hard-denies Grep/Read on code files (philosophically interesting, operationally risky — not adopted), and sem setup rewiring git diff itself (the HN-controversial host write — not adopted). sem setup's host-write surface now also edits ~/.claude/settings.json — SessionStart resident-server + UserPromptSubmit context-injection hooks, unreleased on main — and sem is listed on the official MCP registry as io.github.Ataraxy-Labs/sem (https://raw.githubusercontent.com/Ataraxy-Labs/sem/HEAD/CHANGELOG.md); the strip/not-adopted verdict is unchanged.

Evidence quality. Unusually good epistemic hygiene for a hype-stage repo (CHANGELOG states regressions plainly; token-estimator undercount found and fixed; dependency-accuracy benchmark against PyCG/stack-graphs with published ground truth) — but the headline "2.3× more accurate" was a 3-commit self-benchmark whose questions mirror sem's own JSON (an HN commenter: "Your benchmark doesn't prove that. Your tool is cool. Sell it for what it is."). No independent benchmarks. T1 for mechanisms (ledger suppression, latency, packer arithmetic — trivially reproducible), T4 for product claims. The 2.3× and 75%-fewer-tokens hero claims have since been retired in favor of paired-run SWE-bench Verified A/B speed numbers with the bimodal 1.2× floor disclosed — better hygiene, still first-party, so the T4 tier stands for the new claims too.

Reuse verdict (jackin❯). sem-core = "=0.18.0" (0.16.2 at the 07-03 sweep; two breaking 0.x minors since — 0.16.0's public ContextResult.omitted already broke struct-literal construction, and 0.17/0.18 add unique-method-name edges, Parent::child qualifiers, orient --pack) from crates.io with grammar-core + role languages + parallel: ParserRegistry, EntityGraph, compute_semantic_diff, build_context_result_bounded, orient — ~75% of a structural substrate for one dependency line. The attention-ledger machinery lives in bin-first sem-mcp → reimplement natively in the capsule supervisor, where it can fingerprint every tool result (not just sem fills) — the runtime owner's version is strictly more complete than sem's own. Strip cloud/telemetry/self-update/setup. Pin exact (0.x breaks in minors); watch the wedged cli/mcp crates.io publishing (unpublished sem-cloud-client path-dep) as a workspace-hygiene signal.

mq (harehare/mq)

Identity. jq-like query/transform language + toolchain for Markdown (site mqlang.org); README's first stated use case is LLM workflows. Rust ~127k LOC, MIT, created 2025-02, v0.6.4 (released 2026-07-03), 952★ (2026-07-03); 955★ (2026-07-05), effectively solo (4,706/~5,590 commits by the author; dependabot next). Ships its own agents instruction file and llms.txt, a Claude skill (skills/processing-markdown/), and an MCP server (mq-mcp 0.1.21 as of 2026-07-05: html_to_markdown, extract_markdown, …) — an unusually agent-aware project. Quality signals: codecov, CodSpeed benches, cargo-deny, libFuzzer, proptest.

Architecture. Full language (pipes, functions, pattern matching, try/catch, macros, modules — including HTTP imports behind a default-off allowlisted feature that must stay off in capsules) evaluated per-node over a flat Vec<Node> parsed by markdown-rs (CommonMark/GFM/MDX + frontmatter + footnotes + math + Obsidian extensions behind features; source positions on every node). Engine::compile() → eval_compiled() gives compile-once/run-many; sync feature swaps Rc→Arc for Send/Sync embedding; optional lossless CST with incremental reparse. Workspace of 15+ crates; the two that matter: mq-lang (engine; ~291-crate tree — no format gating upstream, an obvious features PR) and mq-markdown (node model + md/html/text/json render + HTML→Markdown conversion; 19-crate minimal tree; nushell's from md depends on it non-optionally).

Token-relevant features. 46 selectors with typed attribute access (.h.level, .code("rust"), table cells with .row/.column); a section stdlib module (sections/TOC/split/collect — named-section extraction is first-class); update-in-place (|= + -U) so an agent states a transformation in tens of tokens instead of regenerating a file in thousands; grep output mode with node-context (semantically complete snippets); multi-format input (html→md first); -F json with source positions for heading-bounded chunking; ~19 ms cold CLI start measured (negligible); rayon multi-file.

Token case, measured (sweep, this repo's docs). README.md 15,202 B → TOC 286 B (53×); docs/books/src/start/example.md 11,018 B → named section 254 B (43×), TOC 908 B (12×). The agent pattern — cheap TOC, then one section — replaces whole-file reads of large docs; the repo's docs corpus (~234 MD/MDX files, ≈900k–1M tokens, per 20 — Qdrant and vector databases) is exactly the surface. Caveats: T1-local micro-measurements, n=2 files; benefit scales with doc size and section granularity (gate <~2 KB); stdlib 0.x edge-case bugs observed (a section() variant returned empty on the released binary where main was already fixed); tree-walking interpreter — fine for docs, wrong for GB corpora (that's the 0.1.x mq-db prototype's job — watch, don't adopt).

Best-in-class niche. vs mdq: mq is the programmable/embeddable one (engine API, modules, LSP/WASM/MCP). vs pandoc+Lua: unembeddable Haskell binary. vs comrak/markdown-rs ad-hoc: mq is the reusable query layer above markdown-rs. vs remark: Node-in-capsule. Within the surveyed set nothing else does structural markdown retrieval — complementary to everything.

Reuse verdict (jackin❯). mq-lang = { version = "=0.6.4", features = ["sync"] } (http-import off — including the new default-off http-import-ureq feature added in 0.6.4; 0.6.3 at the 07-03 sweep) for canned compiled programs (outline/section/skeleton/code-blocks); mq-markdown = "=0.6.4" alone where only chunking is needed (19 crates); ship the mq binary + skill in capsule images as the ad-hoc escape hatch. Pin exact; treat mq-markdown as the safe long-term dependency (nushell co-pressure), mq-lang as the forkable one.

What this changes in the dossier's standing recommendations

1. The default code-intelligence stack recommendation (19 — Code-intelligence tools/20: rust-analyzer + ast-grep + codedb + fff) gains two axes, not replacements: sem-core for the entity/diff/history axis and mq for the docs/prose axis. codedb's role as structural-verb design donor stands (tools hub 13 records that its own shootout found plain FTS5 beating it end-to-end — output-shape discipline, not index sophistication, is the differentiator).
2. The "nobody ships anything for the prose-read bucket" white space (03 — Prior art and market scan's market-scan finding, sharpened by 10 — First-party measurements's 76.2% measurement) now has a concrete, embeddable, measured lever: markdown-structural retrieval. It joins the negative-cost input-architecture family (bounded retrieval instead of reads) rather than the compression family.
3. The attention-ledger idea (sem) is the first surveyed mechanism that suppresses re-sends rather than shrinking payloads — orthogonal to both compression and retrieval, and maximally effective precisely where cache reads dominate volume (94% measured). It slots into the engine spec as the read-stub/delta-fill layer alongside lean-ctx's [unchanged] stubs, with sem's structural hashes as the change-proof for code.

Source ledger

• sem repo/docs/HN/crates.io: github.com/Ataraxy-Labs/sem · sem-core · HN 175-pt thread · attention architecture doc (accessed 2026-07-03; clone @ v0.16.2-era HEAD; v0.18.0 released 2026-07-03 after the clone — https://github.com/Ataraxy-Labs/sem/releases/tag/v0.18.0, accessed 2026-07-05)
• mq repo/site/crates.io: github.com/harehare/mq · mqlang.org · mq-lang · mq-markdown · nushell dependency: nu-command ≥0.113 from md (accessed 2026-07-03; clone @ bacbc3d)
• Local measurements: byte counts via the sweep's clone-and-run on this repo's docs (method: mq v0.6.3 prebuilt aarch64-darwin; commands recorded in the sweep transcript); token conversions ≈ chars÷4 directional, same caveat as 10 — First-party measurements.