Terminal Observation and Automation

Status: Open — research and design proposal (Phase 2, Agent Orchestrator Research Program)

Problem

jackin' already runs agent CLIs inside visible, isolated terminal sessions, but the terminal is still mostly an interactive surface. The operator can attach, watch, and type; the host can ask for session inventory and pane layout; future workflow items expect session.read, session.wait, and session.send; but there is not yet a first-class contract for programmatic terminal observation and automation: read the visible cells, wait for a visible condition, inject input, capture an artifact bundle, record a replayable trace, and let external orchestrators drive a Capsule session without scraping docker exec output.

The research trigger was kitlangton/cellshot, a small MIT-licensed Rust project that treats terminal state as structured visual data. cellshot can launch a CLI in a PTY, wait for text, send key sequences, snapshot the terminal into text/JSON/ANSI/SVG/PNG artifacts, keep a persistent session daemon, and record a timeline that can render to video. That is close to the missing layer for jackin' agent orchestration, but the implementation shape is not something jackin' should embed wholesale.

Decision

Borrow the product and protocol ideas from cellshot and adjacent terminal-recorder/test tools, but implement them natively inside jackin-capsule. Do not run cellshot as a nested daemon inside role containers and do not make it a core runtime dependency.

The reason is simple: cellshot builds a PTY daemon because it needs one; jackin' already has one. Capsule already owns PTYs, sessions, pane layout, attach takeover, OSC passthrough, jackin-term DamageGrid screen state, and the control socket. A nested runtime would create this bad layering:

jackin host
  └─ role container
      └─ jackin-capsule daemon
          └─ cellshot daemon
              └─ agent PTY

That would complicate resize, focus, OSC policy, auth/runtime setup, state inference, attach/hijack, and session identity. The native shape is instead:

jackin host / future daemon / external adapter
  └─ Capsule control protocol
      └─ existing session PTY + DamageGrid screen + raw output trace

cellshot remains useful as a development/docs/test tool for capturing the outer jackin console experience, and selected MIT-licensed implementation ideas can be reused with attribution if they are simpler to port than rewrite. The core orchestration surface should live in jackin-protocol and jackin-capsule.

What cellshot contributes

cellshot is valuable because it packages several terminal-agent primitives into one coherent CLI:

The feature to copy first is not PNG generation. It is the idea that a terminal session has a queryable visual state and that state can be addressed by other tools.

Adjacent project research

The landscape splits into recorders, scripted demo/test tools, VT emulators, widget test backends, and classic expect-style PTY automation. The recurring design lesson is a two-layer model: timestamp raw PTY bytes for lossless replay, and maintain a VT screen model for queryable state.

Current jackin' leverage

Capsule already has the right substrate. It runs as the in-container control plane, owns PTYs and sessions, keeps DamageGrid state per session, forwards raw output over the binary attach channel, and exposes a JSON control channel for one-shot queries. The relevant local surfaces are crates/jackin-capsule/src/session.rs, crates/jackin-capsule/src/daemon.rs, crates/jackin-capsule/src/client.rs, crates/jackin-protocol/src/control.rs, and crates/jackin-runtime/src/runtime/attach.rs.

The missing pieces are not another PTY runner. They are typed control messages, stable artifact schemas, recording policy, and host/CLI adapters that can use the live screen state safely.

Target capabilities

The product shape is Playwright-like control for live jackin' instances, but against terminal sessions instead of browser pages. A test, workflow runner, or external orchestrator should be able to address an instance, pick a tab/session, wait for visible or semantic state, type into the agent input, press enter, switch focus, capture the resulting terminal moment, and continue from the evidence it receives. The API should feel deterministic and scriptable, while preserving the core jackin' promise that the session remains visible, attachable, and hijackable by the operator.

Conceptually:

jackin instance
  ├─ select tab/session
  ├─ wait for visible text or runtime status
  ├─ type prompt text
  ├─ send enter / key tokens
  ├─ wait for progress, blocker, or completion evidence
  └─ capture text / JSON frame / screenshot / recording marker

Proposed API surface

The API should model the things operators already see, not Docker internals. The stable resource hierarchy is: instance → tab → pane/session. A tab is the operator-visible workspace inside Capsule; a session is the PTY-backed agent or shell running in a pane. The API can start as CLI/control-channel calls and later be exposed through the daemon/MCP adapter without changing the vocabulary.

The minimum useful external flow should read like:

instance = instance.describe("workspace-or-instance")
tab = tab.create(instance.id, label: "implement terminal snapshot API")
providers = provider.list(instance.id, agent: "claude")
session = session.create(tab.id, kind: agent, agent: "claude", provider: providers.default)
session.wait(session.id, status: "idle", visible: /How can I help|>/, timeout: 30s)
session.submit(session.id, "Implement the roadmap item and run the docs checks")
events.subscribe(instance.id, filter: { session_id: session.id })
session.wait(session.id, status: "blocked|done", timeout: 30m)
session.capture(session.id, out: "after-agent-response")

blocked and waiting details should not be invented by this API. They come from the agent runtime status authority, which may use runtime hooks/APIs, foreground process evidence, visible-screen signals, and cursor/readiness probes. Terminal observation consumes that status and combines it with the visible screen and trace evidence so callers can both know that an agent is waiting and inspect what it is waiting on.

Screen text and frame snapshots

Add a control call that returns the current visible terminal state for a target session. The first version should return plain rows and enough metadata to identify the session, dimensions, revision, and cursor. A later version should add a structured frame with sparse styled cells.

Candidate response shapes:

ScreenText {
    session_id,
    cols,
    rows,
    revision,
    lines,
}

TerminalFrame {
    session_id,
    cols,
    rows,
    revision,
    cursor,
    cells,
}

This should power jackin session text, preview panes, golden tests, external MCP tools, and the future workflow runner's evidence model.

Wait-for-visible condition

Add a blocking control call that waits until a visible-screen condition becomes true or times out. Start with literal text and regex over visible screen rows. Later conditions can include effective runtime status, process exit, explicit marker, cursor readiness, or quiet/stable screen windows.

The important rule: a wait result must return evidence, not just success. The caller should know which session revision matched, which text matched, and whether the source was visible text, status authority, marker, or timeout.

Input injection

Add a control call that sends bytes or named key tokens to a target session without needing an interactive attach client. It should share the same parser as future automation scripts and should support literal text, enter, escape, tab, arrows, control keys, paste-mode-safe chunks, and optional pacing for demos.

Input injection is powerful and should be scoped by policy. Local CLI use is fine; daemon/MCP/external callers need capability gates so remote tools cannot silently control a session without operator consent.

Snapshot artifact bundle

Add a CLI command that writes a snapshot bundle from one session:

<stem>.txt
<stem>.json
<stem>.ansi
<stem>.svg
<stem>.png

The first shippable cut can be .txt and .json. Raw ANSI requires either bounded retained output per session or explicit recording mode. SVG/PNG can follow once the frame schema is stable. PNG should be derived from SVG or another deterministic renderer, not captured from the host terminal window.

Recording and replay

Add opt-in recording at the Capsule/session level. The artifact should be append-only, timestamped, redaction-aware, and useful both for replay and debugging. Two formats are worth supporting:

Raw ttyrec-style output is useful as a lowest-level debug export, but the durable jackin' recording should know about sessions, inputs, status markers, and operator interventions.

Automation script surface

After read/wait/send are proven, add a small script runner inspired by VHS and expect. It should be intentionally narrower than a workflow runner:

Session 2
Wait /How can I help/
Type "Run the tests and summarize failures"
Enter
Wait /tests passed|tests failed/ Timeout 10m
Snapshot "after-tests"

This is useful for role smoke tests, docs demos, and reproducing agent runtime issues. It should not decide Git branches, PR lifecycle, merge policy, or task queues; that belongs to agent workflow orchestration.

Architecture

The architecture should keep three surfaces separate:

The control path should extend the existing length-prefixed JSON control channel rather than adding a second socket. Hot-path interactive attach remains binary raw PTY bytes. Observation/automation calls are one-shot or bounded blocking requests.

Relationship to runtime status

This item does not replace the agent runtime status authority. They complement each other:

The workflow runner should prefer status/marker waits when it needs semantic lifecycle truth and visible-text waits when the operator or test explicitly cares that text is on screen.

Phases

Phase 0 — Prototype against existing Capsule state

Add internal helper methods that extract visible text rows from the current GridSnapshot for a session and unit-test them with controlled ANSI streams. No public CLI yet.

Phase 1 — Text snapshot and visible wait

Extend the control protocol with a target-session text snapshot and a bounded wait-for-visible-text request. Add host-side CLI commands only after the protocol behavior is tested in Capsule. Return match evidence and timeout diagnostics.

Phase 2 — Tab/session create, select, and input injection

Add tab selection/creation, session creation, provider resolution, named input tokens, literal text sending, and submit through the control channel. Reuse the same key-token parser in CLI and future automation scripts. Gate non-interactive/external callers behind explicit policy before exposing through daemon or MCP surfaces.

Phase 3 — Structured frame schema

Expose sparse styled cells, cursor, dimensions, default colors, and revisions. Keep the schema stable enough for golden tests and downstream renderers. This is the point where cellshot's frame model is most directly relevant.

Phase 4 — Snapshot artifact bundle

Write .txt and .json first, then .ansi, .svg, and .png. SVG rendering should borrow the block-character geometry idea from cellshot; PNG should be deterministic and generated from the frame model, not from a host-window screenshot.

Phase 5 — Recording and replay

Add opt-in per-session recording, with local redaction warnings and a clear privacy model. Start with native JSONL trace; add asciicast export when the event model is stable. Emit agent-status and operator-intervention markers so recordings explain why a workflow paused or resumed.

Phase 6 — Automation scripts and adapter surface

Add a tiny tape/expect-style runner for smoke tests and demos. Then expose the same primitives through the future MCP/daemon adapter surface so external orchestrators can use jackin' as a visible, isolated execution substrate.

Open questions

Implementation notes

• Keep local file references linked with <RepoFile /> when this roadmap item is updated; the key implementation seams are crates/jackin-protocol/src/control.rs, crates/jackin-capsule/src/socket.rs, crates/jackin-capsule/src/session.rs, crates/jackin-capsule/src/daemon.rs, and crates/jackin-runtime/src/runtime/snapshot.rs.
• Prefer protocol additions that are useful to both the CLI and the future daemon/MCP surface. Do not add CLI-only shell parsing that the daemon later has to reverse-engineer.
• Every blocking wait must have a timeout, cancellation behavior, and diagnostic output. A wait that silently hangs is worse than no wait primitive.
• Add fixture tests for ANSI streams, wide characters, alternate screen behavior, cursor visibility, line wrapping, and OSC passthrough interactions before depending on snapshots for workflow decisions.
• Treat generated artifacts as potentially sensitive. Terminal screens can contain paths, secrets, tokens, prompts, and command output from mounted workspaces.

Related roadmap items

• jackin' Capsule control plane — existing in-container PTY multiplexer and the natural implementation point.
• Console agent session control — shipped operator UI for discovering instances, starting secondary sessions, opening shells, and attaching to panes; this API is the scriptable counterpart.
• Multi-runtime support — agent and provider choices in session.create must follow the same runtime/provider model as interactive launch.
• Agent runtime status authority — semantic state source consumed by waits, markers, recordings, and attention prompts.
• Agent workflow orchestration — future workflow runner that needs session read/wait/send primitives but should not own terminal parsing.
• Agent attention prompts — notifications should consume status events and may include snapshot evidence when appropriate.
• TUI Design Decisions and visual snapshot testing (CLI & TUI) — deterministic renderer/golden-test work that should converge with this terminal-frame artifact model on a shared styled-cell schema; that item asserts render regressions, this one captures live sessions for orchestration.
• Persistent storage layer — durable home for recordings, traces, workflow events, and artifact indexes.