feat: session-level output dedup

[nathanestone-alt]

Summary

Adds session-level output dedup: when an agent re-reads the same file or re-runs the same command and the output is byte-identical to something already emitted within the same Claude Code session, RTK replaces it with a one-line stub instead of re-paying the tokens.

rtk-dedup: identical to output at step 1 · 60 lines / ~1241 tok suppressed [read:…] (force full: rtk proxy <cmd>)

Filtering compresses a single command's output; this attacks a different sink — agents re-sending identical context every turn. Opt-in and off by default.

Measured impact (honest numbers)

I mined my own RTK tracking DB (943 commands over several weeks of real agentic use) to measure the repeat rate directly, instead of inferring it from filtering stats:

• 57% of all commands (537/943) were byte-identical re-invocations of an earlier command; for read-type commands it's 69%.
• Token-weighted by post-filter output (what dedup actually suppresses): 48.5% of all output tokens (~104K) were emitted on repeat invocations — ~80% for reads.
• That 104K is a ceiling, not a forecast: identical command line ≠ identical bytes (files change between reads — the two largest reads in my history were the same file re-read with changed content, which the SHA-256 guard correctly would not suppress), and dedup is session-scoped while this analysis pools all history.

So the honest positioning: dedup is a second-order optimization on top of filtering — filtering removed ~894K tokens on the same history; dedup's realistic additional yield is in the tens of thousands. The repeat pattern it targets is real and pervasive (half of all output tokens), but the per-token payoff is bounded because filtering already crushed the large outputs. I'm dogfooding the branch now and will report actual rtk gain dedup numbers here after some real use.

How it works

• Session plumbing — hooks::hook_cmd::process_claude_payload reads session_id from the PreToolUse payload and splices --session <id> into the rewritten command. core::session resolves it (flag → RTK_SESSION_ID env → none) into a process-global OnceLock. No session (all manual invocations) ⇒ dedup no-ops and output is byte-for-byte unchanged.
• Two-key safety model — the SHA-256 of the raw, pre-filter bytes is the correctness guarantee (different content can never share a hash, so changed files/commands are never suppressed). The command identity is only a cosmetic key for the stub message, so identity collisions are harmless. Reuses the existing sha2 dep — no new dependencies.
• Guards (each falls back to full output): disabled, no session, command exited non-zero (unless suppress_on_error), output below min_tokens, or any DB error. Suppression can only ever omit a repeat, never hide new/changed/failed output.
• Ledger — a session_outputs table in the tracking SQLite DB (keyed by session_id + content_hash), storing hashes and counters only, never content; pruned after 48h idle.
• Seams — core::runner::run_captured_filter (non-tee branch — covers git/gh/cargo and most ecosystems) and cmds::system::read (file + stdin). Non-suppressed output takes the exact original print path (byte-identical, no snapshot churn).
• Reporting — rtk gain shows Dedup suppressed: X over N repeat-emissions, separate from the filtering total so the two never double-count.

Config / toggle

[dedup]
enabled = false          # opt-in; suppression changes agent-visible output
min_tokens = 200
suppress_on_error = false

RTK_DEDUP=1 force-enables without a config file (mirrors RTK_DB_PATH / RTK_HOOK_AUDIT).

Commits (each gated on fmt + clippy + test --all)

1. --session global flag + core::session
2. Hook injects --session <id> from the PreToolUse payload
3. session_outputs ledger + CRUD
4. maybe_suppress decision fn + [dedup] config
5. Wire into read + runner print seams
6. rtk gain dedup reporting
7. Integration tests + ARCHITECTURE docs

Testing

• Unit tests for session resolution, hook injection (incl. shell-injection-safe id validation), ledger CRUD, and every maybe_suppress guard.
• tests/dedup_integration_test.rs drives the real binary: suppresses identical re-read within a session, never suppresses without a session id, isolates across sessions.
• Verified end-to-end on the release binary (re-read → stub; different session → full; no session → full; --session flag path → stub; rtk gain line renders).

Known MVP limitations (safe degradations)

• Compound a && b commands get --session on the first rtk segment only; later segments run session-less and no-op (never incorrect).
• The tee and skip-filter-on-failure runner paths are left full (failure-heavy, low-value).
• Diff-on-change deferred to a follow-up.
• Subagents share the parent session's session_id in PreToolUse payloads, so a freshly-spawned subagent re-reading a file the parent already read receives a stub for content its context has never seen. The stub is self-describing (force full: rtk proxy <cmd>), so the cost is one recovery round-trip rather than silent wrongness — but it's a known soft spot, same class as the compaction caveat (mitigated by the PostCompact hook + recency_window backstop for the main-loop case).

🤖 Generated with Claude Code

[CLAassistant]

All committers have signed the CLA.

[nathanestone-alt]

Context / gauging interest 👋

Opening this to see whether session-scoped dedup is a direction you'd welcome — happy to iterate on the design or close it if it's not something you want to carry.

The problem it targets. RTK's filters compress a single command's output. But a big token sink in agentic use is orthogonal: an agent re-reads the same file or re-runs git status/ls on unchanged state and re-emits byte-identical output into context every turn. Filtering can't catch that — it has no memory across invocations. (In my own usage rtk read alone accounts for the bulk of saved tokens, and a lot of those reads are repeats.)

The approach, kept deliberately low-risk:

• Opt-in, off by default ([dedup] enabled=false) — zero behavior change unless a user turns it on. No session id (all manual invocations) ⇒ complete no-op, output byte-for-byte unchanged.
• Suppresses only a byte-identical re-emission within one Claude Code session to a one-line stub that carries a recovery command.
• Two-key safety: SHA-256 of the raw bytes is the correctness guard (changed content can never be suppressed); the command identity is only cosmetic. Every guard (disabled, no session, non-zero exit, tiny output, DB error) falls back to full output — it can only ever omit a repeat, never hide new/changed/failed output.

The one part I'd especially want your read on: the hook change. To scope by session, process_claude_payload injects --session <id> (validated [A-Za-z0-9_-]) into the rewritten command. That touches the security-sensitive rewrite path, so if you'd prefer a different mechanism (or don't want the hook to carry session state at all), I'm glad to rework it.

Fully tested (unit + an end-to-end integration test) and passing the fmt + clippy + test --all gate, reused the existing sha2 dep (no new deps), and it's split into 7 atomic commits if that's easier to review incrementally. But I'd rather hear whether the concept fits RTK before asking you to review the code in depth. No worries at all if it's not a fit.

[nathanestone-alt]

Update — added compaction protection.

Pushed a follow-up commit closing the one real correctness gap in the original design: dedup assumes the earlier full emission is still in the agent's context, which context compaction can violate (the original output gets summarized away, leaving a stub that points at content no longer present). Two layers now guard this:

• PostCompact ledger reset (primary). A new rtk hook compact handler clears the session's dedup ledger from the PostCompact payload's session_id, and rtk init registers the PostCompact hook (no matcher → fires for auto + manual). So dedup only ever suppresses within a single un-compacted context epoch. Registration is idempotent, sits alongside the existing PreToolUse hook, and uninstall strips it too; existing installs pick it up on re-running rtk init.
• Recency window (backstop). [dedup] recency_window (default 100, 0 = unlimited) refuses to suppress if the prior emission is more than N distinct emissions behind the latest — covering any context reduction that fires no compaction signal.

Also made [dedup] fields serde-defaulted so enabling is a one-liner ([dedup]\nenabled = true).

Still off by default, still fully tested (unit + integration, incl. an end-to-end read → stub → hook compact → full test) and green on fmt + clippy + test --all. Happy to hear whether the PostCompact-hook approach sits right with you, since it's the second hook this adds.

[nathanestone-alt]

Correction to my earlier framing comment: I've now measured the actual repeat rate from my tracking DB instead of inferring it from filtering stats — the earlier claim tying this to the big-read savings line was overstated (those large reads turned out not to repeat byte-identically). The PR description now has a Measured impact section with the honest numbers: the repeat pattern is real (48.5% of post-filter output tokens are byte-identical re-emissions) but the yield is second-order on top of filtering, not a multiplier. Dogfooding the branch now; will report real rtk gain dedup numbers here.