[Epic] Observability & Metrics — open issues on stats, token usage, cost, hooks and telemetry

[Sayt-0]

Several open issues in docker-agent address the same underlying need: being able to observe, measure and introspect what the agent does at runtime — token usage, LLM call costs, tool call stats, and lifecycle hook ordering. They were filed independently with no shared label and no common design connecting them.

This epic is the single reference to group them and drive a coherent solution.

Open issues in scope

Token usage & cost in hooks

• Expose per-turn token usage and cost in the after_llm_call hook payload #2948 — Expose per-turn token usage and cost in the after_llm_call hook payload
  The hook fires after every LLM call but does not include how many tokens were consumed or what it cost, making it impossible to build cost-aware tooling on top of the hook system.

Hook lifecycle & ordering

• decide StreamStopped ordering vs session-end hooks and document teardown trade-offs from #3070 #3074 — Decide StreamStopped ordering vs session-end hooks and document teardown trade-offs
  It is currently undefined whether StreamStopped fires before or after session-end hooks, making telemetry and cleanup logic unreliable.

Run statistics & performance metrics

• Stats on the usage along with performance + interaction summary ->success rate, tool calls etc #1345 — Stats on usage/performance: success rate, tool calls, interaction summary
  There is no structured summary of what happened during a run (tool call count, success/failure rate, wall-clock duration, etc.).

Tool call visibility & latency

• Improve visibility of long-running / externally-blocked MCP tool calls #2079 — Improve visibility of long-running / externally-blocked MCP tool calls
  When an MCP tool call takes a long time or blocks on an external system, docker-agent gives no feedback on duration or status.

Implementation order

1. decide StreamStopped ordering vs session-end hooks and document teardown trade-offs from #3070 #3074 — Define teardown order — design decision and docs only, no large code change; unblocks reliable telemetry for everything that follows.
2. Expose per-turn token usage and cost in the after_llm_call hook payload #2948 — Complete after_llm_call hook payload — add input_tokens, output_tokens, reasoning_tokens, cost_usd; small, self-contained, immediately useful for external tooling.
3. Improve visibility of long-running / externally-blocked MCP tool calls #2079 — Live tool call status — show elapsed time and current status for in-flight MCP tool calls in the TUI and hook events; builds on the hook ordering defined in step 1.
4. Stats on the usage along with performance + interaction summary ->success rate, tool calls etc #1345 — RunSummary structure — aggregate tokens, cost, tool call count, success/failure count, wall-clock duration into a structured end-of-session summary; builds on all three previous steps.

[Sayt-0]

All issues completed. I close this now.