fix(usage): carry cumulative cache tokens in result.usage - #786
Merged
Conversation
Test Results 1 files 1 suites 8m 9s ⏱️ Results for commit 91f18f5. |
`agent_loop_compat`'s aggregator summed only `input_tokens` and `output_tokens`. `input_tokens` counts only cache MISSES — the shared convention across providers (Anthropic natively; OpenAI-compatible ones since #785's cache-read split) — so the cumulative total silently omitted every cached token. That total is not a display nicety. `agent_server` passes this dict straight to `compute_cost`, which then bills the cached portion at nothing: a real turn (2613-token prompt, 2560 served from cache) lost 71.7% of its cost. `eval/harbor/clawcodex_agent.py` had already diagnosed the same defect from the other end — 6 tokens reported against a real ~412 K on a prompt-cached opus run — and routes around it by reading the session cost block. The fix is deliberately ADDITIVE: `cache_read_input_tokens` and `cache_creation_input_tokens` are initialized and summed alongside the existing keys, and `input_tokens` keeps its meaning. Redefining it to include the cache would have been a stream-json contract change; adding counters beside it is not, so every existing reader is untouched and only gains the ability to see the rest. The harbor fallback already computed `input + cache_read + cache_creation` from this payload — it was written for exactly this convention and was simply starved of the data. It becomes correct rather than double-counting. The cost block stays preferred (it also supplies the real dollar cost, and older clawcodex builds still need it); three comments there that described the missing counters as a permanent property are updated. Two consumers in `entrypoints.headless` came along with it: * `/goal`'s token budget summed input+output, so on a warm prefix cache it saw about 2.3% of what had been spent. It now counts every billed token. * The multi-prompt accumulator summed EVERY key generically, including the `last_*` snapshot. Those are last-wins — the live-context measure — and adding a snapshot to itself across prompts produces a number that measures nothing, which then ships in the stream-json `result` payload. They now replace. Both were inline in a long function, so they are extracted as `_accumulate_usage` and `_billed_token_total`. That is not cosmetic: the first version of these tests re-implemented the loop and passed with BOTH headless fixes reverted. The cumulative sum still must NOT be used as a live-context measure — it double-counts context because every turn re-sends the whole conversation. That half of the `C3a` comment stands; only the "drops the cache keys" half is now stale and was corrected. Completing that total made a SECOND defect reachable, fixed here too. `agent_server` priced the turn by calling `compute_cost` on the cumulative dict, but `get_pricing` selects a tier from a PER-REQUEST threshold (gpt-5.6-luna at 272K, MiniMax-M3 at 512K). Cache reads sum to roughly turns x conversation-size, so a 25-turn loop over a 15K conversation reaches 385K and prices at the long-context rate no single request came near — 1.76x the true cost, where before the change it under-reported at 0.60x. Accurate tokens are what push the aggregate over the line, so the two changes could not ship apart. The turn's cost is now a delta of `cost_tracker`'s running total, which prices each response as it arrives with the tier chosen from that one request. That also removes a duplicate cost implementation rather than patching one. Only tiered models were affected; every linear-priced model aggregated exactly. Mutation-tested: both cache sums, the dict initialization, the `last_*` branch, the billed-key tuple, both headless call sites and the server's cost path. Four mutants initially survived, and each exposed a real gap: * every fixture used `cache_creation_input_tokens: 0`, which cannot distinguish summing a value from dropping it; * every fixture ended after ONE API call, so `+=` and `=` were indistinguishable — turn count is the axis that tests accumulation, and it is the same axis the tier-crossing lives on; * the headless assertions called the extracted helpers directly, so reverting the call sites changed nothing; * a single-prompt headless test cannot separate summing from replacing, because both agree on one accumulation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ericleepi314
force-pushed
the
fix/usage-aggregator-cache-keys
branch
from
August 2, 2026 15:59
91f18f5 to
64214f1
Compare
ericleepi314
added a commit
that referenced
this pull request
Aug 2, 2026
Headline: fusion models (#771) — pair a text-only reasoning model with a vision-capable one so it can read screenshots, diagrams and code images. `deepseek-v4-pro` rejects an image content block outright, so a pasted screenshot used to end the turn; a fusion model describes the image with the second model first and hands the base model text. Verified end to end on Terminal-Bench 2.1's `code-from-image` — transcribe handwritten pseudocode from a PNG and reproduce its output — with `deepseek-v4-flash` + `openai:gpt-5.6-luna` (#787). The base model alone returns a 400 on the same image, so the pass is attributable to the fusion path rather than the base coping. Also in 1.4.0: GPT-5.6 Sol/Terra/Luna (#773); groq, cerebras, baseten and xai take the provider registry to 30 (#784); `/mode` becomes `/permissions` with a three-level picker (#768); `AskUserQuestion` renders a real picker instead of returning JSON to the model (#774); the OpenAI provider picks its wire protocol from the model rather than the auth mode (#783); cached prompt tokens bill at the cache rate (#785, #786); headless runs stop reporting a cut-short run as success (#777–#782). Version bumped in all five spots (pyproject, install.sh INSTALLER_VERSION, gatewayClient CLAWCODEX_VERSION, src/__init__.py fallback, uv.lock). CHANGELOG `[Unreleased]` covered only through #773 and was backfilled with #774–#787; PR citations added to the pre-existing entries so coverage is checkable. #766 is docs-only and deliberately uncited. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves the item deferred from #785.
The defect
agent_loop_compat's aggregator summed onlyinput_tokensandoutput_tokens.input_tokenscounts cache misses only — the shared convention across providers (Anthropic natively; OpenAI-compatible ones since #785's cache-read split) — so the cumulative total silently omitted every cached token.That total is not a display nicety:
agent_server.py:4850passes this dict straight tocompute_cost. With the cache keys absent, the cached portion is billed at nothing: a real turn (2613-token prompt, 2560 served from cache) lost 71.7% of its cost.eval/harbor/clawcodex_agent.pyhad already diagnosed the same defect from the other end, documenting 6 tokens reported against a real ~412 K on a prompt-cached opus run. It routes around it by reading the session cost block./goal's token budget summed input+output, so on a warm prefix cache it saw about 2.3% of what had been spent.The fix is additive, on purpose
cache_read_input_tokensandcache_creation_input_tokensare initialized and summed alongside the existing keys, andinput_tokenskeeps its meaning. Redefining it to include the cache would have been a stream-json contract change; adding counters beside it is not, so every existing reader is untouched and only gains the ability to see the rest.The cumulative sum still must not be used as a live-context measure — it double-counts context because every turn re-sends the whole conversation. That half of the
C3acomment stands; only the "drops the cache keys" half was stale.Harbor becomes correct rather than double-counting
The fallback path already computed
input + cache_read + cache_creationfrom this payload — it was written for exactly this convention and was simply starved of the data. The cost block stays preferred (it also supplies the real dollar cost, and older clawcodex builds still need it); three comments there describing the missing counters as a permanent property are updated.Two headless consumers
/goal's budget now counts every billed token.last_*snapshot. Those are last-wins — the live-context measure — and adding a snapshot to itself across prompts produces a number that measures nothing, which then ships in the stream-jsonresultpayload. They now replace.Both were inline in a long function and are extracted as
_accumulate_usage/_billed_token_total. That is not cosmetic: the first version of these tests re-implemented the loop and passed with both headless fixes reverted.Verification
last_*branch, the billed-key tuple. One mutant initially survived because every fixture usedcache_creation_input_tokens: 0, which cannot distinguish summing a value from dropping it — the fixtures now carry two distinct non-zero cache counts.🤖 Generated with Claude Code