Skip to content

fix(llm): normalize provider token usage counts - #3884

Closed
PaleRoses wants to merge 1 commit into
vectorize-io:mainfrom
PaleRoses:upstream/provider-token-normalization
Closed

fix(llm): normalize provider token usage counts#3884
PaleRoses wants to merge 1 commit into
vectorize-io:mainfrom
PaleRoses:upstream/provider-token-normalization

Conversation

@PaleRoses

Copy link
Copy Markdown

Problem

Optional token-usage fields from OpenAI-compatible and Gemini SDK responses are not guaranteed to be concrete integers. Missing attributes on SDK-shaped mocks, proxy response objects, or version-skewed SDK models can surface as sentinel objects. Those values currently reach arithmetic and metrics unchanged: OpenAI-compatible calls can raise TypeError while subtracting reasoning tokens, and Gemini records non-numeric metric values.

Change

  • Normalize provider token counts to non-negative integers at the SDK boundary.
  • Apply the same normalization to prompt, completion, total, cached, and reasoning counts on plain and tool-call paths.
  • Keep numeric strings supported; reject booleans, negative values, absent fields, and arbitrary objects as zero.
  • Add end-to-end provider-call regressions that exercise metrics recording with sparse SDK-shaped usage objects.

Verification

The new tests fail on current main:

OpenAI: TypeError in max(0, output_tokens - thoughts_tokens)
Gemini: metrics input_tokens is a MagicMock instead of 0

After the patch:

pytest -q hindsight-api-slim/tests/test_provider_usage_normalization.py
2 passed

pytest -q \
  hindsight-api-slim/tests/test_openai_compatible_response_hardening.py \
  hindsight-api-slim/tests/test_gemini_cache.py
35 passed

ruff check <two provider modules> hindsight-api-slim/tests/test_provider_usage_normalization.py
All checks passed!

@strix-security

strix-security Bot commented Aug 29, 2026

Copy link
Copy Markdown

Strix Security Review

No security issues found.

Updated for 9b90703.


Reviewed by Strix
Re-run review · Configure security review settings

@PaleRoses PaleRoses closed this Aug 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant