Skip to content

fix(providers): retry mid-stream transport drops on the OpenAI-compat wire - #757

Merged
ericleepi314 merged 1 commit into
mainfrom
fix/openai-compat-stream-drop-retry
Jul 28, 2026
Merged

fix(providers): retry mid-stream transport drops on the OpenAI-compat wire#757
ericleepi314 merged 1 commit into
mainfrom
fix/openai-compat-stream-drop-retry

Conversation

@ericleepi314

Copy link
Copy Markdown
Collaborator

A connection that dies mid-response carries no HTTP status and no salvageable output, so re-issuing is the right call. The Anthropic provider has done this since #747. The OpenAI-compatible provider had no equivalent — no retry, no attempt loop — so one dropped chunk ended the whole agent run.

Measured cost

terminal-bench 2.1, deepseek-v4-pro at effort=max (eval/harbor/jobs/tb21-deepseek-max-1), scoring 0.449 where the same harness scores 0.809 on opus-5:

peer closed connection without sending complete message body
(incomplete chunked read)

ended 8 of 89 trials — 16% of every failure. regex-chess died this way after 391 seconds of successful work. Identical benchmark, identical error string, identical harness; only the wire differed.

Failure breakdown of that run for context:

count cause
16 (33%) timeout / killed
15 (31%) ran to completion, wrong answer
9 (18%) 400 image_url unsupported (separate bug)
8 (16%) stream drop — this PR
1 (2%) 429

What changed

  • src/providers/stream_retry.py (new) holds the classifier and marker list, moved out of anthropic_provider. Dependency-free so openai_compatible can import it without pulling in the Anthropic SDK or creating a cycle. anthropic_provider re-exports under the original private names, so its call sites and test_stream_watchdog keep resolving.
  • OpenAICompatibleProvider.chat_stream_response is now a retry wrapper over _stream_attempt (the previous body, unchanged).

Retry discipline, mirroring #747:

  • Bounded at one re-issue.
  • Never retries anything carrying an HTTP status. That guard matters here specifically: the same run has a real image_url 400 (9 trials, separate bug), and retrying it would buy a second identical rejection.
  • Skipped once any text or reasoning reached the caller, or the transcript would show that prefix twice. Harbor runs --print --output-format stream-json without --include-partial-messages, so both callbacks are None in a scored trial and the retry always fires.
  • Re-raises the original exception on the last attempt.

Not claimed

That this recovers all 8 trials. One retry covers a blip, not a connection that reliably dies at the same point. If drops persist through the retry, that points at deepseek-v4-pro's max_output_tokens=8_192 against DeepSeek's documented "reasoning_effort=max requires the longer output window" — the next thing to check.

Tests

12 new: retry-then-succeed, bounded re-raise of the original, status-bearing errors never retried, both callback paths suppressing the retry, and shared-predicate parity across wires. Revert-sensitivity verified — max_attempts = 1 fails exactly the two retry tests. Full suite 8853 passed, 0 failures.

🤖 Generated with Claude Code

… wire

A connection that dies mid-response carries no HTTP status and no
salvageable output, so re-issuing is the right call. The Anthropic provider
has done this since #747. The OpenAI-compatible provider had no equivalent
-- no retry, no attempt loop -- so one dropped chunk ended the whole agent
run.

Measured cost, terminal-bench 2.1 with deepseek-v4-pro at effort=max
(2026-07-27, eval/harbor/jobs/tb21-deepseek-max-1, score 0.449 vs the same
harness scoring 0.809 on opus-5):

    peer closed connection without sending complete message body
    (incomplete chunked read)

ended 8 of 89 trials -- 16% of every failure. regex-chess died this way
after 391 seconds of successful work. Identical benchmark, identical error
string, identical harness; only the wire differed.

What changed
------------------------------------------------------------------
* src/providers/stream_retry.py (new) holds the classifier and its marker
  list, moved out of anthropic_provider. Dependency-free so
  openai_compatible can import it without pulling in the Anthropic SDK or
  creating a cycle. anthropic_provider re-exports it under the original
  private names, so its call sites and test_stream_watchdog keep resolving.
* OpenAICompatibleProvider.chat_stream_response is now a retry wrapper over
  _stream_attempt (the previous body, unchanged).

Retry discipline, mirroring #747:
* Bounded at one re-issue.
* NEVER retries anything carrying an HTTP status. That guard matters here
  specifically: the same DeepSeek run has a real image_url 400 (9 trials,
  a separate bug), and retrying it would just buy a second identical
  rejection.
* Skipped once any text or reasoning has reached the caller, or the
  transcript would show that prefix twice. Harbor runs --print
  --output-format stream-json without --include-partial-messages, so both
  callbacks are None in a scored trial and the retry always fires.
* Re-raises the ORIGINAL exception on the last attempt, so callers see the
  real cause rather than a retry wrapper.

Not claimed: that this recovers all 8 trials. One retry covers a blip, not
a connection that reliably dies at the same point -- if drops persist
through the retry, that points at deepseek-v4-pro's max_output_tokens=8_192
against DeepSeek's documented "reasoning_effort=max needs a longer output
window", which is the next thing to check.

Tests: 12 new, covering retry-then-succeed, bounded re-raise of the
original, status-bearing errors never retried, and both callback paths
suppressing the retry. Revert-sensitivity verified: max_attempts=1 fails
exactly the two retry tests. Full suite 8853 passed, 0 failures.
@ericleepi314
ericleepi314 merged commit eac5c31 into main Jul 28, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant