fix(query): send reasoning effort on the OpenAI-compatible wire - #776
Merged
Conversation
`--effort` was a silent no-op for every OpenAI-compatible provider on the
headless `-p` path — the path the terminal-bench harness drives.
`_call_model_sync` emitted the level only inside its `is_anthropic` branch,
as `output_config.effort`; the OpenAI-compatible wire takes a top-level
`reasoning_effort` body field, and nothing emitted it. Captured against a
local capture server: `--effort max --provider openrouter` produced a request
body of {messages, model, stream, stream_options, tools} — no effort field of
any kind, no error, no log line.
This invalidates the effort setting on past non-Anthropic evals. The in-repo
`eval/harbor/jobs/tb21-deepseek-max` job is configured `effort=max`, and
DeepSeek is OpenAI-compatible, so it actually ran at DeepSeek's default. Any
comparison against a new run at a real effort level needs that baseline
re-run.
Adds the missing branch, and drops the `xhigh` clamp on this wire:
`_model_supports_xhigh_effort` is an allowlist of ANTHROPIC model-name
substrings, so it matched nothing here and silently downgraded every `xhigh`
to `high`. `xhigh` is a first-class OpenAI level — their reasoning guide
lists none|minimal|low|medium|high|xhigh|max and recommends `xhigh` for long
agentic runs. Probed against `openai/gpt-5.6-luna` on 2026-07-31: the whole
ladder is accepted AND honored, with reasoning tokens rising 148 → 154 → 266
→ 516 across low/medium/high/max.
Deletes `_EffortProvider`. Fixing the above collided with it: the wrapper
injected the same field with `setdefault` while routing returned
`thinking_effort=None`, so query.py filled the key from `settings.effort`
first and the wrapper no-op'd — an explicit session `/effort max` went out as
`medium`, inverting the documented precedence (explicit beats persisted).
`_turn_effort_routing` now just forwards the level, unwrapped, for both
families. One level, one injection site.
BEHAVIOR CHANGE worth knowing: `compact.py` calls `context.provider.chat_async()`
directly, bypassing the wire boundary, and the deleted wrapper's `chat_async`
override was what carried effort there. Compaction summaries on
OpenAI-compatible providers now run at the provider default rather than the
session's `/effort`. Safe direction, and it matches the Anthropic side, which
never got `output_config` on that path either.
Also registers `openai/gpt-5.6-luna` (the OpenRouter id, which is what
reaches the provider when Harbor runs the model — it splits `--model` on the
first slash only). MODEL_CONFIGS is keyed by bare name, so that id matched
nothing and fell back to DEFAULT_CONTEXT_WINDOW (200K) against a 1M-class
window — auto-compact at a fifth of capacity. The bare `gpt-5.6-*` rows from
#773 are left alone, including their ordering-based namespace handling; the
resolver still does NOT strip a `<vendor>/` prefix ("decision #1"), and
`get_model_config`'s docstring now records why, with both failure modes
named. Pricing gains Luna's rates including the >272K long-context tier,
since a 1M-window model on a benchmark will cross it and cost is a number
the eval reports.
Verified: full suite green; effort ladder confirmed on the wire before and
after the rebase; two live tasks completed at effort=max and independently
checked. Both fixes and every new guard mutation-tested.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
--effortwas a silent no-op for every OpenAI-compatible provider on the headless-ppath — the path the terminal-bench harness drives._call_model_syncemitted the level only inside itsis_anthropicbranch, asoutput_config.effort. The OpenAI-compatible wire takes a top-levelreasoning_effortbody field, and nothing emitted it. Captured against a local capture server:No effort field of any kind. No error, no log line — the job config would faithfully record
effort=maxwhile the request carried nothing.This invalidates the effort setting on past non-Anthropic evals.
eval/harbor/jobs/tb21-deepseek-maxis configuredeffort=max; DeepSeek is OpenAI-compatible; it ran at DeepSeek's default. Any comparison against a new run at a real effort level needs that baseline re-run.What's in here
The fix. Adds the missing branch, and drops the
xhighclamp on this wire —_model_supports_xhigh_effortis an allowlist of Anthropic model-name substrings, so it matched nothing here and silently downgraded everyxhightohigh.xhighis a first-class OpenAI level (their guide listsnone|minimal|low|medium|high|xhigh|maxand recommendsxhighfor long agentic runs).Deletes
_EffortProvider. Fixing the above collided with it: the wrapper injected the same field withsetdefaultwhile routing returnedthinking_effort=None, so query.py filled the key fromsettings.effortfirst and the wrapper no-op'd. An explicit/effort maxwent out asmedium— inverting the documented precedence._turn_effort_routingnow just forwards the level for both families. One level, one injection site.Registers
openai/gpt-5.6-luna— the OpenRouter id, which is what reaches the provider when Harbor runs the model (it splits--modelon the first slash only, soopenrouter/openai/gpt-5.6-luna→--provider openrouter --model openai/gpt-5.6-luna).MODEL_CONFIGSis keyed by bare name, so that id matched nothing and fell back to 200K against a 1M-class window. The baregpt-5.6-*rows from #773 are left alone, including their ordering-based namespace handling. The resolver still does not strip a<vendor>/prefix ("decision #1");get_model_config's docstring now records why, with both failure modes named.Pricing gains Luna's rates including the
>272Klong-context tier, since a 1M-window model on a benchmark will cross it and cost is a number the eval reports.compact.pycallscontext.provider.chat_async()directly, bypassing the wire boundary, and the deleted wrapper'schat_asyncoverride was what carried effort there. Compaction summaries on OpenAI-compatible providers now run at the provider default rather than the session's/effort. Safe direction, and it matches the Anthropic side, which never gotoutput_configon that path either.Verification
max→max,xhigh→xhigh, absent when unset; Anthropic still getsoutput_configand neverreasoning_effort.148 → 154 → 266 → 516across low/medium/high/max onopenai/gpt-5.6-luna. A 200 OK would not have shown this.effort=max, independently verified: a git stage+commit, and writingrun_length.py+ pytest tests (5 passed on a clean re-run). Cost accounting exact: 74,179 in + 2,801 out → $0.0090985.mainwith this branch's changes stashed.Reviewed by the
criticsubagent across three rounds. It caught the precedence inversion above, and a differential test of mine that was a tautology (it passed with the code under test deleted) — both fixed; the replacement asserts hardcoded expectations and guards itself withassertNotIn(model_id, MODEL_CONFIGS).Running the model
See
eval/harbor/RUN_LUNA_TB21.md. Containers install from PyPI orgit+, so benchmarking a non-Anthropic model at a specific effort needs--ak source=pinned at this commit or later.🤖 Generated with Claude Code