Skip to content

fix(query): send reasoning effort on the OpenAI-compatible wire - #776

Merged
ericleepi314 merged 1 commit into
mainfrom
feat/openai-compat-reasoning-effort
Jul 31, 2026
Merged

fix(query): send reasoning effort on the OpenAI-compatible wire#776
ericleepi314 merged 1 commit into
mainfrom
feat/openai-compat-reasoning-effort

Conversation

@ericleepi314

Copy link
Copy Markdown
Collaborator

The bug

--effort was a silent no-op for every OpenAI-compatible provider on the headless -p path — the path the terminal-bench harness drives.

_call_model_sync emitted the level only inside its is_anthropic branch, as output_config.effort. The OpenAI-compatible wire takes a top-level reasoning_effort body field, and nothing emitted it. Captured against a local capture server:

--effort max --provider openrouter
  → body: {messages, model, stream, stream_options, tools}

No effort field of any kind. No error, no log line — the job config would faithfully record effort=max while the request carried nothing.

This invalidates the effort setting on past non-Anthropic evals. eval/harbor/jobs/tb21-deepseek-max is configured effort=max; DeepSeek is OpenAI-compatible; it ran at DeepSeek's default. Any comparison against a new run at a real effort level needs that baseline re-run.

What's in here

The fix. Adds the missing branch, and drops the xhigh clamp on this wire — _model_supports_xhigh_effort is an allowlist of Anthropic model-name substrings, so it matched nothing here and silently downgraded every xhigh to high. xhigh is a first-class OpenAI level (their guide lists none|minimal|low|medium|high|xhigh|max and recommends xhigh for long agentic runs).

Deletes _EffortProvider. Fixing the above collided with it: the wrapper injected the same field with setdefault while routing returned thinking_effort=None, so query.py filled the key from settings.effort first and the wrapper no-op'd. An explicit /effort max went out as medium — inverting the documented precedence. _turn_effort_routing now just forwards the level for both families. One level, one injection site.

Registers openai/gpt-5.6-luna — the OpenRouter id, which is what reaches the provider when Harbor runs the model (it splits --model on the first slash only, so openrouter/openai/gpt-5.6-luna--provider openrouter --model openai/gpt-5.6-luna). MODEL_CONFIGS is keyed by bare name, so that id matched nothing and fell back to 200K against a 1M-class window. The bare gpt-5.6-* rows from #773 are left alone, including their ordering-based namespace handling. The resolver still does not strip a <vendor>/ prefix ("decision #1"); get_model_config's docstring now records why, with both failure modes named.

Pricing gains Luna's rates including the >272K long-context tier, since a 1M-window model on a benchmark will cross it and cost is a number the eval reports.

⚠️ Behavior change worth knowing

compact.py calls context.provider.chat_async() directly, bypassing the wire boundary, and the deleted wrapper's chat_async override was what carried effort there. Compaction summaries on OpenAI-compatible providers now run at the provider default rather than the session's /effort. Safe direction, and it matches the Anthropic side, which never got output_config on that path either.

Verification

  • Wire, before and after the rebase: maxmax, xhighxhigh, absent when unset; Anthropic still gets output_config and never reasoning_effort.
  • Effort is honored, not just accepted — reasoning tokens rise 148 → 154 → 266 → 516 across low/medium/high/max on openai/gpt-5.6-luna. A 200 OK would not have shown this.
  • Two live tasks at effort=max, independently verified: a git stage+commit, and writing run_length.py + pytest tests (5 passed on a clean re-run). Cost accounting exact: 74,179 in + 2,801 out → $0.0090985.
  • Full suite green. The 27 MCP-auth failures in a fresh worktree venv are pre-existing — reproduced identically on pristine main with this branch's changes stashed.
  • Both fixes and every new guard mutation-tested: disabling each makes the intended tests fail, and restoring makes them pass.

Reviewed by the critic subagent across three rounds. It caught the precedence inversion above, and a differential test of mine that was a tautology (it passed with the code under test deleted) — both fixed; the replacement asserts hardcoded expectations and guards itself with assertNotIn(model_id, MODEL_CONFIGS).

Running the model

See eval/harbor/RUN_LUNA_TB21.md. Containers install from PyPI or git+, so benchmarking a non-Anthropic model at a specific effort needs --ak source= pinned at this commit or later.

🤖 Generated with Claude Code

`--effort` was a silent no-op for every OpenAI-compatible provider on the
headless `-p` path — the path the terminal-bench harness drives.
`_call_model_sync` emitted the level only inside its `is_anthropic` branch,
as `output_config.effort`; the OpenAI-compatible wire takes a top-level
`reasoning_effort` body field, and nothing emitted it. Captured against a
local capture server: `--effort max --provider openrouter` produced a request
body of {messages, model, stream, stream_options, tools} — no effort field of
any kind, no error, no log line.

This invalidates the effort setting on past non-Anthropic evals. The in-repo
`eval/harbor/jobs/tb21-deepseek-max` job is configured `effort=max`, and
DeepSeek is OpenAI-compatible, so it actually ran at DeepSeek's default. Any
comparison against a new run at a real effort level needs that baseline
re-run.

Adds the missing branch, and drops the `xhigh` clamp on this wire:
`_model_supports_xhigh_effort` is an allowlist of ANTHROPIC model-name
substrings, so it matched nothing here and silently downgraded every `xhigh`
to `high`. `xhigh` is a first-class OpenAI level — their reasoning guide
lists none|minimal|low|medium|high|xhigh|max and recommends `xhigh` for long
agentic runs. Probed against `openai/gpt-5.6-luna` on 2026-07-31: the whole
ladder is accepted AND honored, with reasoning tokens rising 148 → 154 → 266
→ 516 across low/medium/high/max.

Deletes `_EffortProvider`. Fixing the above collided with it: the wrapper
injected the same field with `setdefault` while routing returned
`thinking_effort=None`, so query.py filled the key from `settings.effort`
first and the wrapper no-op'd — an explicit session `/effort max` went out as
`medium`, inverting the documented precedence (explicit beats persisted).
`_turn_effort_routing` now just forwards the level, unwrapped, for both
families. One level, one injection site.

BEHAVIOR CHANGE worth knowing: `compact.py` calls `context.provider.chat_async()`
directly, bypassing the wire boundary, and the deleted wrapper's `chat_async`
override was what carried effort there. Compaction summaries on
OpenAI-compatible providers now run at the provider default rather than the
session's `/effort`. Safe direction, and it matches the Anthropic side, which
never got `output_config` on that path either.

Also registers `openai/gpt-5.6-luna` (the OpenRouter id, which is what
reaches the provider when Harbor runs the model — it splits `--model` on the
first slash only). MODEL_CONFIGS is keyed by bare name, so that id matched
nothing and fell back to DEFAULT_CONTEXT_WINDOW (200K) against a 1M-class
window — auto-compact at a fifth of capacity. The bare `gpt-5.6-*` rows from
#773 are left alone, including their ordering-based namespace handling; the
resolver still does NOT strip a `<vendor>/` prefix ("decision #1"), and
`get_model_config`'s docstring now records why, with both failure modes
named. Pricing gains Luna's rates including the >272K long-context tier,
since a 1M-window model on a benchmark will cross it and cost is a number
the eval reports.

Verified: full suite green; effort ladder confirmed on the wire before and
after the rebase; two live tasks completed at effort=max and independently
checked. Both fixes and every new guard mutation-tested.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ericleepi314
ericleepi314 merged commit aa338fb into main Jul 31, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant