fix(tbench2_env): canonical test.sh scoring, real timeouts, task-image workdir

[nblintao]

Summary

Makes local-mode Tbench2Environment faithful to real Terminal-Bench-2 semantics: evaluate scores with each task's official tests/test.sh (pinned uvx toolchain, /logs/verifier/reward.txt), command execution honors real time budgets, and commands run in the task image's own WORKDIR. Follow-up to #503, which fixed evaluate always returning 0 — this makes the returned reward match the official TB2 verifier.

Details:

• Canonical scoring: bare pytest tests/ (current behavior) skips the toolchain the official harness pins (uvx: Python 3.13, pytest 8.4.1, ctrf) and mis-scores tasks that depend on it. _evaluate_task now runs tests/test.sh when the task ships it; the bare-pytest fallback remains for custom task dirs, upgraded to prefer uvx --with pytest==8.4.1 so it carries its own toolchain too.
• Real timeouts: camel's shell_exec has a per-call default of 20s and ignores the constructor timeout, silently truncating agent commands and evaluations (e.g. circuit-fibsqrt declares a 3600s verifier budget and was cut mid-test, scoring a false 0). Agent execs now pass command_timeout_s (overridable via TB2_COMMAND_TIMEOUT_S); evaluate passes the task's own [verifier].timeout_sec.
• Task workdir: task images pin their own WORKDIR and both agents and oracle solutions assume commands run there — it is not always /app (fix-git: /app/personal-site, prove-plus-comm: /workspace). The shell cwd now resolves from the task's environment/Dockerfile (TB2_TASK_WORKDIR overrides), falling back to the task source dir for plain local mode.
• Drop install_dependencies=['pytest']: evaluation now carries its own pytest via uvx; when the server runs with CAMEL_RUNTIME=true, camel's .initial_env venv then also stops shadowing the task image's native python on PATH.

Type of Change

• Bug fix

Alignment Checklist

• I have read .claude/docs/PRINCIPLES.md and this PR aligns with our principles (notably: rewards are computed inside the environment, per RFC 002)
• I have checked .claude/docs/INVARIANTS.md and no invariants are violated (Gym API signatures unchanged; server-internal changes only)
• I have run bash .claude/hooks/lint.sh — the touched file is clean (pre-existing failures in other envs are untouched)

RFC Status

• Not required (bug fix, docs, minor refactoring)

Test Plan

Validated on real per-task TB2 containers (official task images), scoring official solutions under the official oracle convention:

• Full 89-task golden sweep: 82/89 score 1.0 with 0 infra errors; the 7 remaining failures trace to upstream solution drift in terminal-bench-2 itself (apt version pins gone from Debian, missing setuptools for pystan, etc. — happy to share the per-task evidence).
• Each fix maps to tasks that flip from a false 0.0 to 1.0: circuit-fibsqrt/compile-compcert (timeouts), fix-git/prove-plus-comm (workdir), plus correct scoring across the suite via test.sh.
• 11-task concurrent regression on the final revision: 11/11 pass.

Note: the canonical eval uses the official global paths (/tests, /logs/verifier/reward.txt), which assumes one concurrent session per container — consistent with TB2's one-task-per-container model (and with one-env-one-trajectory); flagging for reviewers running multi-session servers against a single container.

Claude Code Review

/alignment-review run on this change: Tier 1 (format) fixed; flags addressed during review — fallback now carries its own pytest toolchain; workdir resolution centralized server-side so all deployment modes agree. No principle or RFC conflicts identified.

---

Note

Medium Risk
Changes reward computation and shell cwd/timeouts for all local TB2 sessions; canonical eval uses shared /tests paths and requires the lock when multiple sessions share one container.

Overview
Aligns local-mode Tbench2Environment with official Terminal-Bench 2 verifier behavior so rewards and execution context match what agents and the harness expect.

Evaluation now prefers each task’s tests/test.sh when present: tests are staged under /tests, the harness runs from the resolved task workdir, and reward is read from /logs/verifier/reward.txt instead of inferring pass/fail from bare pytest. A thread lock serializes canonical evals so concurrent sessions on one server do not clash on /tests or the reward file. Tasks without test.sh keep a pytest fallback, now preferring uvx --with pytest==8.4.1 and using the agent workdir; verifier runs use [verifier].timeout_sec from task.toml.

Shell execution passes timeout=self.command_timeout_s into shell_exec (camel’s per-call 20s default was truncating long commands). command_timeout_s defaults via TB2_COMMAND_TIMEOUT_S when the env is constructed with no args.

Working directory is resolved on reset from TB2_TASK_WORKDIR, else the task environment/Dockerfile WORKDIR (ignored when it overlaps the server install tree), else the task source dir—replacing always using the task folder as cwd. install_dependencies=['pytest'] is removed from TerminalToolkit because scoring brings its own toolchain.

Tests add coverage for Dockerfile workdir selection and server-tree rejection.

^{Reviewed by Cursor Bugbot for commit f1b0b8e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Multi-stage WORKDIR parsing is wrong

Medium Severity

_task_image_workdir keeps the last WORKDIR anywhere in the Dockerfile and claims that equals the final stage. Docker resets WORKDIR per FROM, so a builder-stage WORKDIR is used when the final stage omits one, sending agents and tests/test.sh to the wrong cwd.

 

^{Reviewed by Cursor Bugbot for commit 85343bd. Configure here.}

[nblintao]

Checked all 89 official tasks: exactly one is multi-stage (financial-document-processor) and its final stage pins WORKDIR /app, so last-wins parses every official task correctly. Leaving the parser as is — TB2_TASK_WORKDIR covers exotic custom Dockerfiles.

[cursor]

Timeout can drop a written reward

Medium Severity

Canonical scoring only trusts the trailing __TB2_REWARD__: echo from shell_exec output. The same timeout_s also covers staging, test.sh, tail, and that echo, so if camel truncates after test.sh has already written /logs/verifier/reward.txt, the reward is ignored and the episode scores 0.0.

 

^{Reviewed by Cursor Bugbot for commit 85343bd. Configure here.}

[nblintao]

Working as intended: timeout_s is the task's declared verifier budget, and blowing it scores 0 in the official TB2 harness too. Recovering a reward written mid-timeout would deviate from that semantics.

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 3 total unresolved issues (including 2 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit f1b0b8e. Configure here.}

[cursor]

Fallback pytest path left broken

Medium Severity

Removing install_dependencies=['pytest'] drops the only mechanism that ensured pytest was available for the bare-pytest fallback, while that fallback still does python -m pytest when uvx is missing. Custom task dirs without tests/test.sh on images that lack both uvx and a preinstalled pytest will fail closed with reward 0.

Additional Locations (1)

• envs/tbench2_env/server/tbench2_env_environment.py#L389-L396

 

^{Reviewed by Cursor Bugbot for commit f1b0b8e. Configure here.}