feat(edgebench): package ClawBench V2 as an EdgeBench/SForge benchmark#253
Open
reacher-z wants to merge 7 commits into
Open
feat(edgebench): package ClawBench V2 as an EdgeBench/SForge benchmark#253reacher-z wants to merge 7 commits into
reacher-z wants to merge 7 commits into
Conversation
Adds clawbench.eval.edgebench_{adapter,judge}: run ClawBench tasks through
ByteDance-Seed/EdgeBench (SForge), the two-container code-agent harness.
Mapping A ('structured_json score task', zero SForge source edits): the Work
container runs the browser agent + interceptor writing evidence/interception.json;
the agent submits evidence/ once; the Judge eval_cmd (clawbench-edgebench-judge)
re-scores it two-stage (Stage-1 interception + Stage-2 LLM judge) and prints the
SForge structured_json block. Fits SForge's offline/archive-only judge because
ClawBench's verify already scores captured interception.
- edgebench_judge.py (clawbench-edgebench-judge): host-importable, unit-tested;
reuses runner.judge.judge_request; fail-closed when judge unconfigured;
--no-judge for Stage-1 only; judge key via CLAWBENCH_JUDGE_* (SForge injects
through SFORGE_JUDGE_EXTRA_ENV).
- edgebench_adapter.py (clawbench-edgebench-adapt): emits tasks/BENCHMARK.yaml +
tasks/<id>.json (base_image/submit_paths/parser=structured_json/selection) +
per-task specs, reusing harbor_adapter discovery + build_instruction.
- docs/edgebench.md: mapping, run command, and the two open ends (a browser
Agent subclass for SForge; container build + live sforge run on a build box).
Tests: 11 (judge Stage-1/2 + fail-closed + structured_json markers; adapter
schema + layout). Full suite green; ruff + pyright clean. Verified: live
adapter emits schema-correct task JSON; judge emits a valid structured_json
block on a real evidence fixture.
…dge, my-info staging) - edgebench_judge: validate interception evidence — malformed/non-object JSON is valid:false score 0; require intercepted IS True (a truthy string 'false' no longer passes Stage-1); require a request object. Wrap judge_request in try/except so a transport exception can't suppress the structured_json block; emit sanitized error categories (never raw provider/exception text). - edgebench_adapter: sforge_task_id() maps every non-[a-z0-9_] char to '_' (dots/mixed punctuation) + collapses; raise on task_id collisions instead of silently dropping; catch OSError/ValueError in main. Stage the my-info bundle (persona + creds + resume) + extra_info into specs so the reused instruction's ./my-info promise holds; work.setup_cmds copy it in. - docs: structured_json example includes pass_rate. - tests: +10 (malformed/non-object/string-false/no-request evidence, judge exception + error sanitization, id sanitization params, dup collision, my-info staged, no-secret-in-task-json). ruff + pyright clean.
…reason leak Applies Codex round-2 findings on feat/edgebench-adapter: - HIGH runtime arming: Work setup now also copies eval-schema to /eval-schema.json (the runtime-server reads it there to arm the interceptor); docs make explicit that the browser Agent/entrypoint must start start-runtime.sh + wait for eval_interceptor_ready at container-run time (setup_cmds are build-time), with the exact bootstrap commands. - MEDIUM extra_info path-traversal: _guard_extra_info rejects absolute/'..'-escaping paths before the (unguarded) copy_extra_info stages them into the shareable benchmark. - MEDIUM reason leak: the judge's free-text reason quotes the intercepted request body (may hold creds/PII); emit generic match/mismatch summaries instead of echoing it to SForge output. - tests: +3 (reason not echoed, extra_info traversal rejected, eval-schema staged).
Applies Codex round-3 HIGH findings on feat/edgebench-adapter: - evidence path mismatch: the runtime writes interceptions to $CLAWBENCH_DATA_DIR (default /data), but SForge submits + the judge reads /app/evidence. Docs bootstrap now exports CLAWBENCH_DATA_DIR=/app/evidence before start-runtime.sh so interception.json lands where submit_paths/--evidence-dir point. - judge executable missing: the default image lacks clawbench-edgebench-judge. judge.setup_cmds now 'pip install clawbench-eval' so the console script exists in the Judge container (a base image bundling it can drop the line). - tests: +3 (judge installs clawbench, submit_paths<->eval_cmd evidence-dir consistency, eval-schema staged).
…usted evidence) Codex round-4 HIGH: the agent controls the submitted evidence archive, so the judge must not trust interception.json's 'intercepted' flag. score_evidence now RE-VERIFIES Stage-1 by matching the submitted request against task['eval_schema'] (url_pattern regex + method + const body/params), mirroring the runtime-server interceptor (_stage1_match/_const_fields_match). Forged evidence (intercepted:true with a non-matching request) now FAILS; a fabricated flag over a matching request still requires the request to actually hit the target. Fail-closed when the task has no eval_schema url_pattern. Verified: forged wrong-URL -> 0.0/FAILED; real matching request -> 1.0/PASSED. tests rewritten around a real eval_schema (+ forged/wrong-method/body-match/ missing-schema cases). 29 edgebench tests; ruff + pyright clean.
Codex round-5: - params must be derived from the request URL (like the runtime), not a forgeable submitted request['params'] field. _stage1_match now always parses query params from the URL. - tamper-resistance for agent-controlled evidence: opt-in HMAC. When CLAWBENCH_EVIDENCE_SECRET is set (shared runtime+judge, not the agent), the judge requires signature = HMAC-SHA256(secret, canonical-json(request)) and rejects unsigned/forged evidence; without the secret it runs in attestable mode (recomputed Stage-1 only). Documented the trust model. - tests: +4 (signature enforced/optional, params-from-url-not-field).
'/etc/passwd' is absolute on POSIX (rejected via the is_absolute 'relative' branch) but not on Windows (no drive letter), where _guard_extra_info rejects it as an 'escapes' containment failure instead. Relax the match to 'relative|escapes' — rejected either way — so pytest (windows-latest) passes.
Perry2004
reviewed
Jul 10, 2026
Perry2004
left a comment
Perry2004
left a comment
Collaborator
There was a problem hiding this comment.
I don't see the point of doing this integration, and I'm not familiar with this framework.
@reacher-z, if you think this is reasonable, feel free to approve & merge it.
Collaborator
Author
|
Validated against the real SForge harness code (not just the documented contract). Imported SForge's actual
So the ClawBench-side contract is confirmed correct against SForge's own loader + grader — the remaining open ends (a browser Agent for SForge, a live container |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Package ClawBench V2 as an EdgeBench/SForge benchmark
Adds
clawbench.eval.edgebench_{adapter,judge}so ClawBench tasks run through ByteDance-Seed/EdgeBench (harness: SForge), the two-container code-agent harness.Design — Mapping A ("structured_json score task"), zero SForge source edits
EdgeBench's judge is offline/archive-only; ClawBench's
verify.pyalready scores a capturedinterception.json— so it fits cleanly:evidence/interception.json; agent runssforge-submitonce.eval_cmd=clawbench-edgebench-judge→ re-scores the evidence two-stage (Stage-1 interception ∧ Stage-2 LLM judge) → prints the SForgestructured_jsonblock.parser: structured_jsonis built-in (no harness edit). Judge key viaSFORGE_JUDGE_EXTRA_ENV→CLAWBENCH_JUDGE_*.--max-submissions 1 --disable-auto-eval --disable-stop-hook(SForge's long-horizon best-of-N machinery is off for a short browser task).What's here
edgebench_judge.py(clawbench-edgebench-judge) — host-importable, unit-tested; reusesrunner.judge.judge_request; fail-closed when judge unconfigured;--no-judge= Stage-1 only.edgebench_adapter.py(clawbench-edgebench-adapt) — emitstasks/BENCHMARK.yaml+tasks/<id>.json+ per-taskspecs/, reusingharbor_adapterdiscovery.docs/edgebench.md.Verified
>>>>> Start Structured Result {…score…}block on a real evidence fixture.Two open ends (documented, not blockers to the ClawBench-side code)
Agentsubclass + factory line in the EdgeBench repo, or a Work-image entrypoint that drives the harness then submits.sforge run— needs Docker + the EdgeBench harness on a build box (as with the ProRL image, validated separately from the offline contract tests).