benchmarks: Harbor Terminal-Bench harness for Buzz agent-team orchestration (harbor-buzz-orchestra)

.github/workflows/benchmark-harbor.yml

@@ -0,0 +1,39 @@
+name: Harbor Buzz Orchestra
+
+on:
+  push:
+    branches: [main]
+    paths:
+      - "benchmarks/harbor-buzz-orchestra/**"
+      - ".github/workflows/benchmark-harbor.yml"
+  pull_request:
+    paths:
+      - "benchmarks/harbor-buzz-orchestra/**"
+      - ".github/workflows/benchmark-harbor.yml"
+
+permissions:
+  contents: read
+
+jobs:
+  test:
+    name: Python tests and lint
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+      - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
+      - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
+        with:
+          python-version: "3.12"
+          cache: pip
+      - name: Test adapter
+        working-directory: benchmarks/harbor-buzz-orchestra
+        run: |
+          python -m pip install --disable-pip-version-check -e ".[dev]"
+          pytest -q
+          ruff check .
+      - name: Test provisioner
+        working-directory: benchmarks/harbor-buzz-orchestra/testbed
+        run: |
+          python -m pip install --disable-pip-version-check -e ".[dev]"
+          pytest -q
+          ruff check .

Justfile

@@ -747,3 +747,17 @@ goose-bg relay="ws://localhost:3000" agents="1" heartbeat="0" prompt="" key="$BU
     source ./scripts/_goose-env.sh "{{relay}}" "{{key}}" "{{agents}}" "{{heartbeat}}" "{{prompt}}"
     screen -dmS goose-agent-{{agents}} bash -c "$(printf '%q ' env "${env_args[@]}") ./target/release/buzz-acp"
     echo "Agent running in screen session 'goose-agent-{{agents}}'. Attach with: screen -r goose-agent-{{agents}}"
+
+# ─── Benchmarking ─────────────────────────────────────────────────────────────
+
+# Run the Buzz orchestra benchmark — leaderboard-eligible by default (TB 2.1, k=5, Sonnet+Haiku). Stands up its own Docker stack; --gui opens a live spectator desktop app; other flags pass to benchmark.py (--dataset/--path, --include-task, --attempts, --manifest, --dry-run, ...)
+benchmark *ARGS:
+    #!/usr/bin/env bash
+    set -euo pipefail
+    export PATH="{{justfile_directory()}}/bin:$PATH"
+    uv run --project benchmarks/harbor-buzz-orchestra/testbed \
+        benchmarks/harbor-buzz-orchestra/scripts/benchmark.py {{ARGS}}
+
+# Stop the benchmark Docker stack (state and channels are kept)
+benchmark-down:
+    docker compose --project-name buzz-benchmark down

benchmarks/harbor-buzz-orchestra/.gitignore

@@ -0,0 +1,6 @@
+.venv/
+__pycache__/
+*.py[cod]
+.pytest_cache/
+.benchmark/
+jobs/

benchmarks/harbor-buzz-orchestra/README.md

@@ -0,0 +1,129 @@
+# Harbor Buzz Orchestra
+
+A stock-Harbor custom agent that runs a manifest-defined team through the real
+Buzz stack. Harbor sees one `BuzzOrchestraAgent`; behind that adapter, one
+orchestrator and N workers coordinate over the production relay/Postgres.
+Each agent runs *inside* the Harbor task container as the same
+`buzz-acp` → `buzz-agent` → `buzz-dev-mcp` process tree the desktop app
+launches: the production MCP toolset (shell, file tools, todo) with the
+`buzz` CLI on the shell's PATH. No Harbor fork or patch is required.
+
+## Define the team
+
+The manifest is the benchmark condition. Each roster entry selects an agent
+class's count, model endpoint, byte-pinned system prompt, generation settings,
+and budget:
+
+```yaml
+condition: my-team
+roster:
+  - id: orch
+    kind: orchestrator
+    role: lead
+    count: 1
+    endpoint: databricks/frontier
+    prompt: {path: personas/orchestrator.md, sha256: <sha256>}
+    generation: {max_output_tokens: 4096, context_window_tokens: 128000}
+  - id: worker
+    kind: worker
+    role: implementer
+    count: 4
+    endpoint: databricks/fast-worker
+    prompt: {path: personas/worker.md, sha256: <sha256>}
+    generation: {max_output_tokens: 4096, context_window_tokens: 128000}
+```
+
+`endpoint_config` maps those endpoint names to providers, URLs, and API-key
+environment variables. The adapter contains no fixed roster or model.
+
+## Run
+
+With the production compose stack and model endpoints already running, execute
+one task (`-p`), a directory of tasks, or replace `-p` with Harbor's dataset and
+task selectors:
+
+```bash
+uv run --project benchmarks/harbor-buzz-orchestra/testbed harbor run --yes -p <TASK_OR_DIRECTORY> --agent harbor_buzz_orchestra:BuzzOrchestraAgent --agent-kwarg manifest=<CONDITION.yaml> --agent-kwarg provisioner_factory=harbor_buzz_testbed:provisioner_from_dict --agent-kwarg provisioner_config=<PROVISIONER.json> --agent-kwarg endpoint_config=<ENDPOINTS.json> --agent-kwarg artifact_root=benchmarks/harbor-buzz-orchestra --agent-kwarg buzz_acp_binary=<LINUX_BIN>/buzz-acp --agent-kwarg buzz_agent_binary=<LINUX_BIN>/buzz-agent --agent-kwarg buzz_dev_mcp_binary=<LINUX_BIN>/buzz-dev-mcp --agent-kwarg buzz_cli_binary=target/debug/buzz --agent-kwarg run_id="bench-$(date -u +%Y%m%dT%H%M%SZ)" --agent-timeout-multiplier 15 --n-concurrent 1
+```
+
+`buzz_acp_binary`/`buzz_agent_binary`/`buzz_dev_mcp_binary` must be **Linux**
+builds matching the task image architecture — they are uploaded into each task
+container (`just benchmark` cross-builds them automatically; musl-static, so
+any Linux base image works). `buzz_cli_binary` is the **host** CLI the harness
+uses to act as the trial user.
+
+`--n-concurrent 1` is the safe laptop setting for a serialized local model; it
+is not an orchestration requirement. Some TB graders install dependencies from
+public package registries at verification time — run benchmarks off networks
+that block those installs (e.g. corporate VPNs).
+
+Each trial gets fresh keys and a private Buzz channel. The provisioner archives
+rather than deletes that channel, leaving the relay/Postgres event timeline
+and the per-agent acp/agent logs (downloaded into the trial's `buzz/`
+artifacts) available for analysis.
+
+## Leaderboard runs
+
+`just benchmark` is the one-command path: it stands up a dedicated Docker
+stack (`buzz-benchmark` compose project — relay :3600, Postgres :5633, secrets
+generated once into the gitignored `.benchmark/`), applies the benchmark
+schema, and defaults to leaderboard-eligible settings (Terminal-Bench 2.1,
+5 attempts per problem, the Sonnet+Haiku team). All selectors pass through:
+
+```bash
+just benchmark                                   # full TB 2.1, k=5
+just benchmark --path <TASK_DIR> -k 1            # one local task, one attempt
+just benchmark -i "cobol*" --attempts 3          # dataset subset
+just benchmark --gui                             # watch the run live
+```
+
+One pinned user identity fronts the whole benchmark environment: it owns
+every trial channel (named after the task) and posts every task prompt, and
+trial channels are kept rather than archived. `--gui` adds that user to the
+relay membership list and opens the Buzz desktop app logged in as them, so
+channels fill the sidebar as the run progresses — watch, don't type; a human
+message mid-trial would taint the run. `just benchmark-down` stops the stack.
+
+Networking: the relay is host-header tenant-bound, so agents must dial its
+canonical address (`ws://localhost:3600`) even from inside a task container.
+`just benchmark` uploads a tiny std-only loopback forwarder
+([`forwarder/relay_forwarder.rs`](forwarder/relay_forwarder.rs)) with the
+agent stack; it listens on the container's loopback and bridges the byte
+stream to the Docker host gateway (`host.docker.internal`, overridable via
+`BUZZ_BENCHMARK_DOCKER_HOST`).
+
+`scripts/run_leaderboard.py` is the layer underneath, for running against an
+already-provisioned stack. It wraps the invocation above with only
+leaderboard-legal settings — it does not accept or forward timeout or resource
+overrides, so the job directory it produces passes Harbor's static validation
+as-is. Give it a problem set, attempts per problem, and a team manifest:
+
+```bash
+uv run --project benchmarks/harbor-buzz-orchestra/testbed \
+    benchmarks/harbor-buzz-orchestra/scripts/run_leaderboard.py \
+    --dataset terminal-bench/terminal-bench-2-1 \
+    --attempts 5 \
+    --manifest benchmarks/harbor-buzz-orchestra/manifests/<TEAM>.yaml \
+    --endpoint-config benchmarks/harbor-buzz-orchestra/testbed/endpoints/<ENDPOINTS>.json \
+    --provisioner-config <PROVISIONER.json>
+```
+
+`--path` replaces `--dataset` for local task directories; `--include-task` /
+`--exclude-task` filter by glob; `--dry-run` prints the underlying `harbor run`
+command. After the job finishes the script derives a `metadata.yaml` from the
+manifest roster (validated schema; review the display names before submitting)
+and prints the `harbor upload` / `harbor leaderboard submit` commands.
+
+## Validate
+
+```bash
+cd benchmarks/harbor-buzz-orchestra
+uv run --extra dev pytest -q
+uv run --extra dev ruff check .
+cd testbed
+uv run --extra dev pytest -q
+uv run --extra dev ruff check .
+```
+
+Live provisioner tests require the benchmark compose stack and opt-in
+environment described in `testbed/tests/test_provisioner_live.py`.

benchmarks/harbor-buzz-orchestra/forwarder/relay_forwarder.rs

@@ -0,0 +1,73 @@
+//! Loopback TCP forwarder for the benchmark task container.
+//!
+//! The benchmark relay is host-header tenant-bound: its community row is the
+//! authority of its own `RELAY_URL` (e.g. `localhost:3600`), and a request
+//! presenting any other `Host` fails closed. Agents inside a Harbor task
+//! container can only reach the host-published relay via the Docker host
+//! alias (`host.docker.internal`), which would present the wrong `Host`.
+//!
+//! So the container runtime uploads this forwarder next to the agent stack:
+//! agents dial `ws://localhost:<port>` — presenting the exact `Host` the
+//! community row expects — and the forwarder bridges the byte stream to the
+//! host gateway. Transparent to everything above TCP (WebSocket, the buzz
+//! CLI, git-over-HTTP). std-only; compiled with plain `rustc` against the
+//! musl target, so it runs on any Linux task image.
+//!
+//! Usage: `relay-forwarder <listen-addr> <target-addr>`
+
+use std::io::{self, Read, Write};
+use std::net::{Shutdown, TcpListener, TcpStream};
+use std::thread;
+
+fn main() -> io::Result<()> {
+    let mut args = std::env::args().skip(1);
+    let (listen, target) = match (args.next(), args.next()) {
+        (Some(listen), Some(target)) => (listen, target),
+        _ => {
+            eprintln!("usage: relay-forwarder <listen-addr> <target-addr>");
+            std::process::exit(2);
+        }
+    };
+    let listener = TcpListener::bind(&listen)?;
+    // Readiness marker: the container runtime polls the log for this line
+    // before launching any agent.
+    println!("forwarding {listen} -> {target}");
+    for client in listener.incoming() {
+        let Ok(client) = client else { continue };
+        let target = target.clone();
+        thread::spawn(move || bridge(client, &target));
+    }
+    Ok(())
+}
+
+/// Connect upstream and pump bytes both ways until either side closes.
+fn bridge(client: TcpStream, target: &str) {
+    let Ok(upstream) = TcpStream::connect(target) else {
+        let _ = client.shutdown(Shutdown::Both);
+        return;
+    };
+    let (Ok(client_read), Ok(upstream_read)) = (client.try_clone(), upstream.try_clone())
+    else {
+        return;
+    };
+    let downstream = thread::spawn(move || pipe(upstream_read, client));
+    pipe(client_read, upstream);
+    let _ = downstream.join();
+}
+
+/// Copy until EOF or error, then half-close the write side so protocols
+/// layered on TCP (WebSocket close handshakes) terminate cleanly.
+fn pipe(mut from: TcpStream, mut to: TcpStream) {
+    let mut buf = [0u8; 16 * 1024];
+    loop {
+        match from.read(&mut buf) {
+            Ok(0) | Err(_) => break,
+            Ok(n) => {
+                if to.write_all(&buf[..n]).is_err() {
+                    break;
+                }
+            }
+        }
+    }
+    let _ = to.shutdown(Shutdown::Write);
+}

benchmarks/harbor-buzz-orchestra/manifests/m1-hello-world.yaml

@@ -0,0 +1,44 @@
+# M1 gate manifest: 1 orchestrator + 1 worker over one Buzz channel solving
+# Harbor's examples/tasks/hello-world via vanilla `harbor run`.
+#
+# Endpoints are LOCAL PLACEHOLDERS for the M1 wiring proof; the pilot/2.1
+# manifests will carry exact Databricks serving endpoint revisions and the
+# frozen price table per the canonical plan.
+schema_version: "1"
+condition: M1-hello-world
+roster:
+  - id: orch
+    kind: orchestrator
+    role: lead
+    count: 1
+    endpoint: local/placeholder-orchestrator
+    model_revision: m1-placeholder
+    prompt:
+      path: personas/orchestrator-m1.md
+      sha256: af64b515de914c99fbdfc5a50830909ac10ab3a4f37e8352a354019a6be74151
+    generation:
+      max_output_tokens: 4096
+      context_window_tokens: 128000
+  - id: worker
+    kind: worker
+    role: implementer
+    count: 1
+    endpoint: local/placeholder-worker
+    model_revision: m1-placeholder
+    prompt:
+      path: personas/worker-m1.md
+      sha256: f3fef3d2c42105c256ff019d71ef99506b31a23e8cb252cac727c8bc470eb304
+    generation:
+      max_output_tokens: 4096
+      context_window_tokens: 128000
+prices:
+  local/placeholder-orchestrator:
+    input_per_million_usd: 0
+    cached_input_per_million_usd: 0
+    output_per_million_usd: 0
+  local/placeholder-worker:
+    input_per_million_usd: 0
+    cached_input_per_million_usd: 0
+    output_per_million_usd: 0
+trial_budget:
+  timeout_seconds: 1800

benchmarks/harbor-buzz-orchestra/manifests/tb-cobol-sonnet-haiku.yaml

@@ -0,0 +1,42 @@
+# Terminal-Bench trial: 1 Sonnet 4.6 orchestrator + 2 Haiku 4.5 workers over
+# one Buzz channel. Endpoint names are exact Anthropic model IDs (the runtime
+# passes the endpoint name to the provider as the model).
+# Prices: Anthropic list $/Mtok as of 2026-07 (Sonnet 3/15, Haiku 1/5).
+schema_version: "1"
+condition: tb-cobol-sonnet46-2haiku45
+roster:
+  - id: orch
+    kind: orchestrator
+    role: lead
+    count: 1
+    endpoint: claude-sonnet-4-6
+    model_revision: claude-sonnet-4-6
+    prompt:
+      path: personas/orchestrator-tb.md
+      sha256: 206829331cdd85fb277266901b9489e190ba23097600a8bb69142d044b9a42e6
+    generation:
+      max_output_tokens: 4096
+      context_window_tokens: 200000
+  - id: worker
+    kind: worker
+    role: implementer
+    count: 2
+    endpoint: claude-haiku-4-5
+    model_revision: claude-haiku-4-5
+    prompt:
+      path: personas/worker-tb.md
+      sha256: 664a994ae3ba93f02ebbb8f61f781f6a59ea3327b21e26e7ffc2d06827ed3e45
+    generation:
+      max_output_tokens: 4096
+      context_window_tokens: 200000
+prices:
+  claude-sonnet-4-6:
+    input_per_million_usd: 3
+    cached_input_per_million_usd: 0.3
+    output_per_million_usd: 15
+  claude-haiku-4-5:
+    input_per_million_usd: 1
+    cached_input_per_million_usd: 0.1
+    output_per_million_usd: 5
+trial_budget:
+  timeout_seconds: 900

benchmarks/harbor-buzz-orchestra/personas/orchestrator-m1.md

@@ -0,0 +1,34 @@
+# Orchestrator — M1 hello-world
+
+You are the orchestrator of a small team solving a terminal task. You do not
+run commands yourself; workers do. You coordinate over a Buzz channel.
+
+Your `shell` tool has the `buzz` CLI on PATH, already authenticated as
+you. Nothing you write is visible to anyone unless you publish it: every
+message — step assignments, verification requests, the final `DONE:`
+— must be sent with
+`buzz messages send --channel <channel-id> --content <text>`. Your team, your
+channel id, and the user you report to are listed in the "Your team"
+section below. Your turn is not complete until you have published your
+message.
+
+Rules:
+1. Read the task instruction. Break it into the smallest concrete steps.
+2. Assign each step to a worker with an @mention. One step per message.
+   State the exact goal and the success check, not just the command to run.
+   The task runs in the worker's terminal working directory. Unless the task
+   instruction itself names a path, refer to files by bare relative name
+   (`hello.txt`) — never invent an absolute path. More generally, relay the
+   task's requirements verbatim and do not add constraints the task does not
+   state (paths, encodings, byte-level rules such as forbidding a trailing
+   newline). Where the task is silent, let standard tool defaults apply.
+3. Wait for the worker's report before assigning the next dependent step.
+4. When a worker reports output, verify it against the task's success
+   criteria yourself before moving on.
+5. When the task is complete, report back to the user: publish a final
+   message starting with `DONE:` that @mentions the user and summarizes
+   what was produced and how you verified it. The task is not finished
+   until this message is published — never conclude silently.
+
+Keep messages short. Never fabricate command output. If a worker's report is
+ambiguous, ask them to re-run with the exact verification command.