Add Pi and OMP harness providers - #913
Conversation
Performance
⚠ Regression detected:
|
📊 Coverage gateThresholds from
✅ Gate passedNo surface regressed past the allowed threshold and the aggregate stayed above the floor. |
📐 Patch coverage gateThreshold: 80% on lines this PR touches vs
✅ Patch gate passedEvery surface whose lines were touched by this PR has patch coverage at or above the threshold. |
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
- Go pi.go: build the plan-mode read-only tool list as a fresh slice instead of the in-place tools[:0] filter, matching the Python/TS providers and removing the aliasing footgun. - Go pi.go: distinguish a negative return code (signal kill) from a plain non-zero exit, reporting 'Process killed by signal N.' to match the Python provider and the gemini/opencode Go providers. - Add a pi_test.go case pinning the signal-kill message.
|
Addressing the review findings on this PR. Code changes (commit
Release-note callouts (PR body, "Compatibility" section):
Not changed (reviewed, left as-is to keep the diff minimal):
Verification: |
AbirAbbas
left a comment
There was a problem hiding this comment.
Heads up that this can't merge as-is any more: main shipped aforge as the harness default (#905, #927-#929) along with an AGENTFIELD_HARNESS_PROVIDER env tier and a provisioning path (af aforge ensure, curl installer, docker images), so the 25 conflicts here are a genuine either/or on "what is the default" rather than anything mechanical — I didn't push a merge because resolving them means picking a winner.
Two calls worth making before we resolve: (a) whether the zero-config default is aforge or omp, and separately whether a provider-less call should silently start a paid run at all versus keeping the loud error this PR removes; (b) the Python HarnessConfig change, where provider goes from required to defaulted and model goes from str = "sonnet" to Optional[str] = None — that second one is a real break for anyone reading config.model as a string.
Pi and OMP as additional providers look good to me and I'd take them regardless; the default question is the only thing actually blocking. Four code notes below. Also note the earlier round's flag on harness_duo's early return isn't a bug — the channel is buffered to 2, so nothing leaks.
… default harness; Pi and OMP become additional providers Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…l providers Sweeps the OMP-as-default remnants left in non-conflicting files after the merge, and brings the Go Pi/OMP provider up to the Result.Model contract main added for aforge/opencode: - af doctor / af harness doctor list aforge-first ordering and drop the "omp default" help text. - skills/agentfield (+ embedded skill_data mirror), harness-v2-design, the harness_duo Go example and its README no longer claim OMP is the default; omp_worker now passes Provider explicitly. - TS Agent usage attribution resolves through resolveProviderName so the explicit > config > AGENTFIELD_HARNESS_PROVIDER > aforge chain is honoured. - sdk/go/harness/pi.go populates Metrics.Model (configured model wins over the model reported in the Pi/OMP JSONL stream), matching the Python and TS adapters. - Tests that asserted an OMP default now assert aforge; explicit pi/omp coverage is retained. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…fault Follow-up to the merge: three places still baked OMP (or an eagerly resolved default) into the no-configuration path, which the gates caught. - sdk/go/harness/runner.go: NewRunner no longer stamps DefaultProvider into DefaultOptions. Run() applies explicit > AGENTFIELD_HARNESS_PROVIDER > aforge, so an env change reaches an already-constructed runner and Agent.HarnessRunner() keeps zero-value options as main expects. - sdk/python/agentfield/agent.py: _harness_provider_name no longer falls back to "omp", so usage attribution goes through resolve_harness_provider. - Tests: the Go/TS supported-provider strings list pi and omp, the TS pi/omp factory test asserts explicit routing instead of an OMP default, and the duplicate aforge-default usage test is dropped in favour of main's. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The TypeScript pi/omp provider returned failures with neither failureType nor returnCode set, so callers could not tell a crash from an API error or an empty completion — the Go and Python pi providers both classify. Mirror their exact ladder: signal death -> crash, non-zero exit -> crash, a stopReason error/aborted on a clean exit -> api_error, a clean exit with no assistant text -> no_output, otherwise none. stderr is now ANSI-stripped and capped at 1000 chars like Go/Python. The catch path classifies a timeout distinctly from a crash. runCli resolved `code ?? 0`, so a child killed by a signal looked like a clean exit 0 and the "Process killed by signal N" branch was unreachable. It now reports the negative signal number, matching Go's os/exec and Python's asyncio subprocess. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ider
runner.ts bases the schema-output directory on `projectDir ?? cwd`, but only
pi/omp and aforge read projectDir — codex, gemini, opencode and claude used
options.cwd alone. `{ schema, projectDir, cwd, provider: 'codex' }` therefore
wrote the instruction file into one directory and ran the CLI in another, so
the agent was told to write a file outside the root it could see.
Add a single resolveRoot() helper (projectDir -> project_dir -> cwd, the
precedence every Python provider already uses) and route all six providers
through it. This also fixes opencode's inverted ladder, which checked cwd
first and then a snake_case project_dir key the TS options object never
carries.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`af doctor --probe` ran `pi --print "Say OK"` with the prompt positional and stdin left at EOF, while all three SDK adapters run `<bin> --print --mode json` and feed the prompt over stdin. The probe therefore exercised a different surface than the harness does, so a healthy install could be reported as empty or error. The registry entry gains ProbeStdin; pi and omp now carry the adapters' exact flag set with the prompt on stdin, and runProbeCommand wires a strings.Reader in when a payload is present. Providers that take the prompt positionally keep a nil stdin, and the 60s probe bound is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Verified against the real CLIs (pi 0.74.2, omp v18.0.7):
- Pi has no approval flag at all. `--approve`, `--auto-approve`, `--yolo`,
`-y`, `--approval-mode` and `--permission-mode` each fail with
`Error: Unknown option: <flag>`, so the `permission_mode="auto"` branch made
every Pi auto-mode run die on argument parsing. Only OMP gets a flag now
(`--auto-approve`, which it does document).
- `--tools` is an enforced allowlist in both CLIs and is Pi's own documented
read-only mechanism ("Read-only mode (no file modifications possible):
pi --tools read,grep,find,ls -p ..."), so plan mode is genuinely read-only
with no approval flag. OMP's default `tools.approvalMode` is `yolo`, and even
`always-ask` auto-approves read-only tiers, so a read-only allowlist never
blocks on approval there either.
Plan mode therefore keeps sending only the read-only allowlist. The flag set is
now pinned by a test in each SDK that rejects every known approval-style flag,
and the ground truth is recorded in a comment at each branch so it does not get
"fixed" back.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… test test_default_provider_is_aforge asserts the built-in default, but HarnessConfig.provider resolves through AGENTFIELD_HARNESS_PROVIDER, so the test failed on any machine that pins a harness provider in the environment. Every sibling test that asserts this default already clears the variable (test_harness_types.py, test_types.py, test_harness_defaults.py, test_harness_runner.py); this one did not. Verified: `AGENTFIELD_HARNESS_PROVIDER=codex pytest tests/test_harness_factory.py` now passes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`--provider` has no default: omitting it surveys every provider. In cobra help, "(default)" reads as the flag's own default value, so annotating aforge that way advertised behaviour the flag does not have. aforge is the SDK's default harness provider, which is a different statement and belongs in the SDK docs, not in this flag's help. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…name The test body never asserted a default — it checks OMP's provider name, auth status, official install command and usability — but its name was residue from the reverted "OMP is the default provider" design. aforge is the default; this was the last OMP-default claim left in the tree. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The two other README spots that enumerate harness providers already list pi and omp; the "Harness (Multi-turn Coding Agents)" table still stopped at opencode. aforge stays the zero-setup default, stated in the row above. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Pi-family event stream can carry several assistant message_end events. All three adapters took the assistant text last-writer-wins but kept the provider error first-writer-sticky: once any message_end reported stopReason "error" or "aborted", nothing cleared it. A run whose model call failed on an intermediate turn and then recovered was surfaced as failure_type=api_error with a stale message, its correct final answer discarded — and since api_error is transient, the runner burned a retry re-running the whole harness invocation. Only the final message_end's stop reason decides now: every assistant message_end sets or clears the provider error. Nothing else about the parse changes, and no exit-code branch moves. Fixed identically in Go, Python and TypeScript with a regression test in each, covering both error-then-recovery (clean run) and recovery-then-error (still an api_error). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…roviders use
resolveRoot() centralised the provider-side working-directory ladder
(projectDir -> project_dir -> cwd), but the runner kept its own two-rung
version that never looked at project_dir. resolveOptions copies overrides
with Object.entries, so a JS caller's project_dir key does reach the
providers: with { project_dir: P, cwd: C } the schema instruction file was
created under C while the provider ran in P, and the harness was told to
write its output to a path outside the directory it was running in. Before
this branch that split existed for aforge alone; centralising the ladder had
widened it to six providers.
The runner now calls resolveRoot on the resolved options, so the two ladders
are identical by construction. Regression test covers the snake_case-only
case; it fails if the line is reverted.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ClaudeCodeProvider goes through @anthropic-ai/claude-agent-sdk rather than cli.runCli, so the runner's provider matrix cannot reach it, and the existing claude tests only ever passed `cwd`. Reverting claude.ts to the old `options.cwd` line left the whole suite green. Two tests close that: projectDir wins over a nested cwd, and an empty-string cwd now leaves the SDK option unset (resolveRoot skips empty strings, where the old code forwarded ''). Both fail against the reverted hunk. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The agentfield skill gates every use of app.harness() on `af doctor` reporting harness_usable: true AND listing the chosen provider. doctor's provider list never contained aforge, so on the default install — aforge shipped with `af`, nothing else present — an agent following the skill concluded the harness was unusable and refused to use the default provider. This branch had made that worse by adding aforge to the skill's provider union while leaving doctor's list alone. Detection now reuses `af harness doctor`'s spec table (findHarnessProviderSpec + probeHarnessBinary) wherever a binary-backed spec exists, so both doctors agree on what "installed" means. That matters for aforge specifically: it answers `version`, not `--version`, and `af aforge ensure` installs it into $AGENTFIELD_HOME/bin, which the current shell's PATH usually does not contain. claude-code has no binary in that table (it is the pip-package wrapper) and keeps the plain PATH check. --probe skips providers that declare no ProbeArgs, which is aforge alone: every other probe is one trivial completion, whereas aforge's only one-shot is a full coding-agent run with write access to the working directory, which is not something a doctor command should start. `af harness doctor` reports aforge's health. Live-verified with a fake aforge in $HOME/.agentfield/bin and an empty PATH: `af doctor --json` reports aforge available with its version, and recommendation.harness_usable true / harness_providers ["aforge"]; `af doctor --probe` produces no aforge probe entry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s ok
classifyProbe decided "empty" purely by an empty stdout. That was correct
while pi/omp probed in plain --print text mode, but they now probe with
`--print --mode json`, and the CLI emits a {"type":"session",...} event
before any assistant output. stdout is therefore never blank and the probe
always fell through to "ok" — so an install that exits 0 with a parsed stream
and no assistant text, and one whose message reports stopReason "error", both
came back healthy. That negates the exact capability --probe's own help text
advertises, for the two providers this branch added.
Providers whose probe output is a JSON event stream are now marked
JSONLStream, and their probes apply the SDK adapters' own success criterion:
an assistant message_end carrying text, with the last assistant message_end's
stop reason not "error"/"aborted" (a turn that errored and then recovered is
not a failure, matching the adapter fix in this branch). An exit-0 stream
error surfaces its message as the probe detail when stderr is silent. Plain
text providers keep the previous rule unchanged.
Live-verified with fakes on PATH: an `omp` printing only
{"type":"session","id":"s1"} and exiting 0 now reports status "empty" where
it reported "ok" before; a `pi` printing a real assistant message_end still
reports "ok".
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The branch in buildDoctorReport that routes detection through the harness
doctor's spec table (findHarnessProviderSpec + probeHarnessBinary) is the
whole behavior of "fix(cli): make `af doctor` survey aforge, the default
harness provider" — it is what makes doctor ask aforge for `version` rather
than `--version`, and what makes it look in $AGENTFIELD_HOME/bin when the
binary is not on PATH. Replacing that branch with main's original
`checkTool(h.Binary, "--version")` left the entire internal/cli suite green,
so a future refactor could revert the fix without CI noticing.
TestBuildDoctorReport_AforgeDetectionUsesHarnessSpec drives buildDoctorReport
with a shell-script aforge stub and covers both halves:
• installed only in $AGENTFIELD_HOME/bin with an empty PATH — doctor must
report it available, with the managed path and its version;
• on PATH but answering `version` only (non-zero on `--version`) — doctor
must still record the version.
Both subtests fail under the `checkTool(h.Binary, "--version")` mutation
(available:false / version:"" respectively), and the full
`go test ./internal/cli/ -count=1` suite stays green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
AbirAbbas
left a comment
There was a problem hiding this comment.
Merged this with main instead of rebasing and kept aforge as the default harness provider — Pi and OMP land as additional providers you pick explicitly or through AGENTFIELD_HARNESS_PROVIDER, so provider-less calls resolve exactly as they do on main. The four findings from my last round are fixed: TS pi/omp results now carry failureType and returnCode, all six TS providers resolve their working directory through one projectDir ladder, af doctor --probe drives pi/omp over stdin the way the adapters do, and Pi's auto-mode approval flag is gone because the real CLI rejects every variant of it. Verified with the per-surface CI-exact gates, a defaults proof in all three SDKs, and real-binary doctor and error-path runs.
Takeover note (2026-08-27)
This PR was taken over and reworked. aforge stays the default harness provider — that was the maintainer decision, and this branch no longer changes it. The branch was merged with
main(no rebase) and the OMP-default logic was removed: Pi and OMP are now additional providers you select explicitly or viaAGENTFIELD_HARNESS_PROVIDER, nothing more. Provider-less calls resolve exactly as they do onmain.The four review findings from the last review round are fixed:
failureType/returnCode), unlike every other TS provider3ae5157dprojectDirwas only honoured by pi/omp, so the schema-output protocol disagreed with the other TS providersba76a4c5af doctor --probepassed the prompt positionally instead of over stdin the way the pi/omp adapters do944b388abdbaf471Follow-up commits on top of those keep
af doctorsurveying aforge itself, stop--probereporting a silently broken pi/omp install as ok, and drop the stale "OMP default" wording from docs and tests.Product outcome
AgentField can now run Pi and OMP as first-class coding harnesses from Python, TypeScript, and Go. aforge remains the zero-configuration default; Pi and OMP are selected explicitly per call, through
HarnessConfig, or viaAGENTFIELD_HARNESS_PROVIDER. A production-shaped Go example fans one task out to Pi and OMP concurrently and joins their typed results in a single AgentField workflow.This closes the practical parity gap with OpenCode: applications use the same AgentField harness contract for model selection, project root, system prompt, tools, permissions, session resume, structured output, retries/timeouts, and normalized usage metrics.
Developer experience
main: per-call override → agent harness config →AGENTFIELD_HARNESS_PROVIDER→ aforge.provider="opencode"orprovider="pi"are unchanged.harness=pi/harness=ompacross Python, TypeScript, and Go.af harness doctorrecognizes both CLIs, reports binary/version/auth/usability, and lists aforge first as the default.Architecture
#variant→--model+--thinking--cwdplus process cwd for OMP--auto-approve), since Pi has none--session; OMP--resumefind; OMPglobParity assessment
Pi and OMP have feature parity with OpenCode at the AgentField contract boundary. Native flags are intentionally not identical: each adapter translates its CLI's approval, resume, project-root, and filesystem-search vocabulary. Those differences do not leak into application code.
Covered contract surface:
Installation and operations
npm install -g --ignore-scripts @earendil-works/pi-coding-agent curl -fsSL https://omp.sh/install | sh af harness doctor --provider pi,omp --jsonThe doctor check is static and does not make a paid model request. Containers should install the selected CLI during image construction and run the doctor as a startup/CI gate.
Go workflow example
examples/go_agent_nodes/cmd/harness_duoregisters:comparestarts both branches concurrently and joins their structured results. It defaults toopenrouter/minimax/minimax-m2.7and acceptsopenrouter/google/gemini-2.5-flashfor a faster smoke path.Verification
0.84.1and OMP17.2.15, both authenticated through OpenRouter and usable.run_20260812_164308_d2tasl3y/exec_20260812_164308_fp15z9ifsucceeded in 19.4s withopenrouter/google/gemini-2.5-flash;compare,pi_worker, andomp_workerall completed with structured output and native cost/token metrics.run_20260812_162041_fw7j8b54withopenrouter/minimax/minimax-m2.7.Compatibility
No default-provider or model-default change: provider-less calls resolve exactly as on main (explicit → AGENTFIELD_HARNESS_PROVIDER → aforge).
Explicit provider configurations are otherwise unchanged.