Skip to content

fix: harden Agent Harness on a 62-file legal workload - #437

Open
zhengdaqi wants to merge 15 commits into
mainfrom
codex/legal-runtime-e2e
Open

fix: harden Agent Harness on a 62-file legal workload#437
zhengdaqi wants to merge 15 commits into
mainfrom
codex/legal-runtime-e2e

Conversation

@zhengdaqi

@zhengdaqi zhengdaqi commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

Summary

This PR hardens generic PilotDeck Agent/Harness behavior using a fresh 62-file legal due-diligence workload as the end-to-end proving ground. Legal reasoning, milestone policy, and review criteria remain in a temporary Domain Plugin/Skill; PilotDeck Core receives only reusable runtime mechanisms and reliability fixes.

The branch now includes the latest main through merge commit 845fc260, preserving both histories after resolving the earlier base divergence.

Changes

Generic Harness behavior

  • Keep required deliverables visible to the model on every request through a transient, model-only reminder until each required artifact exists and is non-empty.
  • Reject zero-byte required files as artifact_empty instead of treating file existence as completion.
  • Normalize exact provider termination errors (terminated, connection terminated, and stream terminated) as retryable connection resets, including Anthropic-compatible SSE errors.
  • Ignore generated Python, macOS, and Office runtime files in extension watching; disable Python bytecode generation in the DOCX wrapper.
  • Retry an empty or failed LibreOffice PPTX conversion once with a fresh isolated profile.

Existing runtime and office delivery work

  • Preserve the generic dynamic-context, lifecycle prompt-patching, Domain Plugin, required-artifact, internal prompt-dispatch, spreadsheet, file-artifact, and UI work introduced by the branch and current main.
  • Keep legal-specific workflow code outside this repository; the temporary benchmark plugin is not committed.

QA & Evidence

Real 62-file legal workload

  • What was tested: Production createLocalGateway in an isolated home/project with 37 DOCX, 16 PDF, 5 XLSX, 3 PNG, and 1 PPTX; temporary legal Domain Plugin/Skill; per-request dynamic prompt patches; required Markdown artifact.
  • Observed result: Primary run completed without runtime error; 40 streamed requests all received the dynamic patch, 41/41 tool calls completed, and the required artifact was emitted. A short recovery session completed manually detected quality evidence after the primary session became compaction-heavy.
  • Artifact: .omo/evidence/20260723-legal-hard-runtime/hard-run-summary.json
  • Why sufficient: It exercises the generic lifecycle, compaction, prompt injection, Skills, tool execution, and artifact contract together on a heterogeneous real workload.

Independent hard-quality gate

  • What was tested: A content-independent verifier checked the 62-entry ledger, machine-readable coverage, 9-slide PPTX manifest, eight non-empty topic-evidence files, reverse coverage, and a final integrity sentinel bound to the current opinion SHA.
  • Observed result: Passed with zero missing or duplicate paths, zero uninspected machine-readable sources, all eight topic files present, and the integrity sentinel valid. Final artifact: 61,313 bytes, SHA-256 fb8e30384b89faa0e975db0cf92aa8463b77d04ffe8da88302a9b8d16609b68a.
  • Artifact: .omo/evidence/20260723-legal-hard-runtime/hard-quality-gate.json
  • Why sufficient: The gate derives success from files, coverage records, and hashes rather than trusting the model's completion message.

Regression and office surfaces

  • Backend: npm test passed 192/192 after merging current main.
  • UI: Filtered baseline-independent Vitest suite passed 293/293; production npm run build passed.
  • Office Skills: DOCX, Spreadsheet, PDF, and PPTX self-tests passed. The PPTX self-test first exposed intermittent empty LibreOffice output; after the fresh-profile retry fix, the complete suite passed.
  • Artifacts: .omo/evidence/20260723-legal-hard-runtime/regression-summary.json and .omo/evidence/20260723-legal-hard-runtime/known-ui-baselines.md

Risks & Residuals

  • Seventeen scan-only files were explicitly classified as unreadable scans in this text-first run; this PR does not claim semantic review of their image contents.
  • The full UI test command and UI typecheck still expose origin/main baseline issues (stream-smoothing expectations, node:sqlite under jsdom, Playwright collection, and mixed React 18/19 type trees). The implicated files/configuration are unchanged; the filtered suite and production build pass.
  • The external 321-checkpoint semantic Judge completed on the bounded retry: 224/321 passed (69.78%), including 117/141 critical checkpoints (82.98%), with zero Judge errors. This is not an all-pass result; 97 total gaps, including 24 critical gaps, remain for follow-up classification. See .omo/evidence/20260723-legal-hard-runtime/judge-status.json.
  • Raw legal text, case identifiers, source filenames, model credentials, raw sessions, generated legal evidence, and Judge payloads are intentionally excluded from git.

@zhengdaqi

Copy link
Copy Markdown
Collaborator Author

Maintainer attention: base branch was force-reset

After this PR was opened, main was force-reset on 2026-07-22 from:

This branch intentionally remains based on 7404ba5c, because the requested legal-runtime QA and the fixes in this PR were implemented and validated against that merged runtime.

As a result, GitHub currently shows this PR as 56 changed files / 12 commits ahead of main: merging it now would both restore all changes from #436 and add the five legal-runtime hardening commits in this PR.

Please approve and merge only if restoring 7404ba5c is intended. If the rollback was intentional, the correct next step is to decide how the runtime from #436 should be reintroduced or redesigned before rebasing this PR; silently rebasing onto 07462de1 would discard the runtime under test.

@zhengdaqi
zhengdaqi requested a review from mssssss123 July 22, 2026 05:31
Preserve the legal-runtime harness work while integrating the latest artifact, office-skill, and UI changes from main. Add generic hard-case fixes, focused regressions, and sanitized 62-file legal QA evidence.
@zhengdaqi zhengdaqi changed the title fix: harden legal runtime artifact delivery fix: harden Agent Harness on a 62-file legal workload Jul 23, 2026
Record only aggregate 321-checkpoint results; keep raw legal content and the Judge payload outside git.
@zhengdaqi

Copy link
Copy Markdown
Collaborator Author

Verification gate update: all CodeQL checks passed on head 2b1b120c. Cubic produced no review or comment during the full bounded 30 x 20-second wait after the latest push, so the Cubic gate is recorded as SKIPPED (no review within bound / service or quota unavailable), not as an approval. There are no Cubic findings to waive.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant