agents: reproducible, agent-steerable plan validation; 5-rollout gate - #148
Merged
Conversation
yichao-liang
force-pushed
the
bridge-gt-hardening
branch
from
August 20, 2026 21:04
30b25da to
666a5ae
Compare
yichao-liang
force-pushed
the
agent-plan-validation
branch
from
August 20, 2026 21:04
a053388 to
255b905
Compare
The BiRRT helper modeled a single held body; bodies weld-attached to it
were dropped from the collision set entirely ("welded partners sweep
unchecked"), so transporting the bridge span row swept its outer spans
through a standing leg and toppled it in 3/3 oracle bridge runs.
Pose every welded partner rigidly with the arm and collision-check it
exactly like the held object itself (run_motion_planning gains a
held_attachments map of end-effector-relative transforms, captured by
the skill factory from the planning simulator). The collision
diagnostics and the stall report cover the attachments too, and a new
regression test rejects a goal that sweeps an attachment into an
obstacle.
Three E2E failure modes seen in the oracle process-planning runs: - Halve the leg release drop (8-13 mm -> 4-6 mm): the 2:1 leg could land rocking near its tipping balance and slowly topple ~30 steps after release with nothing touching it. - PlaceSpanNextTo: two-sided +/-1 mm jitter instead of one-sided +0-4 mm, and a softer drop. Params are frozen at planning time, so the left neighbor's own landing error stacks on any outward bias against the ~1 cm cure-window margin. - SeatSpan3: center the row by the midpoint of the outer spans' actual centers instead of the grasped middle span, so weld-frozen placement offsets split across both seat joints instead of landing on one. - Extend MoveTo's x params bounds one span half-length past the block workspace: a block staged at the workspace edge has its end-face dab point up to 5 cm outside it, and the params clamp silently parked the glue tip 2.5 cm short of the dab (outside the 2 cm wetting radius), so the face never wet and the joint never cured. A genuinely unreachable dab now fails IK loudly and triggers a replan instead. With these plus the welded-partner motion-planning fix, the oracle arm solves the bridge task 3/3 on seeds 0-2.
The fully-observable GT simulator softens its cure-alignment gates with sigmoids for the fitting Jacobian, using the shared SOFT_EPS (0.02, sized for 5-15 cm thresholds) on the bridge's mm-scale windows. That capped a geometrically PERFECT butt joint's gate weight at ~0.33 and a perfect seat at ~0.77 -- and since the cure counter evolves as prog = w * (cure + 1), which fixed-points at w / (1 - w), curing could NEVER reach the latch threshold (25) for ANY geometry. Welding was impossible in the simulator while trivial in the env: all three agent_oracle_hybrid_sim attempts independently mastered legs, glue, and adjacency in the sandbox, then dead-ended on Attached with cure crawling at ~3e-4/step toward a fixed point of 0.5. Recalibrate with a bridge-local GATE_EPS of 1 mm: in-window joints saturate to w ~= 1 and latch in ~cure_threshold steps (matching the env's hard counter), the boundary keeps a +-4 mm differentiable band, and the sim's effective window sits ~3 mm inside the env's -- always conservative, never the reverse. New unit tests pin the latch behavior (the coverage gap that let this ship): in-window butt and stacked joints latch on schedule, out-of-window joints never do.
Fixes for the three demo/execution failures the launched agent_oracle_hybrid_sim runs exposed on train-task instances: - Pose welded partners from the welds' IDEAL frames during motion planning (new env API get_welded_partner_transforms, chained from the constraints' snapped frames), instead of capturing live poses: the carried row swings like a pendulum after a lift, and a live snapshot froze an outer span 19 mm low, failing a seat-descend goal the settled assembly clears. The seat release clearance returns to the long-tested 1.2 cm. - Execute every planned waypoint for bridge (path subsample ratio 2 -> 1): a carried span travels 5-8 cm per physics step between subsampled waypoints, and that corner-cutting swept a just-picked span through a standing leg's top corner (a 0.3 mm graze topples the 2:1 leg, and a toppled leg is unrecoverable -- PlaceLegAtSite requires Standing and nothing re-erects one). - held_bystander_clearance of 1 cm for bridge skills: the carried block lags the end effector's swings by up to centimetres, so plans keep a real berth from pass-by bodies; the planner's endpoint exemption keeps butt joints and seating plannable. - Mirror the current yaml in the oracle bridge smoke test's flattened config (contact margin -0.005, subsample 1): welded-partner checking made a by-design 2 mm frozen-offset graze at the seat goal visible, which the stale 1 mm default margin rejected. Oracle E2E after the fixes: train tasks (the demo path) 3/3 on seeds 0-2; test tasks 2/3 with the remaining miss a marginal seat-landing draw that four probe variants of the identical task all clear.
Glue, cure, and attachment features live in Object.sim_data, which is stored on the Object INSTANCE. States routinely cross env instances (option-model resets, refinement rollouts, fresh per-episode test envs) carrying the source env's Object instances, and the bridge env read and wrote those features through whatever instance was at hand: _set_domain_specific_state wrote onto the incoming state's objects, and _domain_specific_step / _latch_joint wrote onto the objects of self._objects, which _set_state points at the incoming instances. Net effect: a sim rollout in one env silently mutated another env's hidden glue state. Observed in the seed-1/seed-2 agent run post- mortems as impossible values in the real env's final state dump (glue flags the execution monitor had correctly seen missing, and soft-cure floats like 1.55e-142 that the env's integer cure ticks cannot produce) written there by suffix-replan refinement rollouts on the option-model env. It would also have poisoned real execution resuming after any successful replan, and it made the wet-patch visuals (which read the env-owned instances) disagree with the imported state. Fix: resolve every sim_data access to the env-owned canonical block first. _attr becomes an instance method that canonicalizes via the new _own_block (name lookup into self._blocks), and all raw setattr sites go through the new _set_attr counterpart. Adds a regression test that imports a glued state into a second env instance and asserts the first env's observation stays clean (fails before this change).
…catter Place releases were open-loop drops: descend to release_z, open, and let the block free-fall the last few mm. The bounce-and-slide settle scatters the landing by 1-3 mm run to run, and this domain's margins are exactly that size (butt-joint cure windows, 2:1 legs that topple from sub-mm grazes, the seat landing). That scatter was the bridge demonstrator's dominant flakiness. Place skills gain an optional guarded SettleToContact phase (settle_to_contact_depth): after the collision-checked descent, an incremental-IK contact stroke lowers the held object until the held ASSEMBLY (the object or anything welded to it, discovered by BFS over the client's fixed constraints) first touches a body outside itself, then releases at essentially zero gap. The stroke runs as a gentle stroke (new Phase.max_step_norm): 3 mm steps bound the post-contact overshoot, a joint-jump guard suppresses IK branch flips (single-shot panda IK once answered a plain 2 cm descent with a wrist-flipped solution whose retreat batted the released block 9 cm across the table), and the incremental-IK stall abort is armed as the escape hatch. make_move_to_phase forwards terminal_fn and max_step_norm. Bridge wiring and margin retuning on top: - settle_to_contact_depth=0.03 on Place; sampler drop clearances cut to 2-3 mm (now descend-goal clearances, not free-fall heights). - Seat descend clearance raised 12 -> 20 mm: extra clearance now costs nothing, and the descend-goal check inherits the held span's LIVE pitch through the grasp transform (a carried row rides at ~0.1 rad, hanging an outer span up to 19 mm low), which at 12 mm collided with a leg top and killed otherwise-sound seat goals. - PickBottle lift raised to 3 cm and Place's first phase allows shallow held start contacts; the bridge config deepens the shallow margin to -12 mm: post-pick reconstruction artifacts occasionally model the held object up to ~9 mm into the surface it was just lifted off, rejecting the next Place's start config. Demonstrator sweep (4 seeds x 3 train + 3 test): 24/24 solved under the final configuration, up from 21/24; the three old failures (two seat grazes, one pick handoff) and the scatter class are gone. Full suite: 1483 passed. New physics regression test pins the contract: a release_z 8 mm high still lands the block at resting height, on target, without spin.
Wetting was instantaneous: the moment the bottle tip crossed the 2 cm apply radius for a single step, the face wet. That let a DRIVE-BY graze glue -- a marginal target whose approach never dwells near the dab could still wet when its retreat clipped the radius for one step, a step-phasing coin flip. The seed-1 agent run died exactly this way: its glue hover validated 6/6 in the sandbox on lucky phasing, then missed by ~2 mm for real, and the run had no recovery path. Wetting now requires the tip inside the radius for wet_streak_steps (3) CONSECUTIVE steps, implemented identically in the env and both GT simulator variants (FO + PO) so the sandbox can never validate a graze that real phasing then misses. The streak rides in the existing glue_* feature as partials (0.2/0.4, kept <= 0.5 so every is-wet reader -- classifiers, cure gate, patch visuals -- still sees a dry face), so it round-trips through _set_state like any other feature and needs no new state schema. This immediately exposed that the oracle itself only spent ~1 step in range (2.4 cm/step approach, immediate retreat), so the phase machinery gains Phase.dwell_steps -- hold at the reached target for N extra policy steps before advancing -- and the bridge MoveTo dwells wet_streak_steps + 1 at its target. Deliberate aims wet reliably; grazes never do; any agent using MoveTo inherits the dwell. Blocks also gain observable body-frame half extents (half_x/y/z, in FO and PO modes): agents needed block dimensions to compute face centers, dab points, and touch spacings, and were burning solve budget probing the physics for them (one attempt spent $12 re-deriving geometry the type now simply reports). Verified: 24-instance demonstrator sweep 22/24 (both failures are known flaky modeling-artifact tails unrelated to wetting; every glue step wet correctly), full suite 1485 passed, new regression tests pin the contract on both sides (env: drive-by graze never wets, streak resets on interruption; GT sim: identical behavior on hand-built states).
Two rails that turn rare hard aborts into soft, replannable outcomes. Gentle strokes (Phase.max_step_norm, e.g. Place's settle-to-contact): when the EE makes no progress toward the stroke target for 8 consecutive steps -- pinned by the joint-jump guard, blocked by a contact on the ROBOT itself (which the held-assembly contact terminal cannot see; a settle once stalled 25 steps this way and the stall abort killed the episode), or saturated at a joint limit -- the stroke now gives up and ADVANCES to the next phase instead of aborting the option. Gentle strokes are best-effort contact seeks below an already-validated pose, so releasing from wherever the stroke reached is strictly better than losing the episode; only a final-phase stroke keeps the stall abort (it has no next phase). This subsumes the previous pin-advance counter. Shallow held start contacts become per-body ESCAPE margins: a normal shallow body keeps the configured shallow margin, while bodies named in the new run_motion_planning unbounded_shallow_bodies (the skill layer passes the sim's static support / table ids) allow whatever depth the START config shows, minus 3 mm of slack -- the start is escapable at any modeled depth, but the path can never go deeper than it began. Rationale: a lift-off phase legitimately begins with the held assembly resting on its support, and planning-time modeling artifacts can show that resting contact tens of mm deep (a welded row's outer span was modeled 20.9 mm into the table it sat on, one mm past the old fixed cap -- an unwinnable margin race), while deep start penetration into a MOVABLE body still signals genuine trouble and keeps the cap. The bridge config's shallow margin stays -0.02 for the movable-body case. Verified: the two targeted failure modes no longer occur (the settle stall now releases and replans; the row-pick start rejection is gone); 24-instance demonstrator sweep holds its ~21-23/24 plateau with the experiment-critical instances (train0 + test0, all seeds) at 8/8 in every sweep; full suite 1485 passed.
Two structural fixes for why agent-planned bridges came out bent and open-loop plans kept invalidating themselves: 1. Weld anti-creep. A PyBullet JOINT_FIXED constraint between two table-resting bodies skates 7-9 mm and up to 0.13 rad of yaw per 200 idle steps (invariant to maxForce, erp, pair-collision filtering, and a zero-error anchor; present even for a pair welded 5 cm apart), while unwelded pairs move < 1.5 mm. _relax_resting_welds re-anchors every weld at the current snapped pose each step while its assembly is quiescent, unheld, and untouched by the robot, so the solver never accumulates an error to fight; under load the anchor holds and the weld stays rigid. Welds are also created with a zero-error child teleport and welded partners no longer collide with each other (restored on weld removal). Drift is now 0.0-0.1 mm / 200 steps. 2. Verified place release. Plant sag (position control under gravity + payload) walks the settle-to-contact stroke ~15 mm toward the robot base, and the stroke releases at FIRST contact, so every robot placement landed with a repeatable ~15 mm bias the cure gates tolerate but that bends every butt row. The settle phase now verifies the held object is within 4 mm of the commanded xy before opening; on failure it rewinds to the descend phase and re-runs the approach with a learned aim offset that targets upstream of the measured drift (feedforward, so the servo stays unstrained -- a strained anti-bias servo drags the released block through the finger pads when the grasp constraint drops). Landings are now within ~2-4 mm and a butt placement leaves its neighbor untouched. Supporting fixes: the rewound descend allows shallow held-object start contacts (a rewind legitimately starts resting on the support); a failed rewind degrades to best-effort release instead of aborting; and run_motion_planning's contact-partner scan now re-evaluates bodies at both endpoints (a butt-joint neighbor 3.1 mm away at the start but 1.8 mm at the re-aimed goal was permanently classified a bystander and its own goal proximity rejected the plan). Verified: weld drift probes at 0.0-0.1 mm; place accuracy probes at 2-4 mm across the workspace; the full glue-cure-weld chain drift-free across five Waits; demonstrator sweep 22/24 with all experiment- critical instances 8/8; full suite 1486 passed.
Goal IK returned the FIRST pose-accurate branch, with no collision awareness. IK branches reach the same end-effector pose with different arm configurations, and they are not collision-equivalent: one grasp branch can sweep a link ~6 cm through a neighboring block while another clears it. Which branch a single solve lands on depends on the IK seed, so goal-config acceptance was a per-seed coin flip -- observed as validation flakiness in the bridge agent runs (a capture candidate reached the goal on 2/3 rollouts and was rejected FLAKY because one decorrelated repeat drew a grasp branch modeled 59 mm inside a standing leg). _solve_goal_ik is now _solve_goal_ik_candidates: it collects ALL distinct pose-accurate limit-clamped branches from the existing multi-seed restarts (deduplicated, current-joints branch first), and run_motion_planning takes them as goal_candidates, planning to the first branch whose goal configuration (and fingers-open variant, when a release follows) passes its collision check. When none passes, the primary branch is kept so failure diagnostics still report the blocking contacts. Verified: skill-factory unit tests updated to the candidates API plus a new branch-collection test; oracle bridge E2E; demonstrator sweep 22/24 with experiment-critical instances 8/8; full suite 1487 passed.
Two BiRRT margin rules that stop start-config contacts from poisoning whole plans: 1. Robot links can begin a phase already in modeled contact (a finger or wrist link 5-15 mm inside the object it just grasped or settled on, from execution-side sag and settle). Each such body gets a per-body escape allowance -- the start depth minus 3 mm slack -- active only within 0.5 rad of the start config, so the arm may pull out of a contact it began in but can never deepen it or re-enter it later in the path. 2. Contact-partner status earned SOLELY at the start config no longer licenses hard-margin penetration for the whole path. A movable body the robot merely starts near keeps the hard margin only inside the start radius and gets a no-penetration margin beyond it (touching stays legal). Static bodies keep their partner margin throughout -- they cannot be shoved -- and so do goal-earned partners. Rationale: the hard margin tolerates enough penetration to shove a free-standing object, and a retreat after a glue dab repeatedly nudged an assembled row it had grazed on the way out. Tests: a thin-wall scene asserts no penetration beyond the start radius across seeds while a static wall still plans; a robot start-escape test pins the per-body allowance contract.
When BiRRT fails and diagnostics show the GOAL config in collision, re-solve goal IK with 32 restarts instead of 8 and re-plan: the pose is often reachable by another arm branch the small restart budget missed. If none of the escalated branches is collision-free either, the diagnostics now say so explicitly -- the pose itself sits in clutter, and no path or arm branch can fix it. Goal-IK candidates are returned proximity-first: the priority seeds (current joints, home) lead, and restart-derived branches sort by max-abs joint distance from the current config, so planning tries the least contorted branch that reaches the pose before the exotic ones. _log_collision_diagnostics gains log_errors so the escalation path can inspect diagnostics without spamming the error log; a final failure still logs them once.
A wet glue joint was held only by friction while curing: a release impulse or a neighbouring operation could shear the mated pair apart mm-scale during the cure dwell, and the eventual weld froze that drift in. Every wet, mated joint now carries a weak JOINT_FIXED tack (0.5 N max force, below a block's ~1 N weight, so a tack can never lift or drag its neighbour) at the current relative pose, replaced by the rigid weld at latch. _latch_joint now returns bool so a refused latch keeps its tack instead of silently leaving the joint loose, and _sync_welds_to_state tears tacks down before rebuilding welds. Tests: a welded-pair-plus-newcomer stage drifts 1.98 mm under a release impulse with the tack off and 0.03 mm with it on; a second test pins the tack lifecycle from wetting to weld.
Bridge was the only domain running with pybullet_ik_validate on (common.yaml turns it off everywhere else), on the theory that the cure gates need placement accuracy. Validation is not what enforces that -- a goal-IK branch is accepted on its forward-kinematics error against move_to_pose_tol either way -- and it steered the arm into worse-executing IK branches. Measured on the oracle demonstrator, seeds 0-3, 6 tasks each: 22/24 with validation on, 24/24 with it off (both failures under validation were process-plan exhaustion).
The override papered over plan-start configs that modeled a just-picked assembly 9-21 mm inside the surface it was lifted off. Those poses were real states produced by weld creep (7-9 mm of skate plus 0.13 rad of yaw per 200 idle steps) and the ~15 mm plant-sag place bias, both fixed since (weld anti-creep re-anchoring, verified place release); static supports also get unbounded start-escape allowances now. Measured with instrumented planning calls at the default -0.006: oracle demonstrator seeds 0-3, 24/24 solved, and no held-assembly start contact ever exceeded 3 mm (the artifact the override was sized for was 9-21 mm).
A grasp used to be granted off a SINGLE finger with an aligned closest point within 0.5 mm. When a pick's grasp_z_offset put the pads' grip band at a block's top corners, the closing fingers cammed over the top (shoving the block into the table), one finger's corner graze passed the check, and the constraint froze the block dangling below the gripper - a ghost grasp that doomed the downstream place (seed0 run_20260819_053515: leg0 pressed into the table, landed 16-18 mm off site, episode dead after refinement exhaustion). Under position control, detection now also requires the PARTNER finger to have an aligned closest point within grasp_partner_tol (3 cm): it must at least be closing in on the object's other side. A finger pressing a perpendicular face (the cam-over's top-face press) fails alignment at any distance, while an off-center pre-pinch (jug handle: first pad touches, second still arriving) stays a legitimate capture. Under "reset" control the fingers teleport closed in one step, so closing-instant closest-point normals are overlap artifacts (a legitimately captured jug handle shows the same vertical partner normal as a cam-over) - those envs keep the single-finger rule. Measured from the failed run's staging: offsets 0.0-0.009 stay proper grasps 14/14; the failure's 0.01 becomes an honest miss 7/7.
Verified advancement (Phase.verify_fn) always advanced best-effort once the retry budget ran out. That is right for a place's settle stroke (release where it ended beats aborting) but wrong for phases whose failed verification proves the option's outcome is already lost - pressing on just defers the failure to a downstream option with less context to report it. - Phase.verify_failure_msg: when set, exhausting the verification budget raises OptionExecutionFailure with the message instead of advancing (also when a retry rewind itself fails). - PhaseSkill._terminal now enforces the FINAL phase's verification: executors check terminal before calling the policy, so a final phase's verify_fn never ran in the policy's advance path and the option could end unverified. - create_pick_skill(verify_lift=True): at the end of LiftSlightly the target object must have gained at least half of lift_dz, else the option fails with a message telling the agent to fix its grasp_z_offset. This is the honest failure for grasps contact-level detection cannot reject: pads pinching a block's top EDGE show properly aligned normals on both fingers, but the support drags the block out of the constraint during the lift (proper grasps track the gripper to within a few mm; degenerate ones gain at most a third of the lift).
…test Root cause of the seed0 run_20260819_053515 failure: the agent's plan used grasp_z_offset=0.01 on a standing 10 cm leg, which sits on a sub-millimeter knife edge (0.009 grasps properly 7/7, 0.010 cams over the top corners 7/7 from the run's exact staging). The cam-over shoved the leg 17.6 mm into the table, held detection latched a degenerate constraint off a corner graze, the 3 cm lift left the leg still touching the table, and the place jammed it - 16-18 mm off site0, lying, episode dead. The agent's validation rollouts (4/4 with 0.01) happened to land on the good side of the edge; real execution landed 2.5 mm high and cammed. All six solved runs in the experiment used 0.0. PickBlock now sets verify_lift=True: a pick whose block did not rise with the gripper fails with OptionExecutionFailure instead of handing a ghost-held block to the place - the honest signal that taught seeds 1 and 2 to converge on offset 0.0. The regression test pins all three guards from the failed run's geometry: a single-finger touch with nothing closing on the other side is refused, an off-center pre-pinch is still captured, and PickBlock(leg0)[0.01] from reset must either fail honestly or genuinely lift the leg - never end "done" with the leg still on the table. Also clears two redundant local pybullet imports and an unused variable pylint flagged in this file.
finger_dist was assigned a bare None, so mypy inferred its type as None and flagged the `contact_distance < finger_dist` branch as unreachable. Annotate it Optional[float], which is what the loop below already assumes. create_pybullet_block takes orientation as a 4-tuple, but the motion planning tests passed a list. Only one of the four call sites was reported, because the other three sit in function bodies mypy does not check; fix all four so they do not surface later.
The endpoint loop skipped the held-proximity probe for any body already in contact_partners - including bodies that had just earned partner status via robot proximity in the same iteration. Two consequences, both against the documented contract: - held_near_endpoint missed such bodies, so held_body_clearances held the held assembly to the wide held-bystander clearance (1 cm in bridge) against an intended contact partner: a place goal with a finger pad within the bystander clearance of the support made the goal config collide and silently degraded the phase to non-collision-checked incremental IK. - endpoint_partners[1] missed a robot-earned start partner the held assembly deliberately approaches at the goal, so a movable placement neighbor was wrongly demoted to the zero margin outside the start neighborhood, over-rejecting flush goal placements. Run the held probe for every body whenever a held assembly exists; the robot probe no longer short-circuits it.
The start-escape rule for robot links gated on the shallow held-object margin. When the bridge -0.02 override was dropped (measured only on held-assembly start contacts), the robot-link escape window silently shrank to the default 6 mm - hard-rejecting the 7-15 mm reconstruction artifacts that motivated the rule. The masking test config override is removed, so test_robot_start_escape now exercises the default margins and pins the window. Give the rule its own _ROBOT_START_ESCAPE_MAX_DEPTH = -0.02: the allowance stays escape-only (never deeper than the start, only near the start), so the generous depth cannot be exploited elsewhere.
Restart seeds randomize every joint and IK leaves the fingers wherever the seed put them (they do not move the EE pose), so restart-derived goal candidates carried essentially random finger values. The chosen candidate's goal config is collision-checked (and the goal-side BiRRT tree grown) at those values while replay drives the fingers per finger_status - a fallback branch checked with near-closed fingers could execute with open fingers, under-checking gripper clearance in exactly the cluttered scenes where fallback branches get picked. It also made dedup treat arm-identical branches as distinct. Pin accepted candidates' finger entries to the current finger positions (after the FK accuracy check, which fingers cannot affect). The fake goal-IK robot grows finger indices and the exact-value expectations assert the pinning. Also fixes a stale _solve_goal_ik comment reference.
docformatter reflows in pybullet_bridge.py and the two test files (pure reflow, no wording changes); scripts/configs/predicatorv3/envs/ all.yaml lost its trailing newline in the margin-override drop.
The GT-sim hardening in this PR (cure gates, samplers, IK branches, grasp verification) makes the end-to-end solve deterministic; drop the xfail marker the previous PR added at its boundary.
An n-rollout gate passes a plan whose per-rollout success rate is p with probability p^n, so the old 3-rollout gate let marginal plans through: at p=0.85 it passes 61%. Bridge run_20260819_053515 is the motivating case - a grasp offset sitting on a sub-millimeter knife edge validated 3/3 (plus one probe rollout), then cammed out on the single real episode. Five rollouts cut the slip rate to 44% at the same p and much lower for worse plans; the escalated after-flaky gate doubles to 10 since a flaky rejection is direct evidence the agent is tuning in a marginal region. Cost is ~2 extra sim rollouts (seconds) per capture attempt.
Validation rollouts sampled execution variability the agent could neither see nor replay: repeats ran at undisclosed decorrelated planner seeds, so a FLAKY report named a failing step but gave no way to re-run that exact draw - the agent could only re-sample and hope (bridge run_20260819_053515: the knife-edge failure mode was only reachable by luck). Seeds are now first-class on both surfaces: - Every validation rollout and probe trial reports the planner seed it ran at; the FLAKY rejection names the failing seed and how to reproduce it. - evaluate_option_plan gains rollout_seed=S: one rollout at exactly that seed on a fresh env with full per-step reporting. Diagnostic only - never captured (an agent-chosen seed must not pass the capture gate) and stale-env scene renders are skipped. - evaluate_option_plan gains validation_rollouts=N: a stricter gate on demand, effective count max(configured, N) - it can raise the gate, never lower it. Combined with rollout_seed it instead runs exactly N diagnostic trials at seeds S..S+N-1, mirroring sim.run. - sim.run (explore_python) gains seed=S: single runs execute at S, trials=N runs trial i at S+i with per-trial seeds in the report, physics sweeps run every point at S. - absolute_rollout_seed joins decorrelated_rollout_seed in context.py as the shared scope both surfaces use. The SUBMIT guidance teaches the loop: flaky -> reproduce the reported seed -> fix the actual failure -> add margin -> resubmit (optionally with a stricter gate).
Import-order fix plus docstring reflows; one docstring docformatter would split mid-sentence is reworded into a proper summary and body.
yichao-liang
force-pushed
the
agent-plan-validation
branch
from
August 20, 2026 21:22
255b905 to
9f44c7e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 3/6 of the bridge-domain stack (based on #147).
Review fixes folded in on top: isort/docformatter conformance for the capture tests; one docstring docformatter would split mid-sentence is reworded.
🤖 Generated with Claude Code