Skip to content

fix: avoid payload prep failures on gloas reorgs - #9723

Open
lodekeeper wants to merge 2 commits into
ChainSafe:unstablefrom
lodekeeper:fix/gloas-proposer-reorg-payload-rootcause
Open

fix: avoid payload prep failures on gloas reorgs#9723
lodekeeper wants to merge 2 commits into
ChainSafe:unstablefrom
lodekeeper:fix/gloas-proposer-reorg-payload-rootcause

Conversation

@lodekeeper

Copy link
Copy Markdown
Contributor

Motivation

Replaces the closed #9721 canonical-head retry workaround. If Gloas proposer-head selection intentionally builds on a weak-block parent, payload preparation must keep the Engine FCU inputs coherent with that selected head and avoid canonicalizing the EL to a weak late child when the next proposer is local.

Description

  • Add getSafeExecutionBlockHashForHead() so payload prep sends a safe hash only when the confirmed block is ancestor/equal to the selected FCU head.
  • Track locally submitted Gloas proposer preferences separately from gossiped remote preferences.
  • Use local Gloas proposer preferences in importBlock FCU suppression, so a weak late block does not advance the EL canonical head before a local proposer-boost reorg duty.
  • Mark duplicate local proposer-preference submissions as local without replacing the already-verified pool entry.

Testing

  • pnpm build
  • pnpm vitest run --project unit packages/fork-choice/test/unit/forkChoice/safeBlocks.test.ts packages/beacon-node/test/unit/api/impl/validator/produceBlockV4.test.ts packages/beacon-node/test/unit/chain/opPools/proposerPreferencesPool.test.ts
  • pnpm --filter @lodestar/fork-choice check-types
  • pnpm --filter @lodestar/beacon-node check-types
  • pnpm lint

AI assistance: implemented with Lodekeeper/Codex assistance.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6cebf799a3

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

await validateGossipProposerPreferences(chain, signed);

chain.proposerPreferencesPool.add(signed);
chain.proposerPreferencesPool.add(signed, {local: true});

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Defer local marking until gossip publish succeeds

When publishProposerPreferences() rejects after validation has inserted the entry, this line still marks the preference as local even though the API call fails. A retry then hits ALREADY_KNOWN and returns before publishing, while importBlock() can suppress the EL FCU via isKnownLocal(), so a transient publish error can leave builders without the preference but make the node act as if local Gloas prep succeeded. Mark it local only after publish succeeds, or roll back the pool entry on failure.

Useful? React with 👍 / 👎.

@lodekeeper lodekeeper Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, this was valid. validateGossipProposerPreferences() inserts the entry before the API publishes it, so marking it local before the publish await made the retry path unsafe exactly as you described.

Fixed in ba014eeca4: the API now rolls back the just-validated pool entry if publishProposerPreferences() rejects, and only marks it local after publish succeeds. I added a pool regression test so rollback only removes the exact stored entry and clears the local marker.

Verification:

  • pnpm vitest run --project unit packages/beacon-node/test/unit/chain/opPools/proposerPreferencesPool.test.ts
  • pnpm --filter @lodestar/beacon-node check-types
  • pnpm exec biome check packages/beacon-node/src/api/impl/validator/index.ts packages/beacon-node/src/chain/opPools/proposerPreferencesPool.ts packages/beacon-node/test/unit/chain/opPools/proposerPreferencesPool.test.ts
  • pnpm lint

@nflaig

nflaig commented Aug 1, 2026

Copy link
Copy Markdown
Member

@lodekeeper what is this?

@lodekeeper

Copy link
Copy Markdown
Contributor Author

Sorry for the thin context @nflaig — short version:

Problem (Gloas): when local proposer-boost-reorg logic intentionally selects a weak late block's parent as the FCU head, payload prep could still hand the EL a safe/canonical hash derived from the other head — so the Engine FCU inputs and the beacon node's selected head disagree. The motivating case was the devnet-7 missed local proposal: a weak late block had already advanced the EL canonical head right before a local proposer-boost reorg duty, and reth then returned payloadId=null on the self-build.

What it does:

  • getSafeExecutionBlockHashForHead() (fork-choice) — only send a safe hash when the confirmed block is an ancestor of / equal to the selected FCU head, so payload prep can't canonicalize the EL onto a weak late child.
  • Track locally-submitted Gloas proposer preferences separately from gossiped remote ones, and use the local set in importBlock FCU suppression — so a weak late block doesn't advance the EL canonical head just before a local reorg duty.

History: this replaces #9721 (closed). That was a produceBlockV4 retry workaround, which you + Codex rightly flagged as the wrong layer — this moves the fix down to the fork-choice / FCU / payload-prep layer instead.

It's a subtle path, so happy to walk through any of it — and if you think this is still the wrong layer or approach, I'd rather hear that before polishing it further.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

2 participants