Gloas: builder bids validated after the one-shot bid-pool read — ~20% of proposals fall back to self-build

### Summary

On glamsterdam-devnet-7 (Gloas/ePBS, `v1.44.0/94954a2`, `glamsterdam-devnet-7` branch), lodestar proposers fall back to self-building on **~20% of proposals** even though a correctly-keyed builder bid was on the wire ~390 ms before slot start. The cause is measurable end-to-end: **the bid's gossip validation frequently completes *after* `produceBlockV4` has already done its single, synchronous bid-pool read at slot start.** The bid loses a race it was never given a chance to re-enter.

Full write-ups (interactive charts, per-slot evidence, reproduction queries):
- Investigation: https://panda-uploads-production.devops-539.workers.dev/panda/uploads/05a34d/lodestar-bid-validation-race.html
- Validation-latency charts: https://panda-uploads-production.devops-539.workers.dev/panda/uploads/ef97ac/lodestar-bid-validation-latency.html
- Context (network-wide block-building report): https://panda-uploads-production.devops-539.workers.dev/panda/uploads/0e1170/glamsterdam-devnet-7-self-built-blocks-round-2.html

### Mechanism (at `94954a2`)

1. [`produceBlockV4`](https://github.com/ChainSafe/lodestar/blob/94954a26409a881ddd3e929365ac14efbd135411/packages/beacon-node/src/api/impl/validator/index.ts#L939-L944) runs at slot start and calls `executionPayloadBidPool.getBestBid(slot, bidParentBlockHash, parentBlockRootHex)` **once** — a synchronous exact-triple map read with no wait, no re-check, and no pool-insert wake-up. `null` → self-build (there is no value comparison yet; the `TODO GLOAS: respect builderSelection…` above it).
2. The only path into that pool is [`validateGossipExecutionPayloadBid`](https://github.com/ChainSafe/lodestar/blob/94954a26409a881ddd3e929365ac14efbd135411/packages/beacon-node/src/chain/validation/executionPayloadBid.ts), which includes a per-bid `chain.regen.getBlockSlotState(..., {dontTransferCache: true})` plus BLS verification.
3. buildoor (the devnet builder) publishes its first bid at **−400 ms** and its last at **−100 ms** before slot start (its defaults), so the entire budget between bid arrival and the pool read is ~390 ms — all of which is consumed by queueing + validation when the tail bites.

### Evidence

Bid pipeline latency per implementation — each node's own beacon-event emission (fires post-validation) minus the earliest emission network-wide for the same bid; same bids, same mesh, 2.58 M observations, 2026-07-22 → 07-31:

| CL | p50 | p90 | p99 |
|---|---|---|---|
| prysm / grandine / lighthouse / teku / nimbus | 4–7 ms | 8–13 ms | 29–85 ms |
| **lodestar** | **22 ms** | **751 ms** | **1.36 s** |

The distribution is bimodal: a fast mode like every other client, plus a **second population (~16% of bids) at 400–1,500 ms** — a hump, not a tail, suggesting discrete stalls (event-loop pause / queue behind an expensive item / regen miss) rather than uniformly slower validation.

Per-slot causality on 7,850 lodestar proposals where the proposing node's own bid timeline is measurable:

| outcome | own validation of the bid (median) | n |
|---|---|---|
| bid taken | **−365 ms** (before slot start) | 6,461 — 1 late |
| self-built | **+510 ms** (after the decision; p90 +1,034 ms) | 1,389 — 1,267 late |

The daily fraction of late-validated bids tracks the daily self-build rate almost 1:1 (e.g. 24.1%/20.8%, 28.0%/21.3%, 24.0%/25.0%). The excess on 07-23 is the `BuilderCircuitBreaker` (working as intended during a payload non-reveal incident). A node restart on 07-28 did **not** reset the latency, so it is workload-shaped, not accumulated process state; what fattened the tail from 07-25 onward is still open and likely needs your internal gossip-queue/regen metrics.

### Impact

On the devnet this is benign (the self-built fallback works and reveals early — every one of those payloads landed). But with the current one-shot read, a validator forfeits the builder bid on every slot the race is lost — the bids left on the table in this window were worth ~2× the local block. It also means lodestar's effective bid acceptance depends on its own validation latency, which seems unintended.

### Possible fixes (any one would eliminate the effect)

- Re-check the pool (or wait briefly / wake on pool insert) before committing to the local payload in `produceBlockV4` — even a bounded ~500 ms grace re-read would recover nearly all of the 400–1,500 ms hump.
- Get the per-bid state regen out of the gossip-validation hot path (cache the prepared parent-branch state for the expected next slot, as bid validation for slot N almost always needs the same state).
- A metric on bid gossip-validation queue+processing time would make the tail visible without external tooling.

Happy to provide raw data, per-slot samples, or run instrumented builds on the devnet.

Start by reading produceBlockV4 in packages/beacon-node/src/api/impl/validator/index.ts and validateGossipExecutionPayloadBid in packages/beacon-node/src/chain/validation/executionPayloadBid.ts, then trace how bids enter executionPayloadBidPool. Compare the slot-start getBestBid call with the reported validation timing and possible re-check or wake-up approaches. Done means validated bids are not lost to the one-shot read while the self-build fallback remains intact.

Written by the indexing model from the issue text.