ChainSafe / ChainSafe/lodestar
Gloas/ePBS: block production drops same-FFG attestations voting for the other payload-status variant (FULL/EMPTY)
- Dominant language
- TypeScript
- Stars
- 1.4k
- Forks
- 483
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 150
Description
### Summary
Under Gloas/ePBS, a parent block can be voted on as two distinct LMD variants — **FULL** (payload delivered timely) or **EMPTY** (payload not/late) — which fork choice tracks as separate nodes. When Lodestar produces a block, it appears to drop attestations whose head (LMD) vote points at the *other* payload-status variant than Lodestar's own fork-choice head, **even though their FFG (source/target) vote is identical and they are valid to include**.
Other clients (Lighthouse, Teku, Prysm) consistently pack these other-variant attestations; Lodestar consistently does not. Net effect is **lost attestation inclusion / reduced proposer rewards** — not a safety issue (FFG matches, fork choice tracks correctly).
Reported by @nflaig on `glamsterdam-devnet-5`; investigated together with @twoeths. Filing so we don't lose it — **status is "needs further devnet observation"** (see below), not a confirmed root cause.
### Observed behavior
- Lodestar block at [devnet-5 slot 38871](https://dora.glamsterdam-devnet-5.ethpandaops.io/slot/38871#attestations) did **not** include the EMPTY votes for parent [slot 38868](https://dora.glamsterdam-devnet-5.ethpandaops.io/slot/38868).
- The next-proposer Lighthouse block at [slot 38872](https://dora.glamsterdam-devnet-5.ethpandaops.io/slot/38872#attestations) **did** include them.
- Confirmed on further examples: slots **38890**, **41167**, **41398** (Teku), and [**41307**](https://dora.glamsterdam-devnet-5.ethpandaops.io/slot/41307) as an opposite example. At [slot 41483](https://dora.glamsterdam-devnet-5.ethpandaops.io/slot/41483) Lodestar *did* pack them (after correctly reorging two malicious `lodestar-ethrex-1` blocks).
> "I am pretty sure we should include votes in our blocks even if they do not support our head view (LMD) as we still have same FFG" — @nflaig
>
> "I am just confused why other clients consistently pack these attestations voting on EMPTY while we do not" — @nflaig
**Reproduction on the devnet:** `lodestar-ethrex-1` blocks were "malicious" (voting EMPTY despite a timely payload), which reliably creates the FULL↔EMPTY divergence. Check any `lodestar-ethrex-1` block where another Lodestar node is the next proposer, and inspect whether that Lodestar block packs the other-variant attestations.
### Expected behavior
Lodestar block production should include valid same-FFG attestations regardless of which payload-status (FULL/EMPTY) LMD variant they vote for, matching Lighthouse/Teku/Prysm. It should not drop otherwise-includable attestations just because their head vote points at the non-canonical-from-our-view variant of the parent.
### Suspected mechanism (needs confirmation)
Block-production attestation selection runs through `AggregatedAttestationPool.getAttestationsForBlockElectra` → `getValidateAttestationDataFn` → `isValidShuffling`, which resolves the attestation's `beaconBlockRoot` via **`forkChoice.getBlockHexDefaultStatus(beaconBlockRootHex)`** — i.e. the *default-status* variant only. Under ePBS the same block root has FULL and EMPTY variants; resolving to the default status can mis-handle attestations that voted for the other variant.
- `packages/beacon-node/src/chain/opPools/aggregatedAttestationPool.ts` — `getAttestationsForBlockElectra` (validation via `validateAttestationDataFn`), `getValidateAttestationDataFn`, and the `getBlockHexDefaultStatus` lookup in `isValidShuffling`.
- Entry point: `packages/beacon-node/src/chain/produceBlock/produceBlockBody.ts` → `aggregatedAttestationPool.getAttestationsForBlock(...)`.
Related: **#9500** ("fix: get attestation block head", @twoeths) selects the correct head-block variant. Per @twoeths it's a precision improvement — *"does not look like an issue because we track correctly in forkchoice"* — so #9500 alone may not be the fix for the packing gap.
### Open questions
1. Is the omission the LMD-variant filtering above, or the separate **known `SingleAttestation`-not-packed gap** (block producer not aggregating same-data `SingleAttestation`s at pack time), or both? These got somewhat conflated during the investigation — e.g. the index-0 attestation in LH block 38872 was actually for slot 38869, and there were only 19 of them.
2. Does a Lodestar block ever contain *both* FULL and EMPTY attestations for the previous slot? (Never definitively answered.)
3. Is it possible no aggregator had produced an aggregate for those attestations beforehand (so nothing was in the pool to pack)?
4. Does #9500 change any observed packing behavior once merged?
### Status
Needs further observation on the current devnet to re-confirm frequency and to isolate variant-filtering vs `SingleAttestation`-aggregation as the actual cause(s). Confirmed so far: FFG votes match across variants (so inclusion is valid), fork choice tracks the correct variant, and other clients consistently include these attestations while Lodestar does not.
### Additional context
- Network: `glamsterdam-devnet-5` (ethpandaops). Fork: **Glamsterdam / Gloas (ePBS)** — the FULL/EMPTY LMD variant split only exists under ePBS.
- Origin: ChainSafe/lodestar dev Discord thread "attestation packing issue" (2026-06-09 → 2026-07-05). Participants: @nflaig, @twoeths; cc @wemeetagain.
- Distinct from #7883 (post-Pectra packing of newly-seen attestations on mainnet — aggregate-pool `SeenAggregatedAttestations` committee-index root cause). Different fork, different mechanism.
---
🤖 Filed with AI assistance (lodekeeper) at @nflaig's request, summarizing the Discord investigation thread.
Contributor guide
Research direction
Start with packages/beacon-node/src/chain/produceBlock/produceBlockBody.ts and trace getAttestationsForBlock into aggregatedAttestationPool.ts, especially getAttestationsForBlockElectra, getValidateAttestationDataFn, and the getBlockHexDefaultStatus lookup. Reproduce the behavior on the listed devnet examples, then distinguish LMD-variant filtering from the SingleAttestation aggregation gap and check the effect of #9500. Done means valid same-FFG FULL and EMPTY attestations are both included when available.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- blockchain
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100