ChainSafe / ChainSafe/lodestar

Lodestar failed to produce block on glamsterdam-devnet-8

Open
#9,874 1 comment 0 reactions 0 assignees View on GitHub
spec-gloas
Dominant language
TypeScript
Stars
1.4k
Forks
483
Avg merge
1d 16h
Merged PRs (30d)
150

Description

On glamsterdam-devnet-8, `lodestar-reth-2` missed its slot 50656 proposal (gloas). Both the engine and builder assembly paths aborted at the same CL step:

```
Aug-20 12:51:12.241 [api] error: Block production failed slot=50656, parentSlot=50652, fork=gloas,
bidValue=200000000, builderIndex=9998,
engineReason=Error: SHUFFLING_CACHE_ERROR_NO_SHUFFLING_FOUND,
bidReason=Error: SHUFFLING_CACHE_ERROR_NO_SHUFFLING_FOUND
```

## Root cause

Attestation packing resolves the committee shuffling through `ShufflingCache.getBeaconCommittee` → `getShufflingOrThrow` → `getSync`, which **throws synchronously on a cache miss** ([shufflingCache.ts#L213-L223](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/chain/shufflingCache.ts#L213-L223)), called from [aggregatedAttestationPool.ts#L779](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/chain/opPools/aggregatedAttestationPool.ts#L779). Block production is the only shuffling consumer that neither populates the cache nor falls back on a miss, so a single uncached shuffling aborts the whole proposal.

## Reconstructed stack trace

The inner stack is not in the logs: [index.ts#L1038](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/api/impl/validator/index.ts#L1038) throws a fresh `Error` from `String(engineReason)`/`String(bidReason)` (the `Promise.allSettled` rejection reasons), discarding the original `ShufflingCacheError`. The only logged beacon-node frame is `at Object.produceBlockV4 (…/api/impl/validator/index.ts:1038)`. Reconstructed from source:

```
ShufflingCacheError: SHUFFLING_CACHE_ERROR_NO_SHUFFLING_FOUND { epoch, decisionRoot }
at ShufflingCache.getShufflingOrThrow (chain/shufflingCache.ts:216) <- throw; getSync() returned null at :214 (miss++)
at ShufflingCache.getBeaconCommittees (chain/shufflingCache.ts:209)
at ShufflingCache.getBeaconCommittee (chain/shufflingCache.ts:205)
at notSeenValidatorsFn (closure) (chain/opPools/aggregatedAttestationPool.ts:779)
at getAttestationsForBlockElectra (chain/opPools/aggregatedAttestationPool.ts:297)
at getAttestationsForBlock (chain/opPools/aggregatedAttestationPool.ts:221)
at produceCommonBlockBody (chain/produceBlock/produceBlockBody.ts:1043)
at BeaconChain.produceCommonBlockBody (chain/chain.ts:1048)
at produceBlockV4 (api/impl/validator/index.ts:922) <- commonBlockBodyPromise
-- rejection awaited via resolveOrRacePromises (index.ts:971), re-thrown at index.ts:1038 --
```

## Evidence

1. **Every other consumer populates the cache; block production doesn't.** `shufflingCache.processState(state)` is called from `importBlock` ([L473](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/chain/blocks/importBlock.ts#L473)), `verifyBlock` ([L86](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/chain/blocks/verifyBlock.ts#L86)), gossip validation ([block.ts#L264](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/chain/validation/block.ts#L264)), and `onCheckpoint` ([chain.ts#L1649](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/chain/chain.ts#L1649)). Attestation *verification* additionally has an async fallback (`insertPromise` + regen, [chain.ts#L1437-L1457](https://github.com/ChainSafe/lodestar/blob/unstable/packages/beacon-node/src/chain/chain.ts#L1437-L1457)). The block-production path (`produceCommonBlockBody` → `getAttestationsForBlock`) only *reads* the cache and does neither.

2. **The shuffling was available — only uncached.** Metrics for the node in the same minute as the failure:

| metric (delta 12:51-52) | value |
|---|---|
| `shuffling_cache_miss_count` | **2** |
| `shuffling_cache_insert_promise_count` | **1** |
| `shuffling_cache_promise_resolution_time_seconds_count` | **1** (resolved, ~0 s) |
| `shuffling_cache_hit_count` (baseline) | ~500-600 / min |

The commonBlockBody is built once and shared by the engine and builder `produceBlock` calls, and `getShufflingOrThrow` throws on the first committee — so block production accounts for **1** of the 2 misses. The other miss is concurrent gossip attestation verification hitting the *same* uncached `(epoch, decisionRoot)`: it inserted a promise, regenerated the shuffling, and resolved it near-instantly (the `insertPromise=1` / `resolved=1` above). Same root cause, two consumers — the async one recovered, block production (sync `getSync`) had no fallback and died.

3. **Trigger was a late block import, not a reorg.** At proposal time the head was `0x3c77…`@50652; the slot-50655 block `0x3ebc…673b` hadn't been imported yet (`Synced slot: 50655 - head: (slot -3) 0x3c77…3764`). It arrived ~1.6 s later (`duties reorg … head=0x3ebc…` at 12:51:13.8) — a forward extension / dependent-root shift, no `Chain reorg`, nothing orphaned. The epoch-1582 shuffling for that head's decision root wasn't cached. Rare condition, made likely by a badly unfinalized chain (finalized epoch 1534 while at 1583).

## Suggested fix

Call `shufflingCache.processState(currentState)` in the block-production path before attestation packing (mirroring `verifyBlock`); the regenerated state's `epochCtx` already holds the shuffling with the correct decision roots. Optionally, have `getAttestationsForBlock` skip an uncached committee rather than throw, so one missing shuffling can never abort an entire proposal.

Contributor guide

Open the contributing guide

Research direction

Start with packages/beacon-node/src/chain/shufflingCache.ts and packages/beacon-node/src/chain/opPools/aggregatedAttestationPool.ts, then trace produceCommonBlockBody through packages/beacon-node/src/api/impl/validator/index.ts. Compare block production with the processState and async fallback paths named in the issue. Done means an uncached shuffling no longer aborts the slot-50656 proposal, with the relevant block-production behavior covered by the repository's existing tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
blockchain
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.