ChainSafe / ChainSafe/lodestar
Gloas/Heze mainnet sanity/slots/historical_accumulator spec test is ~7.5x slower than pre-Gloas forks
- Dominant language
- TypeScript
- Stars
- 1.4k
- Forks
- 483
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 150
Description
## Summary
`sanity/slots/pyspec_tests/historical_accumulator` takes ~23-24s for gloas and heze on
the mainnet preset, vs ~1-6s for every pre-gloas fork. The sanity `slots` timeout is
30000ms (`packages/beacon-node/test/spec/presets/sanity.test.ts:49`), so these two cases
run at 76-81% of the budget and are a timeout-flake risk on slower CI runners.
## Measurements
mainnet preset, `sanity/slots/pyspec_tests/historical_accumulator`, local run @ v1.7.0-alpha.13:
| fork | duration |
| --- | --- |
| phase0 | 1003ms |
| bellatrix | 2241ms |
| deneb | 2598ms |
| electra | 3032ms |
| fulu | 3142ms |
| altair | 5003ms |
| capella | 5847ms |
| **gloas** | **22948ms** |
| **heze** | **24234ms** |
gloas is 7.3x fulu; heze is 7.7x fulu. heze is consistently ~1.3s above gloas.
On the **minimal** preset all forks finish in <=27ms (gloas 26ms, heze 27ms), so this is
mainnet-only.
## Why mainnet only
The test runs `process_slots(state, state.slot + SLOTS_PER_HISTORICAL_ROOT)` — 8192 slot
transitions on mainnet vs 64 on minimal.
## Likely cause
Two gloas-specific changes make each of those 8192 slot transitions more expensive:
1. `packages/state-transition/src/slot/index.ts:34-40` — every post-gloas slot writes into
`executionPayloadAvailability`, a `BitVector[SLOTS_PER_HISTORICAL_ROOT]` (8192 bits on
mainnet):
```ts
if (fork >= ForkSeq.gloas) {
// Unset the next payload availability
(state as CachedBeaconStateGloas).executionPayloadAvailability.set(
(state.slot + 1) % SLOTS_PER_HISTORICAL_ROOT,
false
);
}
That is 8192 ViewDU writes into a large bitvector, each dirtying nodes that must be
re-merkleized on commit.
2. The gloas BeaconState is a ProgressiveContainer (EIP-7688), which has a different
merkleization cost profile than the fixed-depth containers used pre-gloas. heze widens
it further, consistent with heze being slower than gloas.
Not yet profiled — the split between (1) and (2) is unknown.
Impact
- Timeout-flake risk: 76% (gloas) / 81% (heze) of the 30s budget.
- Inflates total mainnet spec-test wall clock.
- Any future fork inherits the regression.
Reproduce
pnpm vitest run --project spec-mainnet \
packages/beacon-node/test/spec/presets/sanity.test.ts -t historical_accumulator
Proposed short term
Skip the two cases with a pointer to this issue, rather than raising the timeout —
raising it hides the regression. See #9763 discussion.
Investigation ideas
- Profile to attribute cost between the bitvector write and progressive-container hashing.
- Consider deferring/batching the executionPayloadAvailability reset so process_slots
does not commit a large bitvector every slot.
- Compare against a fixed-depth BitVector control to isolate the EIP-7688 contribution.
Contributor guide
Research direction
Run the reproduced command for the mainnet sanity test and inspect packages/beacon-node/test/spec/presets/sanity.test.ts:49 and packages/state-transition/src/slot/index.ts:34-40. Profile the historical_accumulator cases to separate the BitVector write cost from progressive-container hashing. Done means the regression is attributed and an agreed mitigation reduces the timeout-flake risk without simply raising the timeout.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- blockchain, performance, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100