ChainSafe / ChainSafe/lodestar

gloas block processing is ~5x slower than pre-gloas with a 4M-validator state (1M active)

Open
#10,024 0 comments 0 reactions 0 assignees View on GitHub
spec-gloas
Dominant language
TypeScript
Stars
1.4k
Forks
483
Avg merge
1d 16h
Merged PRs (30d)
150

Description

### Describe the bug

range sync post-gloas is bound by main thread cpu in block state transitions, a gloas block costs ~5x a pre-gloas block at the same validator count.

validator set: 1M active validators, but the registry holds ~4M validator records (genesis state 533 MB, the validator ranges include large exited ranges), so anything iterating the full registry (epoch processing, participation, inactivity scores) pays for 4M, not 1M.

measured on lodestar-geth-3 syncing glamsterdam-devnet-9 (1M active / ~4M total validators, gloas from slot 7200) from genesis on 2026-09-04, commit 605fa5a, 60 s windows of the prometheus counters:

| | pre-gloas | post-gloas |
|---|---|---|
| `lodestar_stfn_process_block_seconds` per block | 0.18 s (9239 calls, 1684 s) | 0.85-1.0 s |
| `lodestar_stfn_epoch_transition_seconds` per epoch | ~2.9 s | ~3 s |
| main thread busy (stfn + epoch + hash tree root) | | 0.9-1.0 s/s |
| `nodejs_eventloop_lag_p99_seconds` | | 4.9 s |
| blocks imported | ~1.4-1.7 slots/s | 0.35-0.47 blocks/s, ~1.0 slots/s |
| batch of 32 slots, processing time | 3-4 s | 29-34 s |

epoch transition steps by cumulative time over 232 epochs: `processRewardsAndPenalties` 282 s, `beforeProcessEpoch` 254 s, `processProposerLookahead` 97 s, `processInactivityUpdates` 55 s, `processEffectiveBalanceUpdates` 19 s, `processPtcWindow` 5 s.

ruled out as the bottleneck:

- downloads: the next batch was ready 5-7 s after processing of the current one started, `lodestar_sync_range_download_by_range_*` never limiting
- execution layer: `notifyNewPayload` 10 ms per call on geth, `lodestar_execution_engine_http_client_request_time_seconds`
- bls: 1.5 s/s of work but on the worker pool, 0.5 s/s queue wait
- gc: 0.01 s/s

part of the post-gloas number is #10011 (batches recomputed after the confirmed root throw, `process_block` 0.67/s vs 0.35-0.47 imports/s), the rest is the per-block cost itself. with #10011 merged the expected ceiling is ~2 slots/s on this hardware, i.e. ~3 hours for a 21k slot devnet.

### Expected behavior

a gloas block should not cost 5x a fulu block to process. needs a cpu profile of `processBlock` on gloas blocks with this validator set (1M active, ~4M in the registry) to see which step grew (payload attestations, bid processing, `processParentExecutionPayload`, ptc handling) and whether the epoch transition steps above can be cached or moved off the main thread.

### Steps to reproduce

wipe chain-db and sync glamsterdam-devnet-9 (or any gloas devnet with a multi-million validator registry) from genesis, compare `rate(lodestar_stfn_process_block_seconds_sum[1m]) / rate(lodestar_stfn_process_block_seconds_count[1m])` before and after the fork slot.

Contributor guide

Open the contributing guide

Research direction

Reproduce the slowdown by wiping chain-db and syncing glamsterdam-devnet-9, then compare the pre- and post-fork process_block metrics. Profile processBlock on gloas blocks with 1M active and about 4M total validators, checking payload attestations, bid processing, processParentExecutionPayload, ptc handling, and the listed epoch-transition steps. Done means identifying which step causes the regression and whether caching or off-main-thread work is viable.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
blockchain, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.