lambdaclass / lambdaclass/ethlambda
Proposer silently skips its slot after every proposer-rotation wrap (slot ≡ 1 mod 512); no failure counted
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 82
- Forks
- 28
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 20
Description
Observed
On a 512-validator / 64-node devnet (image devnet5-leanvm-main, rev 691df39, 8 validators/node, 8 subnets), validator 1 has missed 100% of its proposal slots: 8/8 at first measurement (slots 1, 513, 1025, 1537, 2049, 2561, 3073, 3585 — every slot ≡ 1 mod 512), and still 5/5 when re-sampled four days and ~170 rotations later (slots 84481…86529 all absent, adjacent control slots present).
The node never logs Building block for these slots, and lean_block_building_failures_total stays 0 fleet-wide: the interval-4 build tick never fires, so nothing counts the miss.
Context
- The rotation-wrap block (slot ≡ 0 mod 512, proposer = validator 0) is imported by peers 7–8 s after its slot start at every observed wrap (7.3 s / 7.2 s / 7.8 s at slots 512 / 3072 / 3584).
- node_1 (host of validator 1, also an aggregator) is seconds behind real time when its slot arrives: its attestations for slot 512k publish 3–7 s late and its aggregation sessions run ~3 s.
- At 3 of 5 recently sampled wraps the following proposer (validator 2's node) also lost its block — net cost 1–2 blocks per 512-slot cycle, invisible to every failure metric.
Expected
A proposer that cannot build in time should at least increment a failure counter; a deterministic 100%-miss pattern tied to the proposer-rotation wrap should be visible in metrics.
Evidence
- Chain: devnet-eth35, genesis 2026-08-28 16:53:53Z (
GENESIS_TIME 1787936033), hosts ethlambda-3/ethlambda-5. - Missed-slot verification via
GET /lean/v0/blocks/<slot>(404 for every s ≡ 1 mod 512 sampled; 200 for controls). - Full measurement context: 3.9 h instrumented window + 4-day soak; per-node logs and Prometheus/Loki history retained on the hosts.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the slot pattern on the devnet-eth35 setup and compare GET /lean/v0/blocks/ with adjacent control slots. Start from the proposer and interval-4 build paths that produce the “Building block” log and lean_block_building_failures_total metric. Done means missed proposer slots are counted and the deterministic post-wrap misses are visible in metrics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- prometheus, rust
- Domain
- blockchain, distributed-systems, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100