lambdaclass / lambdaclass/ethlambda

Proposer silently skips its slot after every proposer-rotation wrap (slot ≡ 1 mod 512); no failure counted

Open
#602 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
82
Forks
28
Avg merge
1d 20h
Merged PRs (30d)
20

Description

Observed

On a 512-validator / 64-node devnet (image devnet5-leanvm-main, rev 691df39, 8 validators/node, 8 subnets), validator 1 has missed 100% of its proposal slots: 8/8 at first measurement (slots 1, 513, 1025, 1537, 2049, 2561, 3073, 3585 — every slot ≡ 1 mod 512), and still 5/5 when re-sampled four days and ~170 rotations later (slots 84481…86529 all absent, adjacent control slots present).

The node never logs Building block for these slots, and lean_block_building_failures_total stays 0 fleet-wide: the interval-4 build tick never fires, so nothing counts the miss.

Context

  • The rotation-wrap block (slot ≡ 0 mod 512, proposer = validator 0) is imported by peers 7–8 s after its slot start at every observed wrap (7.3 s / 7.2 s / 7.8 s at slots 512 / 3072 / 3584).
  • node_1 (host of validator 1, also an aggregator) is seconds behind real time when its slot arrives: its attestations for slot 512k publish 3–7 s late and its aggregation sessions run ~3 s.
  • At 3 of 5 recently sampled wraps the following proposer (validator 2's node) also lost its block — net cost 1–2 blocks per 512-slot cycle, invisible to every failure metric.

Expected

A proposer that cannot build in time should at least increment a failure counter; a deterministic 100%-miss pattern tied to the proposer-rotation wrap should be visible in metrics.

Evidence

  • Chain: devnet-eth35, genesis 2026-08-28 16:53:53Z (GENESIS_TIME 1787936033), hosts ethlambda-3/ethlambda-5.
  • Missed-slot verification via GET /lean/v0/blocks/<slot> (404 for every s ≡ 1 mod 512 sampled; 200 for controls).
  • Full measurement context: 3.9 h instrumented window + 4-day soak; per-node logs and Prometheus/Loki history retained on the hosts.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the slot pattern on the devnet-eth35 setup and compare GET /lean/v0/blocks/ with adjacent control slots. Start from the proposer and interval-4 build paths that produce the “Building block” log and lean_block_building_failures_total metric. Done means missed proposer slots are counted and the deterministic post-wrap misses are visible in metrics.

Written by the indexing model from the issue text.

Assessment

Tech stack
prometheus, rust
Domain
blockchain, distributed-systems, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.