erigontech / erigontech/erigon
Bump pectra kurtosis suite to assertoor v0.1.2 (currently kept on v0.0.17 — flaky)
- Dominant language
- Go
- Stars
- 3.6k
- Forks
- 1.5k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 455
Description
## Summary
The `pectra` kurtosis/assertoor suite is flaky under assertoor **v0.1.2**. In #21449 (which bumps assertoor v0.0.17 → v0.1.2 for the other suites), pectra is **left on its main config** (assertoor v0.0.17, lighthouse v7.0.1, teku 25.9.1, 2 nodes) so it keeps passing as on main — it **still runs in CI**, just not on v0.1.2. This issue tracks bumping pectra to v0.1.2.
All findings below are **CL test-infra / assertoor strictness, not erigon bugs** (erigon EL stays `VALID`; caplin isn't in this suite — it's teku + lighthouse).
## Root causes found during #21449
1. **2-node gossip-mesh fragility.** pectra runs `1 erigon × {teku, lighthouse}` = 2 nodes (1 peer each). gossipsub can't sustain the block-propagation mesh with a single peer, so teku fails to publish blocks (`No peers for message topics [beacon_block]`) and clients diverge. Verified locally: 2-node → 7+ publish failures + slot gaps; **4 nodes (2 teku + 2 lighthouse) → 0 failures, clean chain.**
2. **Zero-tolerance `check_consensus_forks` (`want <= 0`).** v0.1.2's stability-check fails on `too many forks` when clients are even 1 block apart transiently (normal propagation lag). Even with the 4-node gossip fix (0 `No peers`), CI `pectra_parallel` failed on a transient 1-block fork (teku group at block N, lighthouse group at N+1). Timing-sensitive: passes locally (fast), fails on slower CI runners.
3. **`el-triggered-consolidation` timing.** `error generating consolidation: source validator (index: …) not found in validator set` — consolidation runs before deposited validators activate. Seen locally after stability-check passed.
## To bump pectra to v0.1.2
- Give pectra ≥3–4 nodes (2 teku + 2 lighthouse) — fixes the gossip mesh.
- Relax the stability-check fork tolerance (allow transient ≤1-block forks / require a fork to persist) — needs a custom assertoor stability-check instead of `run_stability_check: true`.
- Address the consolidation test's deposit-activation timing.
- Then bump `pectra.io` to assertoor v0.1.2 + lighthouse v8.1.3 + teku 26.4.0 + 4 nodes.
> Note: the `regular` suite is also 2-node with the same clients; it passes today but shares the gossip fragility (latent flake risk).
Contributor guide
Research direction
Start with the pectra.io configuration and the pectra_parallel CI suite, then compare the assertoor v0.1.2 changes from #21449. Reproduce the gossip, fork-tolerance, and consolidation timing failures before changing the node layout or stability check. Done means pectra runs on assertoor v0.1.2 with the specified client versions and four nodes, without these failures.
Written by the indexing model from the issue text.
Assessment
- Domain
- ci-cd, testing-qa
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100