ethereum-optimism / ethereum-optimism/optimism

C-flake: TestUnsafeChainNotStalling_CLSync family regressing again (post-#20950)

Open
#21,041 2 comments 0 reactions 0 assignees View on GitHub
C-flake
Dominant language
Go
Stars
6.5k
Forks
4k
Avg merge
2d 38m
Merged PRs (30d)
164

Description

**Claude:** Tracking flaky test investigation, regression after the previous fix family was closed.

**Test:** `TestUnsafeChainNotStalling_CLSync` (and `_RestartOpNode`, plus `_ELSync` and `_ELSync_RestartOpNode` siblings)
**Package:** `github.com/ethereum-optimism/optimism/op-acceptance-tests/tests/depreqres/syncmodereqressync/{clsync,elsync}` and `.../reqressyncdisabled/{clsync,elsync}`
**Job:** `memory-all-opn-op-reth` / `memory-all-kona-op-reth`
**Recent flake counts (CircleCI Insights, May 14–26):**
- `TestUnsafeChainNotStalling_CLSync_RestartOpNode`: 3 flakes in pipeline 126337 (May 26)
- `TestUnsafeChainNotStalling_CLSync_Long`: 9 flakes in pipeline 126044, 3 in 125875
- `TestUnsafeChainNotStalling_ELSync_Long`: 7 flakes in pipeline 125889, 5 in 125657
- multiple other 1-2-flake bursts across `_Short`, `_Long`, `_RestartOpNode`, `_RestartOpNode_Long` variants

**Symptom (most recent failure on PR #21036, build 5117903):**
```
operation failed permanently after 30 attempts: expected head to advance: unsafe
Test: TestUnsafeChainNotStalling_CLSync
```
Right before the assertion, `Chain sync status name=b chain=901 label=unsafe target=44 current=40` — L2CLB ran out of attempts 4 blocks short of the (fixed) target snapshot.

**Context:**
- PR #20950 (May 22) collapsed `_Short` and `_Long` variants into a single 20-block run and set `WithUniformL2BlockTimes(1)` (1s L2 blocks). The flake predates this PR — it has been intermittent on the test family for weeks — but the new configuration may shift the timing margins.
- The post-reconnect catch-up uses `sys.L2CLB.Reached(safety.LocalUnsafe, ssA_after.UnsafeL2.Number, 30)`, which polls every 2 s for a maximum of 60 s. The sequencer keeps producing at 1 block/s, but the target is fixed at the snapshot taken before reconnect, so L2CLB needs to sync ~20 blocks via ReqRespSync inside that 60 s window.
- The earlier fix family lives in PRs #19612, #19613, #19683 (`d47414a378 fix(op-acceptance-tests): replace time.Sleep with deterministic block advancement in UnsafeChainNotStalling`), #19418, #20868 (`replace sleep polling`).

**Test source:**
- `op-acceptance-tests/tests/depreqres/common/common.go` — `UnsafeChainNotStalling_Disconnect` and `_RestartOpNode` helpers
- `op-acceptance-tests/tests/depreqres/syncmodereqressync/clsync/clsync_test.go`
- `op-devstack/dsl/l2_cl.go` — `ReachedFn`, `AdvancedFn`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.