ethereum-optimism / ethereum-optimism/optimism
op-devstack: supernode route readiness timeout flakes ZK dispute-monitor test
- Dominant language
- Go
- Stars
- 6.5k
- Forks
- 4k
- Avg merge
- 2d 38m
- Merged PRs (30d)
- 164
Description
## CI failure
**CI run:** https://app.circleci.com/pipelines/github/ethereum-optimism/optimism/134225/workflows/adf3a44c-bbf9-4e53-9a53-1155c50b5774/jobs/5557165
**Job:** `memory-all-opn-op-reth-fusaka` (shard 3)
**Commit:** `bf7f8937cf9c80e6274a43851680953569197319`
```text
Error Trace: op-devstack/sysgo/multichain_supernode_runtime.go:777
Error: supernode route readiness
Test: TestZKDisputeMonitorCanonicalChildOfInvalidParent
Messages: timed out waiting for supernode route http://127.0.0.1:43139/901
```
## What happened
The test failed during devstack setup, before exercising any ZK dispute-monitor behavior:
1. Both op-reth L2 processes started and reported ready.
2. The supernode and its RPC router started.
3. Both supernode virtual nodes began initialization and opened their safe-head databases, but did not finish initialization/register their per-chain RPC APIs within the readiness budget.
4. `waitForSupernodeRoute` failed while probing `optimism_rollupConfig`.
5. Test teardown canceled the virtual nodes, which then reported `failed to init the runtime config: failed to load runtime configuration repeatedly, last error: context canceled`.
`waitForSupernodeRoute` has a hard-coded 15-second wall-clock deadline, but each `CallContext` uses the RPC client's default 10-second call timeout and the deadline is checked only between calls. In this run two blocked probes consumed about 20 seconds before the helper failed. The helper therefore neither enforces its stated deadline precisely nor tolerates transient full-devstack startup delays.
The acceptance runner executes eight full-devstack packages concurrently on each CircleCI node. Another heavy multi-node test was failing concurrently on this same shard, so scheduling/resource contention is a plausible trigger, although the logs do not prove exactly where virtual-node initialization stopped making progress.
## Flake evidence
- The tested commit only changed executor sizing for the separate Kona acceptance job; no supernode, devstack, ZK test, or OP Node executor behavior changed.
- The immediately preceding 14 sampled `develop` runs passed this test.
- The immediately following `develop` run passed the test in 84.41s: https://app.circleci.com/pipelines/github/ethereum-optimism/optimism/134226/workflows/5359111f-a882-4fe7-88c7-c21416cc8eb3/jobs/5557266
- CircleCI Insights showed 97 recent successful runs in the Kona acceptance variant before this failure.
## Suggested direction
Replace the ad hoc deadline loop with a context-bounded readiness wait whose individual probes are bounded by the remaining overall budget, and use a startup budget appropriate for full devstack initialization (or a direct virtual-node readiness signal). Preserve useful diagnostics for the last probe/startup error.
Contributor guide
Research direction
Read op-devstack/sysgo/multichain_supernode_runtime.go around waitForSupernodeRoute and the failing TestZKDisputeMonitorCanonicalChildOfInvalidParent setup. Reproduce the CircleCI failure or run the named test, then verify that readiness handles full-devstack startup delays within a bounded budget and preserves the last probe or startup error in diagnostics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- devops, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100