ethereum-optimism / ethereum-optimism/optimism

flaky test: `TestSupernodeEngineControllerConnectsAfterELUnavailableAtStartup`

Open
#22,013 0 comments 0 reactions 0 assignees View on GitHub
C-flake
Dominant language
Go
Stars
6.5k
Forks
4k
Avg merge
2d 15h
Merged PRs (30d)
145

Description

**CI run:** https://app.circleci.com/pipelines/github/ethereum-optimism/optimism/130975/workflows/c13e58bf-7b81-4e91-ad3c-5d711e9fabbd/jobs/5371028 (job `memory-all-opn-op-reth` #5371028, pipeline 130975, `develop` @ `820ee9ac`)

Sole failing test (1 of 769 results): `TestSupernodeEngineControllerConnectsAfterELUnavailableAtStartup` (`op-acceptance-tests/tests/supernode/interop/startup_resync/startup_resync_test.go:136`).

## Failure

```
startup_resync_test.go:136: sys.L2ELA.WaitForOnline()
Condition never satisfied
Expected to be online
```

`WaitForOnline` (`op-devstack/dsl/el.go:84-90`) polls `InfoByLabel(unsafe)` every 500ms with a 10s budget. Chain 901's op-reth was healthy the whole time (2s block cadence, ~3ms inserts) — the client's requests were being rejected, not the EL being down.

Log evidence (12:29:20–12:29:33, ~13s window):

- 112× `Initializing rollup node` and 223× `failed to init L2: incorrect L2 RPC chain id 901, expected 902` at ~1ms spacing — the supernode's chain-902 virtual node hot-looping restarts because the endpoint it dialed was answering as chain 901's EL
- 20× `websocket: bad handshake (HTTP status 429 Too Many Requests)` on l2-batcher block loads — op-reth 901's WS acceptance exhausted by the dial flood
- No stall/OOM/panic

Verdict from triage: flake, new signature. The kona sibling job passed on the same commit; the prior develop pipeline was green; the test file and `el.go` are byte-identical between the failing and prior-green commits. Commit `820ee9ac` is the merge of #21975 (additive safedb change, unrelated).

## Root cause — two defects

### 1. Devstack EL stop/start endpoint cross-wiring (primary)

In sysgo, each op-reth is fronted by two persistent `tcpproxy.Proxy` instances (auth + user) whose address is what every consumer (supernode `L2EngineAddr` via `multichain_supernode_runtime.go:611`, batcher, test clients) dials. op-reth itself binds OS-assigned ports, discovered from its startup log and installed via `SetUpstream` at the end of `OpReth.Start()` (`op-devstack/sysgo/l2_el_opreth.go:211-212`).

`OpReth.Stop()` (`l2_el_opreth.go:217-227`) kills the subprocess but leaves both proxies running with `upstreamAddr` still pointing at the dead process's now-freed ports (`tcpproxy` has no unset; `SetUpstream` is only called from `Start`). The test stops both ELs, then restarts them sequentially. When the OS ephemeral allocator hands chain 901's new op-reth a port previously held by chain 902's old op-reth, chain 902's proxy silently pipes every connection to chain 901's EL until chain 902's `Start()` finally reaches `SetUpstream` (which happens only after awaiting op-reth's "RPC WS/auth server started" log lines — a multi-second window). The chain-902 virtual node's L2 init then sees chain id 901 and fails the check at `op-node/rollup/types.go:286`.

This is the F15 port-TOCTOU family from `docs/ai/flake-prevention.md` in a new form: not probe-then-bind, but a stale proxy upstream re-pointed at whoever reuses the freed port.

`OpGeth.Stop()`/`StopControlled` (`op-devstack/sysgo/l2_el_opgeth.go:134-170`) and `OpReth.StopControlled` have the identical pattern.

### 2. Supernode chain-container restart loop has no backoff

`simpleChainContainer.Start` (`op-supernode/supernode/chain_container/chain_container.go:371-450`): when `vn.Start(ctx)` returns an error (here: immediately, at L2-init), the loop logs, stops the VN, and re-iterates with **no delay** — the only sleep in the loop is the 1s paused branch. A fast-failing init therefore retries at ~1000 dials/sec, converting the mis-wiring into connection exhaustion on the EL it is mistakenly reaching (the 429s that starved the batcher and the test's `WaitForOnline` client on the *healthy* chain). This is a production defect, not a test defect: any transiently mis-configured or unreachable L2 endpoint makes a real supernode DoS its own EL.

## Why flaky, not consistent

Cross-wiring requires the OS to reassign one chain's freed EL port to the other chain's restarting EL within the stop/start cycle, and the damage window is bounded by how long the second EL takes to start and re-point its proxy. Usually the port is not reused and the test passes.

## Occurrence log

| Date | Job | Notes |
|------|-----|-------|
| 2026-07-24 | [memory-all-opn-op-reth #5371028](https://app.circleci.com/pipelines/github/ethereum-optimism/optimism/130975/workflows/c13e58bf-7b81-4e91-ad3c-5d711e9fabbd/jobs/5371028) | First recorded occurrence (new signature). Commit was #21975 merge — unrelated. |

Note: this job has no in-job retries, so a single occurrence fails the pipeline.

## Fix direction

Per team policy: do NOT widen the test's 10s `WaitForOnline` budget or add test-side retries — that papers over the defect. Fix targets:

1. Clear the proxy upstream on EL stop in sysgo so a stopped EL's endpoints refuse connections instead of piping to whichever process reuses the freed port (makes cross-wiring structurally impossible: the upstream address is only ever set while the owning process is alive).
2. Add backoff to the chain-container VN restart loop so fast-failing init cannot hot-spin.

---
_Generated by [Claude Code](https://claude.ai/code)_

Contributor guide

Open the contributing guide

Research direction

Start with op-devstack/sysgo/l2_el_opreth.go and the corresponding opgeth stop paths, then inspect op-supernode/supernode/chain_container/chain_container.go and the named acceptance test at op-acceptance-tests/tests/supernode/interop/startup_resync/startup_resync_test.go. Reproduce the restart scenario and verify that stopped EL proxies cannot retain stale upstreams, fast-failing virtual nodes back off, and TestSupernodeEngineControllerConnectsAfterELUnavailableAtStartup passes without test-side retries.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems, infrastructure, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.