ethereum-optimism / ethereum-optimism/optimism
flaky test: `TestInteropHappyTx`
- Dominant language
- Go
- Stars
- 6.5k
- Forks
- 4k
- Avg merge
- 2d 38m
- Merged PRs (30d)
- 164
Description
**CI run:** https://circleci.com/gh/ethereum-optimism/optimism/5353712 (job `memory-all-opn-op-reth`, pipeline 130628, `develop` @ b088905c — commit unrelated)
**Date:** 2026-07-21
**Error:** `operation failed permanently after 5 attempts: not found` / `exec msg receipt not found` at `op-devstack/dsl/eoa.go:336` (`EOA.SendExecMessage` receipt wait), called from `op-acceptance-tests/tests/interop/message/interop_happy_tx_test.go:41`.
Sibling occurrence (same failure family): `TestInitExecMsgWithDSL` (same package) failed on 2026-07-19 in the kona variant of the job with the same `operation failed permanently after 5 attempts: not found` at `op-devstack/dsl/contract/call.go:73`.
### Mechanism observed in the job log
The test uses `presets.NewTwoL2SupernodeInterop(t, 0, presets.WithInteropFilter())`.
- 16:16:59 — both chains' op-reth logged `failed to query failsafe state ... inbox entry validation timed out, timeout: 2 secs` (the in-process op-interop-filter's RPC did not answer within op-reth's 2s request timeout, `rust/op-reth/crates/txpool/src/interop_filter/client.rs` `DEFAULT_REQUEST_TIMEOUT`).
- 16:17:14 — chain 902's sequencer op-reth logged `interop failing closed: too few endpoints returned a definitive verdict to reach quorum; all interop transactions are rejected until enough endpoints respond` (received=0 required=1 endpoints=1). Timeouts/soft out-of-sync answers are non-responses that don't count toward quorum, so the txpool fails closed by design (`InteropTxValidatorError::QuorumNotReached`).
- The interop-filter ingester simultaneously logged `Chain head is behind ingestion progress, waiting for node to catch up chain=902 head=9 next_block=10` — chain 902's head was stalled, so no inclusion was happening either.
Bob's executing message was consequently not included while the transient fail-closed / stall window lasted. The DSL receipt wait (`op-devstack/dsl/eoa.go` `EOA.Plan()` → `txplan.WithRetryInclusion(elClient, 5, retry.Exponential())`) only polls 5 times ≈ 15s, which is smaller than the window the system is designed to recover from on its own (the filter self-heals as soon as its endpoint answers again; nothing is permanently wrong).
Why flaky, not consistent: the interop filter is an in-process Go service answering trivial lookups — its RPC normally responds in milliseconds. Only under CI CPU-contention spikes do responses exceed op-reth's 2s timeout, opening a temporary fail-closed window; the test only fails when such a window overlaps its ~15s receipt-wait budget.
Created by Claude (flake-fix session, requested via Slack).
Contributor guide
Research direction
Start with op-acceptance-tests/tests/interop/message/interop_happy_tx_test.go:41 and trace the receipt wait through op-devstack/dsl/eoa.go:336 and txplan.WithRetryInclusion. Then read rust/op-reth/crates/txpool/src/interop_filter/client.rs and the cited CI logs to understand the transient quorum failure. Done means the test accommodates the observed recovery window and no longer flakes when interop-filter responses temporarily time out.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, rust
- Domain
- distributed-systems, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100