ethereum-optimism / ethereum-optimism/optimism

flaky test: `TestInteropHappyTx`

Open
#21,940 1 comment 0 reactions 0 assignees View on GitHub
C-flake
Dominant language
Go
Stars
6.5k
Forks
4k
Avg merge
2d 38m
Merged PRs (30d)
164

Description

**CI run:** https://circleci.com/gh/ethereum-optimism/optimism/5353712 (job `memory-all-opn-op-reth`, pipeline 130628, `develop` @ b088905c — commit unrelated)
**Date:** 2026-07-21
**Error:** `operation failed permanently after 5 attempts: not found` / `exec msg receipt not found` at `op-devstack/dsl/eoa.go:336` (`EOA.SendExecMessage` receipt wait), called from `op-acceptance-tests/tests/interop/message/interop_happy_tx_test.go:41`.

Sibling occurrence (same failure family): `TestInitExecMsgWithDSL` (same package) failed on 2026-07-19 in the kona variant of the job with the same `operation failed permanently after 5 attempts: not found` at `op-devstack/dsl/contract/call.go:73`.

### Mechanism observed in the job log

The test uses `presets.NewTwoL2SupernodeInterop(t, 0, presets.WithInteropFilter())`.

- 16:16:59 — both chains' op-reth logged `failed to query failsafe state ... inbox entry validation timed out, timeout: 2 secs` (the in-process op-interop-filter's RPC did not answer within op-reth's 2s request timeout, `rust/op-reth/crates/txpool/src/interop_filter/client.rs` `DEFAULT_REQUEST_TIMEOUT`).
- 16:17:14 — chain 902's sequencer op-reth logged `interop failing closed: too few endpoints returned a definitive verdict to reach quorum; all interop transactions are rejected until enough endpoints respond` (received=0 required=1 endpoints=1). Timeouts/soft out-of-sync answers are non-responses that don't count toward quorum, so the txpool fails closed by design (`InteropTxValidatorError::QuorumNotReached`).
- The interop-filter ingester simultaneously logged `Chain head is behind ingestion progress, waiting for node to catch up chain=902 head=9 next_block=10` — chain 902's head was stalled, so no inclusion was happening either.

Bob's executing message was consequently not included while the transient fail-closed / stall window lasted. The DSL receipt wait (`op-devstack/dsl/eoa.go` `EOA.Plan()` → `txplan.WithRetryInclusion(elClient, 5, retry.Exponential())`) only polls 5 times ≈ 15s, which is smaller than the window the system is designed to recover from on its own (the filter self-heals as soon as its endpoint answers again; nothing is permanently wrong).

Why flaky, not consistent: the interop filter is an in-process Go service answering trivial lookups — its RPC normally responds in milliseconds. Only under CI CPU-contention spikes do responses exceed op-reth's 2s timeout, opening a temporary fail-closed window; the test only fails when such a window overlaps its ~15s receipt-wait budget.

Created by Claude (flake-fix session, requested via Slack).

Contributor guide

Open the contributing guide

Research direction

Start with op-acceptance-tests/tests/interop/message/interop_happy_tx_test.go:41 and trace the receipt wait through op-devstack/dsl/eoa.go:336 and txplan.WithRetryInclusion. Then read rust/op-reth/crates/txpool/src/interop_filter/client.rs and the cited CI logs to understand the transient quorum failure. Done means the test accommodates the observed recovery window and no longer flakes when interop-filter responses temporarily time out.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, rust
Domain
distributed-systems, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.