ethereum-optimism / ethereum-optimism/optimism
op-interop-filter: verdict RPC can block behind ingester write lock, turning slow ingestion into fail-closed timeouts
- Dominant language
- Go
- Stars
- 6.5k
- Forks
- 4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 145
Description
_Found via [Slack thread](https://oplabs-pbc.slack.com/archives/C03N11M0BBN/p1784651517475319) CI-flake investigation._
### Problem
The verdict path (`CheckAccessList` → `ValidateAccessEntry`) reads ingester state via `Contains` and `LatestTimestamp`, which take `c.mu.RLock` (`op-interop-filter/filter/logsdb_chain_ingester.go:243-253`, `:290-309`). Meanwhile `processBlockLogs` holds `c.mu.Lock` across the entire logsDB block write — every `AddLog` plus the `SealBlock` (`logsdb_chain_ingester.go:824-869`, lock taken at `:827`).
So when ingestion stalls mid-write (slow disk, starved goroutine), verdict RPCs don't return a fast soft "not yet available" answer — they block on the lock. op-reth's verdict query is a one-shot with a hardcoded 2s timeout (`rust/op-reth/crates/txpool/src/interop_filter/client.rs:44` `DEFAULT_REQUEST_TIMEOUT`, no retry). A blocked verdict times out, counts as a non-response, and with `endpoints=1` / `min_responses=1` (the devstack/in-process setup) quorum can't be reached, so the pool fails closed and rejects all interop txs until the filter answers again.
### Observed
CI job [5353712](https://circleci.com/gh/ethereum-optimism/optimism/5353712) (`memory-all-opn-op-reth`, `develop`, 2026-07-21): during a ~9.6s machine-wide scheduler stall, the single in-process filter endpoint gave no answer for 2s+, both chains' op-reth logged `inbox entry validation timed out, timeout: 2 secs`, and chain 902's pool logged `interop failing closed: too few endpoints returned a definitive verdict to reach quorum` (received=0 required=1 endpoints=1). Full narrative in #21940.
The design intent is that ingestion lag degrades to a soft "data not yet available" verdict (which op-reth treats as a non-response without failing closed on its own) — not to silence.
### Proposed fix
Don't answer verdicts under the ingestion write lock:
- narrow `processBlockLogs`'s critical section (e.g. stage logs outside the lock, hold it only for the actual DB mutation), and/or
- serve the read path from a snapshot / atomically-published latest-seal, so a slow ingest yields a fast `ErrFuture`-style soft response instead of a blocked RPC.
Cross-ref: #21940 (flake where this surfaced).
Contributor guide
Research direction
Start with processBlockLogs in op-interop-filter/filter/logsdb_chain_ingester.go and trace CheckAccessList through ValidateAccessEntry, Contains, and LatestTimestamp to map the lock span. Read rust/op-reth/crates/txpool/src/interop_filter/client.rs for the 2-second timeout. Done means stalled ingestion no longer blocks verdict RPCs, allowing a fast soft not-yet-available response instead of a timeout and fail-closed quorum result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, rust
- Domain
- api, backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100