ethereum-optimism / ethereum-optimism/optimism

op-interop-filter: verdict RPC can block behind ingester write lock, turning slow ingestion into fail-closed timeouts

Open
#21,943 1 comment 0 reactions 0 assignees View on GitHub
C-bug
Dominant language
Go
Stars
6.5k
Forks
4k
Avg merge
2d 15h
Merged PRs (30d)
145

Description

_Found via [Slack thread](https://oplabs-pbc.slack.com/archives/C03N11M0BBN/p1784651517475319) CI-flake investigation._

### Problem

The verdict path (`CheckAccessList` → `ValidateAccessEntry`) reads ingester state via `Contains` and `LatestTimestamp`, which take `c.mu.RLock` (`op-interop-filter/filter/logsdb_chain_ingester.go:243-253`, `:290-309`). Meanwhile `processBlockLogs` holds `c.mu.Lock` across the entire logsDB block write — every `AddLog` plus the `SealBlock` (`logsdb_chain_ingester.go:824-869`, lock taken at `:827`).

So when ingestion stalls mid-write (slow disk, starved goroutine), verdict RPCs don't return a fast soft "not yet available" answer — they block on the lock. op-reth's verdict query is a one-shot with a hardcoded 2s timeout (`rust/op-reth/crates/txpool/src/interop_filter/client.rs:44` `DEFAULT_REQUEST_TIMEOUT`, no retry). A blocked verdict times out, counts as a non-response, and with `endpoints=1` / `min_responses=1` (the devstack/in-process setup) quorum can't be reached, so the pool fails closed and rejects all interop txs until the filter answers again.

### Observed

CI job [5353712](https://circleci.com/gh/ethereum-optimism/optimism/5353712) (`memory-all-opn-op-reth`, `develop`, 2026-07-21): during a ~9.6s machine-wide scheduler stall, the single in-process filter endpoint gave no answer for 2s+, both chains' op-reth logged `inbox entry validation timed out, timeout: 2 secs`, and chain 902's pool logged `interop failing closed: too few endpoints returned a definitive verdict to reach quorum` (received=0 required=1 endpoints=1). Full narrative in #21940.

The design intent is that ingestion lag degrades to a soft "data not yet available" verdict (which op-reth treats as a non-response without failing closed on its own) — not to silence.

### Proposed fix

Don't answer verdicts under the ingestion write lock:

- narrow `processBlockLogs`'s critical section (e.g. stage logs outside the lock, hold it only for the actual DB mutation), and/or
- serve the read path from a snapshot / atomically-published latest-seal, so a slow ingest yields a fast `ErrFuture`-style soft response instead of a blocked RPC.

Cross-ref: #21940 (flake where this surfaced).

Contributor guide

Open the contributing guide

Research direction

Start with processBlockLogs in op-interop-filter/filter/logsdb_chain_ingester.go and trace CheckAccessList through ValidateAccessEntry, Contains, and LatestTimestamp to map the lock span. Read rust/op-reth/crates/txpool/src/interop_filter/client.rs for the 2-second timeout. Done means stalled ingestion no longer blocks verdict RPCs, allowing a fast soft not-yet-available response instead of a timeout and fail-closed quorum result.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, rust
Domain
api, backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.