ethereum-optimism / ethereum-optimism/optimism
bug: disruption to L1 RPC caused a sequencer outage
- Dominant language
- Go
- Stars
- 6.5k
- Forks
- 4k
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 145
Description
**Bug Description**
I've received a report from a chain operator that their network had downtime because a disruption of service to their L1 RPC.
**Steps to Reproduce**
1. Run a sequencer node
2. Bring down the L1 RPC provider
3. Receive L1 RPC error and sequencer gets stuck
**Expected behavior**
Rather than recovering like we would expect, the active sequencer does not produce any more blocks until manual intervention.
**Environment Information:**
- op-node was running off of commit `224c5fd65cb4e8204258291e74263c1e70466176`
- There have been more recent reports of this behavior with the latest OP Stack releases
**Configurations:**
**Logs:**
we see an L1 RPC error and then op-node logs
```
Engine failed temporarily, backing off sequencer
temp: failed to fetch receipts of L1 block 0x8d38ff4b38edfdab2b92a0f224fac9ff801a9a8dcdffa0d2e3866ef11ace1a0c:6972342 (parent: 0x3a57dc8565c0bc66a7b32e86f99845f094ba876baa3366c4cacc0780842e31a7:6972341) for L1 sysCfg update: invalid character '{' after array element
```
**Additional context**
- We also see `resubscribing after failed L1 subscription` with `err unexpected EOF` near the end of the active sequencer producing blocks. The node found new L1 blocks (confirmed via advancing bq origin & new finalized / safe L1 blocks), but the sequencer block building stayed offline for minutes.
---
⚠️ **Notice:** Issues that do not include the following sections will be subject to closure:
- Bug Description
- Steps to Reproduce
- Environment Information
Please ensure all required sections are filled out accurately to expedite the debugging process and improve issue resolution efficiency.
Contributor guide
Research direction
Start with the op-node sequencer and L1 RPC behavior described in the reproduction steps, then inspect the logs around the failed receipt fetch and failed L1 subscription. Reproduce the provider outage and observe recovery after L1 connectivity returns. Done means the active sequencer resumes producing blocks without manual intervention.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100