ethereum-optimism / ethereum-optimism/optimism

Fatal panic in StateRootTask when upgrading op-reth from 1.x to 2.x on OP Stack chains

Open
#20,462 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
6.5k
Forks
4k
Avg merge
2d 15h
Merged PRs (30d)
145

Description

Bug Description
Upgrading op-reth from 1.11.5 to any 2.x release (tested 2.0.0 and 2.1.0-rc1) causes a fatal panic in the parallel trie proof workers (StateRootTask) when processing the first block after the persisted chain head. This affects all OP Stack chains tested (Zora mainnet, Unichain mainnet).
The node starts successfully and loads the existing database, but crashes immediately when it attempts to execute the first new block. The same panic also occurs on fresh 2.x nodes when they first attempt block execution against old state.
Notably, upgrading from 2.0.0 → 2.1.0-rc1 on a node already running 2.0.0 works without issue — the problem is specific to the 1.x → 2.x upgrade path.

Steps to Reproduce
Existing node upgrade path:

Run op-reth 1.11.5 on any OP Stack chain until fully synced
Stop the node
Upgrade binary to op-reth 2.0.0 or 2.1.0-rc1, keeping the same datadir
Start the node
Node loads existing DB, starts consensus engine, then fatally panics on the first block it attempts to execute

Fresh node (also reproduces):

Start a fresh op-reth 2.x node on an OP Stack chain from block 0
Node syncs and begins backfilling blocks
When the first block is executed against existing/cold state, the same panic occurs

Expected behavior
The node should start successfully and continue processing blocks from the persisted head without panicking. Upgrading from 1.x to 2.x should either work transparently or provide a clear migration step if a database migration is required.

Environment Information

Operating System: Linux (Kubernetes, Ubuntu-based container)
op-reth versions tested: 1.11.5 (working), 2.0.0 (broken), 2.1.0-rc1 (broken)
op-node version: v1.16.12
Chains affected: Zora mainnet, Unichain mainnet

Configurations
op-reth command:
op-reth node --chain zora --datadir /data/op-reth --storage.v2 --log.stdout.format json --authrpc.jwtsecret /shared/jwt.hex --http --http.addr 0.0.0.0 --http.api eth,net,web3,debug,txpool --ws --ws.addr 0.0.0.0 --ws.api eth,net,web3,debug,txpool --rollup.sequencer-http https://rpc.zora.energy --rollup.disable-tx-pool-gossip --rollup.discovery.v4 --full --port 30304
op-node command:
op-node --l2=http://localhost:8551 --l2.jwt-secret=/shared/jwt.hex --log.format=json
Relevant environment variables:
OP_NODE_SYNCMODE=consensus-layer
OP_NODE_L2_ENGINE_KIND=reth
RUST_LOG=debug

Logs
Persisted head at startup: block 45433669
Crash block: 45433670 (hash 0xa85c318bfd43d7b026452ae682afd8423d1bf61fee2b9e10de36da39ddba2284)
Parent hash: 0xa69a4eeaf4d5eae3972dc41905221ee81f60ac50be9b71ee4b1c5969b9f125d9
Log sequence leading to crash:
json{"message":"found canonical state for block in database","hash":"0xa69a4eeaf4d5eae3972dc41905221ee81f60ac50be9b71ee4b1c5969b9f125d9","number":"45433669"}
{"message":"Decided which state root algorithm to run","strategy":"StateRootTask"}
{"message":"Parent found on disk, no lazy overlay needed"}
{"message":"Spawning proof worker pools","storage_worker_count":8,"account_worker_count":8,"halve_workers":true}
{"message":"Executing block"}
{"message":"Executed block","elapsed":"32.650983ms"}
Panic output:
thread 'proof-acct-04' panicked at crates/trie/common/src/nibbles.rs:208:28:
index out of bounds: the len is 5 but the index is 32

thread 'proof-strg-05' panicked at alloy-trie-0.9.5/src/nodes/branch.rs:306:9:
assertion `left == right` failed
left: 0
right: 17

thread 'engine' panicked at alloy-trie-0.9.5/src/nodes/branch.rs:306:9:
assertion `left == right` failed
left: 0
right: 17

{"level":"ERROR","message":"Fatal error in consensus engine"}
{"level":"ERROR","message":"shutting down due to error","err":"Fatal error in consensus engine"}
Error: Fatal error in consensus engine
crates/node/builder/src/launch/engine.rs:332:43

Additional context

The panic is fully reproducible — happens every time on the same block across both Zora and Unichain
Block execution itself completes successfully (Executed block log appears before the panics) — the crash is in the parallel proof workers computing the state root, not in execution itself
The StateRootTask strategy is selected because the parent is found on disk rather than in memory, suggesting the issue is in how parallel proof workers read trie data written by 1.x
--storage.v2 is in use; unclear if this is a contributing factor
No database migration step was found in the 2.0.0 release notes that would apply here

Contributor guide

Open the contributing guide

Research direction

Reproduce the upgrade and fresh-node failures, then inspect StateRootTask and the panic sites in crates/trie/common/src/nibbles.rs and alloy-trie-0.9.5/src/nodes/branch.rs. Trace how proof workers read trie data when the parent is found on disk, including the engine path at crates/node/builder/src/launch/engine.rs. Done means OP Stack nodes can execute the affected block without a proof-worker or consensus-engine panic, with any required migration documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, rust
Domain
blockchain, databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.