Bug: storage v2 — valid block marked consensus-invalid: L1-attributes deposit reverts due to transient bad storage read during persistence commit (halts sync until restart)

# Bug: storage v2 — valid block marked consensus-invalid: L1-attributes deposit reverts due to transient bad storage read during persistence commit (halts sync until restart)

## Summary

On an OP-stack replica running with `--storage.v2`, the engine tree intermittently rejects a **valid canonical block** with `receipt root mismatch`. The block is inserted into the in-memory `InvalidHeaderCache`, every descendant is then rejected with `links to previously rejected block` / `Bad block with existing invalid ancestor`, and **sync halts until the node is restarted** (restart clears the in-memory cache; the node then validates the same block fine).

By brute-force decoding the faulty receipt roots from the logs (details below), we can prove what the node actually computed in two independent incidents 8 days apart: **the per-block L1-attributes deposit transaction REVERTED** (status=0) **with the correct depositNonce**, consuming exactly the same gas-shape both times. Gas-trace analysis against the canonical execution pins the failure to a single instruction: **the EIP-1967 implementation slot of the L1Block predeploy proxy transiently read as zero** (→ `Proxy: implementation not initialized` revert) — an ancient, written-once, cold slot served by the storage-v2 historical read path while the persistence task was mid-commit.

This did not occur on storage v1 (same chain, op-reth 1.11.x).

## Environment

- op-reth v2.3.1 equivalent: reth pin `81c026181e96ef33a823f3ef4d2a28940e9fa4fe` (May 19) + hotfix `7680d6d8a931c0af4f4eed26e971596970238b54` (`version="2.2.0-dev (7680d6d)"`)
- first observed on op-reth v2.2.3; still reproduces on the v2.3.1 hotfix build
- Custom OP-stack chain: Bedrock @0, Canyon+Ecotone @1712890200; 2s blocks; quiet chain — most blocks contain exactly 1 deposit tx (the L1-attributes tx)
- Replica (no sequencing, no flashblocks); payloads via op-node engine API
- `StorageSettings { storage_v2: true }`
- **Reproduces with `--engine.disable-state-cache`** (second incident below was captured with the cross-block execution cache disabled)

## Incident logs

### Incident 1 (block 39862528, right after node restart during catch-up burst)

```
12:32:40.132 DEBUG Received new payload ... number=39862528 hash=0xf8b6befb...
12:32:40.143 WARN Invalid block error on new payload invalid_hash=0xf8b6befb... invalid_number=39862528
validation_err=receipt root mismatch:
got 0x2bc1c091221beff541aac69f301ef57d55726cfd8464e19bdb6e4090615b0e8c,
expected 0xe030480a8278ed9e411b4289301689f61780233010396571e31e226bd59a0869
12:32:40.149 INFO Persistence threshold reached: sending to persistence service count=5 start_block=39862513 end_block=39862517
...
12:32:44.161 WARN Encountered invalid block number=39862528 ... error=links to previously rejected block
```

Context: the node had just restarted and validated 14 blocks (39862514–39862527) in ~150 ms while the first persistence batches were being written.

### Incident 2 (block 39871049, steady state, state cache disabled, debug logging)

```
17:16:43.000 newPayload 39871048 → validated OK (elapsed ~3ms), provider anchor historical=0x622d0eba...
17:16:43.018 newPayload 39871049 (parent 39871048, in-memory) → provider anchor historical=0x622d0eba...
17:16:43.018-035 Executed block elapsed=16.597093ms <-- ~50x slower than neighbors (0.3ms)
17:16:43.132 WARN Invalid block error on new payload invalid_number=39871049
validation_err=receipt root mismatch:
got 0x186d2fe62e6430138f33d78f184accce29a30d67a29bcf171ba4b96b8bfe3990,
expected 0x5ff7f6f5535290d571be000b947753e44325edf63f416261c8bf11cf9745eb52
17:16:43.136 DEBUG Finished persisting, calling finish last_persisted_block_number=39871044 elapsed=188.939676ms
^^^^^^ a 189ms persistence commit (blocks 39871041-44, BELOW the anchor in use) was in
flight for the entire validation of 39871049 and completed 4ms after the failure
```

In **both** incidents the failure landed inside an active persistence batch.

## Decoding the faulty receipts

These blocks contain exactly one deposit tx with no logs, so post-Canyon the receipt root is a pure function of `(status, cumulativeGasUsed, depositNonce)`. We brute-forced these against the roots in the logs using `op-alloy-consensus` encoding (`OpReceiptEnvelope::Deposit` + `ordered_trie_root_with_encoder`); the model is calibrated byte-exact — it reproduces the canonical receiptsRoot of the parent block 39871048 (`0x0c601a0c...`) from on-chain values.

| block | canonical receipt (`expected`) | faulty node receipt (`got`) |
|---|---|---|
| 39862528 | success, gas=43815, depositNonce=39862528 | **status=FAILED**, gas=**24943**, depositNonce=39862528 (**correct**) |
| 39871049 | success, gas=43827, depositNonce=39871049 | **status=FAILED**, gas=**24955**, depositNonce=39871049 (**correct**) |

Observations:

1. **The deposit nonce is correct in the faulty receipts.** The sender account (`0xDeaD...0001`, changes every block, served by the in-memory overlay) was read correctly. This rules out stale hot state.
2. **The success-vs-failure gas delta is exactly 18,872 in both incidents.** The revert is deterministic.
3. **Confirmed by tracing the canonical tx** (`debug_traceTransaction` of `0xc805f967...` on op-geth, filtered to SLOADs): the entire successful execution performs exactly **one** SLOAD — the **EIP-1967 implementation slot `0x360894a13ba1a3210667c828492db98dca3e2076cc3735a920a3ca505d382bbc` of the L1Block proxy**, cold (2,100 gas), at pc 340, with 977,302/1,000,000 gas remaining. Gas consumed including that SLOAD = 1,000,000 − 977,302 + 2,100 = **24,798**. The faulty receipts report **24,955 / 24,943** — the faulty execution died ~150 gas after this exact SLOAD, which is precisely the proxy's `require(implementation != address(0), "Proxy: implementation not initialized")` check followed by `REVERT`. In the canonical run the next operation after that check is the delegatecall into the implementation (continuing to 43,8xx total).

**Conclusion: the node transiently read the L1Block proxy's implementation slot as zero.** That slot was written ~once (genesis/upgrade) and never appears in the in-memory overlay — it is exactly the kind of read that falls through `MemoryOverlayStateProvider` to the **storage-v2 historical/hashed-state path**, which was concurrently being written by the 189 ms persistence commit.

So the failure chain is:

```
persistence task mid-commit (blocks below the provider's historical anchor)
→ cold-slot read through the v2 historical path transiently returns empty/zero
→ L1Block proxy call reverts → deposit receipt flips to status=0
→ receipt root mismatch → ConsensusError::BodyReceiptRootDiff
→ block inserted into in-memory InvalidHeaderCache as permanently invalid
→ all descendants rejected ("links to previously rejected block") → sync halted
→ restart clears the cache → same block validates fine
```

## What has been ruled out

- **Cross-block execution cache / prewarming**: incident 2 captured with `--engine.disable-state-cache`; prewarming is already skipped for <5-tx blocks (`SMALL_BLOCK_TX_THRESHOLD`).
- **Stale sender state / depositNonce**: decoded receipts show the correct nonce.
- **The v2.3.1 deadlock fix (#24875)**: present in the running build; bug still reproduces.
- **#24384 / #24584 / #24506** (post-pin fixes): inspected — #24384 only affects the `InsertExecutedBlock` path (sequencer/locally-built; not used here), #24584 affects the BAL path (not active pre-Karst), #24506 affects the state-root fallback, not receipt computation.

## Prior art

- #23013 — receipt root mismatch on OP replica, storage-v2-only, same payload valid elsewhere (closed unresolved)
- #22598 — same invalid-ancestor poisoning signature on Ethereum mainnet (closed by stale bot)
- ethereum-optimism/optimism#19930 — related report

## Root cause analysis (code trace at the pin)

Two conditions stack:

**1. The engine only consults the history index inside the persistence-commit window.**
`StateProviderBuilder` carries the historical anchor hash from `TreeState::blocks_by_hash` — the old persisted tip (39871040 here), which stays stale until the engine processes the persistence result. `build()` → `try_into_history_at_block` (`crates/storage/provider/src/providers/database/provider.rs:918-967`):

- if `anchor == best_block` (steady state) → `LatestStateProvider` — plain pinned-tx `HashedStorages` reads, no history index involved;
- if the persistence batch's MDBX commit landed **before** the validation's RO tx opened, `best (44) > anchor (40)` → **`HistoricalStateProviderRef` at `anchor+1`**, which trusts the RocksDB storage-history index.

So in steady state validation never touches the history index; only when a commit lands mid-validation does it flip provider type. (The MDBX tx itself is pinned at build time — MVCC is fine; the issue is which provider type the race selects.) This also explains the measured 16.6 ms vs 0.3 ms execution on exactly the failing blocks.

**2. The historical lookup returns zero without reading plain state when the index has no entry.**
`HistoricalStateProviderRef::storage_by_lookup_key` (`crates/storage/provider/src/providers/state/historical.rs:239-281`):

```rust
HistoryInfo::NotYetWritten => Ok(None), // line 248 — zero, plain state never consulted
```

For the L1Block implementation slot (written ~once at genesis/upgrade), the RocksDB `StoragesHistory` lookup (`providers/rocksdb/provider.rs:1588-1667`) reaches `NotYetWritten` iff there is **no shard at all for `(0x42…0015, impl_slot)`** and no `PruneSegment::StorageHistory` checkpoint (which would soften the result to `MaybeInPlainState` → plain-state read → correct). The #23335 `visible_tip` capping and the per-(address,slot) shard seek were both walked branch-by-branch and are sound *given a complete index* — sibling-slot writes every block cannot collide with the ancient slot's key.

So the failure requires a **missing genesis/upgrade-era index entry** for that slot in RocksDB (e.g. from v1→v2 migration/init not backfilling it, or the startup healer's `checkpoint == 0` path in `rocksdb/invariants.rs:298-321`, which clears the `StoragesHistory` CF and re-inserts only the ChainSpec genesis alloc). Outside the persistence window the gap is invisible — the engine takes the Latest fast path; inside the window, the lookup concludes "slot not yet written at block 39871041" → returns zero.

**Confirmed on the affected node** — any *historical* read of that slot goes through the same lookup, no race required:

```
$ cast storage 0x4200000000000000000000000000000000000015 \
0x360894a13ba1a3210667c828492db98dca3e2076cc3735a920a3ca505d382bbc -r # latest
0x00000000000000000000000007dbe8500fc591d1852b76fee44d5a05e13097ff

$ cast storage 0x4200000000000000000000000000000000000015 \
0x360894a13ba1a3210667c828492db98dca3e2076cc3735a920a3ca505d382bbc \
--block -r # historical
0x0000000000000000000000000000000000000000000000000000000000000000
```

The index gap is real and permanent. Note the broader severity: **all historical RPC reads (`eth_getStorageAt`, `eth_call`, tracing) of any key untouched since the node's storage-v2 initialization silently return zero/empty** — the engine marking valid blocks invalid is just the moment this defect becomes consensus-visible (only during the persistence window does validation flip onto the history-indexed provider).

## Suggested fixes

1. **Don't flip to the history-indexed provider when the in-memory overlay already covers the gap.** When `anchor < best` but the blocks `(anchor, best]` are all present in the validation's `MemoryOverlayStateProvider` overlay (true by construction for canonical-head extensions), use `latest()` as the historical base: any slot whose value changed between anchor and best is shadowed by an overlay bundle before reaching the DB. Removes the engine's dependency on history-index completeness entirely and the 50× read-latency cliff in the window. Needs a reorg guard (`block_hash(anchor_number) == anchor_hash` inside the pinned tx).
2. **Engine-built historical providers should degrade "no index entry" to `MaybeInPlainState`, not `NotYetWritten`** (e.g. a builder flag setting `lowest_available_*_history_block_number = Some(block_number)` in `try_into_history_at_block`). Plain state at `best` equals state at `anchor` for any slot not in the overlay, so this is provably correct for the engine path; pair it with a warn/counter when the index claims "never written" but plain state is non-empty — that's also the cheap detector for affected nodes. Must not apply to RPC historical providers.
3. **Mitigation:** on post-execution root mismatch for a canonical-head extension validated on a historical provider with `anchor < best` (the race signature), retry validation once with a freshly built provider before inserting into `InvalidHeaderCache`; and/or classify such mismatches via `Consensus::is_transient_error`. Plus data repair: a completeness check in `RocksDBProvider::check_consistency`, and making the `checkpoint == 0` heal path refuse to clear `StoragesHistory` for chains initialized from a state dump.

Happy to provide full debug logs, the receipt brute-force tool, or run instrumented builds — this reproduces every few hours/days on our infra.

Start with StateProviderBuilder and try_into_history_at_block in crates/storage/provider/src/providers/database/provider.rs, then trace HistoricalStateProviderRef::storage_by_lookup_key in crates/storage/provider/src/providers/state/historical.rs. Reproduce the historical storage read with cast storage while persistence is active, and inspect the RocksDB history lookup in providers/rocksdb/provider.rs plus the healer path in rocksdb/invariants.rs. Done means historical reads return the stored value and persistence cannot cause a valid block to be rejected.

Written by the indexing model from the issue text.