ethereum-optimism / ethereum-optimism/optimism

proofs-history v2: unwinding a history shard without re-deriving its key makes `eth_getProof` serve chain-tip values for historical blocks

Open
#22,501 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
6.5k
Forks
4k
Avg merge
2d 15h
Merged PRs (30d)
145

Description

**Component:** `rust/op-reth/crates/trie` (`reth-optimism-trie`), v2 proofs storage — v2 only.
**Affected:** `3bccc60` (2026-06-04) through `f863ff4` (`op-reth/v2.4.2`); still on `develop` @ `9da7897a`.
**Severity:** silent wrong answers. No error, no log line; the proof window looks healthy.

## Summary

After any reorg — or any `op-reth proofs unwind` — a band of historical blocks starts answering
`eth_getProof` with values from the **chain tip**. The witness is well-formed, so the only way to
notice is to hash it: the top account-proof node no longer hashes to the block's `stateRoot`.

Forward-fill does not repair it, and a deeper unwind *widens* it. On an affected production node
the band is `[~3.45M, 3,989,999]` — a downstream witness consumer rejected every proof in it and
stalled for ~170k blocks. The canonical chain is unaffected, and so are the leaf values: only the
trie nodes served in `accountProof` are wrong.

## Root cause

The v2 store answers a historical read as *current state with every later block's changesets
reverse-applied*, finding them via a sharded history bitmap keyed `(logical_key,
highest_block_number)`; a key's last shard uses the `u64::MAX` sentinel.

`find_source` (`db/store_v2/cursor/mod.rs`) seeks `ShardedKey(key, max_block_number + 1)` and
deliberately has no `next()` fallback, per its own doc comment:

> This lands on the first shard whose `highest_block_number > max_block_number`, guaranteeing the
> shard contains at least one entry after the target block and eliminating the need for a
> `cursor.next()` fallback.

That requires every non-sentinel shard key to equal the largest block its shard holds.
`prune_history_range_for_key` (`db/store_v2/write.rs`) breaks it — the partial-prune branch writes
survivors back under the **old** key:

```rust
} else if filtered.len() < original_len {
// Partial prune — update shard and advance.
let new_list = BlockNumberList::new_pre_sorted(filtered);
cursor.upsert(key, &new_list)?; // ← key no longer matches contents
entry = cursor.next()?;
}
```

| Caller | Trims | Shard maximum | Key still valid? |
|---|---|---|---|
| `prune_earliest_state` (retention) | low end | unchanged | yes |
| `unwind_history` (reorg / CLI unwind) | **high end** | **drops** | **no** |

The boundary shard is left keyed above everything it contains, so it shadows every later shard for
targets below that stale key: `select` finds nothing past the target and the read resolves
`FromCurrentState` — the tip's value, for a historical block. Forward-fill can't recover because
`append_history_indices_batched` only ever seeks the sentinel, so refilled blocks land in a fresh
sentinel that the stale shard hides.

Reth's own `unwind_history_shards` (`providers/database/provider.rs`) gets this right: it deletes
the boundary shard and reinserts survivors under `ShardedKey::last(..)`.

Both live paths reach it — `ChainReorged`/`ChainReverted` → `EngineState::unwind` →
`PersistenceService::try_unwind` → `unwind_history`, and `proofs unwind --target N` →
`unwind_history(block.block_with_parent())`, the identical call.

Shards hold ≤`NUM_OF_INDICES_IN_SHARD` (2000) entries and a key gets one per block, so a key's
broken interval is `[last surviving entry, stale key − 1]`: narrow for a hot key, huge for a cold
key touched 2000 times over a long span. `eth_getProof` needs every node on a path, so the damage
is the union of those intervals.

## Reproduction — unit test

Drop this into `crates/trie/src/db/store_v2/tests.rs`. It fails at the shard assertion on an
unmodified tree:

```
assertion `left == right` failed: boundary shard must be re-keyed
left: [(2000, 1800)] # key says 2000, contents end at 1800
right: [(1800, 1800)]
```

Comment that assertion out and it fails on the reads instead: 1800/1850/1900/1999 return nonce
2100 — the tip's — while 1500, 1799, 2000 and 2100 are correct.

```rust
#[test]
fn unwind_rekeys_boundary_history_shard() {
use reth_trie::hashed_cursor::HashedCursor;

const ADDR: B256 = B256::repeat_byte(0xA1);
const TIP: u64 = 2_100;
const UNWIND_TO: u64 = 1_801; // strictly inside the first, non-sentinel shard

fn bhash(n: u64) -> B256 {
let mut b = [0u8; 32];
b[24..].copy_from_slice(&n.to_be_bytes());
B256::new(b)
}
// Block `n` touches only ADDR, setting nonce = n, so a read at `n` must see nonce `n`.
fn diff(n: u64) -> BlockStateDiff {
let mut ps = HashedPostState::default();
ps.accounts.insert(ADDR, Some(Account { nonce: n, ..Default::default() }));
BlockStateDiff {
sorted_trie_updates: TrieUpdates::default().into_sorted(),
sorted_post_state: ps.into_sorted(),
}
}

let db = setup_db();
{
let p = MdbxProofsProviderV2::new(db.tx_mut().unwrap());
p.store_hashed_accounts(vec![(ADDR, Some(Account::default()))]).unwrap();
p.set_initial_state_anchor(BlockNumHash::new(0, bhash(0))).unwrap();
p.commit_initial_state().unwrap();
OpProofsInitProvider::commit(p).unwrap();
}
let store = |range: std::ops::RangeInclusive| {
for n in range {
let p = MdbxProofsProviderV2::new(db.tx_mut().unwrap());
p.store_trie_updates(make_block_ref(n, bhash(n), bhash(n - 1)), diff(n)).unwrap();
OpProofsProviderRw::commit(p).unwrap();
}
};
// Nonce as of `at`, through the cursor that backs eth_getProof.
let nonce_at = |at: u64| {
let p = MdbxProofsProviderV2::new(db.tx().unwrap());
p.account_hashed_cursor(at).unwrap().seek(ADDR).unwrap().unwrap().1.nonce
};
// (shard key, largest block the shard holds) for ADDR, in key order.
let shards = || {
let tx = db.tx().unwrap();
let mut c = tx.cursor_read::().unwrap();
let mut out = vec![];
let mut e = c.seek(HashedAccountShardedKey::new(ADDR, 0)).unwrap();
while let Some((k, list)) = e {
if k.0.key != ADDR {
break;
}
out.push((k.0.highest_block_number, list.iter().next_back().unwrap()));
e = c.next().unwrap();
}
out
};

store(1..=TIP);
assert_eq!(shards(), vec![(2_000, 2_000), (u64::MAX, TIP)], "two shards before unwind");
for at in [1_500, 1_799, 1_800, 1_900, 1_999, 2_000, TIP] {
assert_eq!(nonce_at(at), at, "baseline read at {at}");
}

{
let p = MdbxProofsProviderV2::new(db.tx_mut().unwrap());
p.unwind_history(BlockWithParent::new(
bhash(UNWIND_TO - 1),
NumHash::new(UNWIND_TO, bhash(UNWIND_TO)),
))
.unwrap();
OpProofsProviderRw::commit(p).unwrap();
}
// The shard now holds 1..=1800, so its key must say 1800 — not 2000.
assert_eq!(shards(), vec![(UNWIND_TO - 1, UNWIND_TO - 1)], "boundary shard must be re-keyed");

store(UNWIND_TO..=TIP); // forward-fill the canonical chain, as the ExEx does
for at in [1_500, 1_799, 1_800, 1_850, 1_900, 1_999, 2_000, TIP] {
assert_eq!(nonce_at(at), at, "read at {at} after unwind + forward-fill");
}
}
```

## Reproduction — end-to-end over RPC

One `op-reth` node following a chain, with the ExEx on v2 storage and a retention window larger
than the run, so nothing is pruned:

```
--proofs-history --proofs-history.storage-version=v2 --proofs-history.window=1000000
```

Initialise the store at genesis so it accrues over the whole chain, and let the chain pass 2000
blocks (the account-trie root node changes every block, so its bitmap rechunks into a
non-sentinel shard at 2000 — that shard is what the unwind mis-keys):

```sh
op-reth init --chain genesis.json --datadir /data
op-reth proofs init --chain genesis.json --datadir /data \
--proofs-history.storage-version v2 --proofs-history.window 1000000 \
--proofs-history.skip-backfill
```

The probe is the check any witness consumer makes — hash the top account-proof node and compare it
to the header:

```sh
probe() { # $1 = block number
hex=$(printf '0x%x' "$1")
hdr=$(cast rpc eth_getBlockByNumber "$hex" false --rpc-url "$RPC" | jq -r .stateRoot)
node=$(cast rpc eth_getProof 0x4200000000000000000000000000000000000016 '[]' "$hex" \
--rpc-url "$RPC" | jq -r '.accountProof[0]')
root=$(cast keccak "$node")
[ "$root" = "$hdr" ] && echo "$1 MATCH" || echo "$1 MISMATCH hdr=$hdr proof=$root"
}
```

At head 2261 every probe matches. Then, with the node stopped, unwind and restart — the ExEx
forward-fills off the canonical chain with no error, because the unwind restores the
*current-state* tables correctly and every re-executed block passes its state-root check. Only the
index is wrong:

```sh
op-reth proofs unwind --chain genesis.json --datadir /data \
--proofs-history.storage-version v2 --target 1801
```

```
1797 MATCH 1800 MISMATCH 1997 MISMATCH 2000 MATCH
1798 MATCH 1801 MISMATCH 1998 MISMATCH 2001 MATCH
1799 MATCH 1802 MISMATCH 1999 MISMATCH 2002 MATCH
1950 MISMATCH
```

The band is exactly `[1800, 1999]` — surviving tail to stale key minus one, predicted from the code
before the run. It starts *below* the unwind target and extends above it, so it can't be located by
reasoning about reorg depth. Every failure returns the same root here, and that root is the
canonical `stateRoot` of block 2270, the store's `latest` at read time; the collapse to one root is
an artifact of a trie holding a single interesting account.

## Confirmation on a production node

The same defect on a real chain, band `[~3.45M, 3,989,999]`, upper edge landing exactly on a shard
key:

- Blocks 3.45M–3.9M are poisoned but were **never reorged**, and blocks 3.99M–4.02M — the range
that *was* reorged — read clean. The poisoned set and the reorged set are nearly disjoint, so the
store is not replaying orphaned data.
- Leaf values are canonical: a fee vault's balance from `eth_getProof` matches `eth_getBalance`
against reth's own history exactly, at every block including inside the reorg band. Only the
branch nodes are wrong, which is enough to break the witness.
- Repeating the identical request at a poisoned block returns a **different root every time**,
while a clean block is byte-stable:

```
eth_getProof at 3,988,275 (poisoned), same request, ~8s apart:
0x52c18723edd187da8f537c2e242e1c923231735c2d9edb9bff73602ea3dd4a8a
0xb7be05c9227e297049651c6ba45d9f42e3353d90704b67fdf6bf36df831d7944
0x2f1daba2b1dc6679522fd8d1acafe443de54e0f84d94ba7176917a62a5c09849

eth_getProof at 3,990,000 (clean), same request, ~8s apart:
0x43d3fd66c76f578fca1fab35d63742458251e2012f318b2d3bcc6a33a4738e2a
0x43d3fd66c76f578fca1fab35d63742458251e2012f318b2d3bcc6a33a4738e2a
```

A historical proof is immutable, so drift can only mean current state is being spliced in — it also
rules out the store merely holding stale bytes. And because the stale key can never be raised while
each further unwind lowers the shard's surviving tail, repeated `proofs unwind` attempts extend the
band *downward*: this one grew from ~3.9M to ~3.45M across recovery attempts.

## Note

`find_source`'s no-`next()` design makes an undocumented cross-module invariant load-bearing for
correctness, with nothing on the write side enforcing it. Both halves of the store are
individually reasonable; the bug lives in the contract between them.

Contributor guide

Open the contributing guide

Research direction

Start with prune_history_range_for_key in db/store_v2/write.rs and its interaction with find_source in db/store_v2/cursor/mod.rs. Add the reproduction to crates/trie/src/db/store_v2/tests.rs and run it, then verify the boundary shard and historical reads remain correct after unwind and forward-fill. Done means the test passes without stale-key assertions or incorrect historical values.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.