[architecture] State layer: ordered scan, historical reads and state verification are split across two backends with incompatible guarantees

Nobody has claimed this yet.

Filed after tracing #4998 down to its root. That bug is one symptom; this issue is
about the structure that produced it, and about a second, larger gap found along
the way — the delta state digest does not commit to state, so an archive node's
answers cannot be verified by anything.

Everything under "Verified" was read out of the tree at `rc1_2.5.0`
(`e3bfc422`) and confirmed by running it. The Erigon 3 section is second-hand and
labelled as such.

---

## Verified findings

### 1. The erigon store contributes nothing to consensus

```go
// state/factory/erigonstore/workingsetstore_erigon.go
func (store *ErigonWorkingSetStore) Digest() hash.Hash256 {
return hash.ZeroHash256
}
```

And the `stateRoot` served over web3 is iotex's own digest, not erigon's:

```go
// api/web3server_marshal.go:247
StateRoot: "0x" + hex.EncodeToString(deltaStateDigest[:]),
```

Erigon is a mirror. It is not in the block header, not in the consensus digest,
and not the source of the `stateRoot` clients see.

### 2. What the contract layer actually buys is history — nothing else

State on the erigon path lives in EVM contract storage (`systemcontracts/GenericStorage.sol`),
committed through:

```go
tsw := erigonstate.NewPlainStateWriter(tx, tx, height)
store.backend.intraBlockState.CommitBlock(rules, tsw)
tsw.WriteChangeSets()
tsw.WriteHistory()
```

Erigon 2's changeset/history machinery is defined over **accounts and contract
storage slots**. Putting iotex state into contract storage is the only way it
gets covered by that machinery, which is what makes archive reads work at all.

That is the whole benefit. It is not for an EVM-compatible state root — finding 1
rules that out.

### 3. The price is that iotex key ordering does not exist on the erigon path

Ordering is destroyed twice:

- Solidity `mapping(bytes => uint256) keyIndex_` — the slot is `keccak(key . slot)`.
MDBX iterates in hash order, unrelated to iotex key order.
- The contract's own enumerable index is worse than unordered. `keys_` is
insertion-ordered, and `remove()` uses swap-and-pop, which actively scrambles
what is left:

```solidity
keys_[indexToRemove] = keys_[lastIndex];
values_[indexToRemove] = values_[lastIndex];
keys_.pop();
```

`keySplitContractStorage.List()` and `.Batch()` are both `not implemented`, so
even the unordered enumeration is unavailable for the namespace IIP-59 uses.

`ErigonWorkingSetStore.States` is therefore right to reject `RangeOption` — a
differently-ordered answer would be worse than an error. The defect is upstream
of that refusal.

### 4. Historical prefix enumeration is expensive in Erigon 2 by design

The history index is keyed by the item, not by position:

```
AccountChangeSet: bigEndian(N) + A -> X
AccountsHistory: A + shard_id -> RoaringBitmap(block numbers)
```

This answers "what was A at height X" in one seek. It cannot answer "which keys
existed at height X" without walking every key that ever existed. So even if
iotex state moved to erigon's native KV tables, historical *ordered scans* would
still not be cheap — only latest-state scans would.

### 5. `deltaStateDigest` is a delta, not a state commitment

```go
func (store *stateDBWorkingSetStore) Digest() hash.Hash256 {
return hash.Hash256b(store.flusher.SerializeQueue())
}
```

It hashes the block's **ordered write queue**. It proves "this block performed
these writes in this order". It does not commit to the resulting state.

Two consequences that are already visible in the tree:

- It is order-sensitive in a way that is easy to break by accident. The comment
in `GrantEpochReward` explaining why the balance update must stay ahead of the
sentinel exists precisely because reordering two unrelated writes changes the
digest of every epoch-boundary block, at every height, including history.
- `DumpWriteQueue` exists as a debugging tool to find the first differing line
between two nodes' write queues — which is what diagnosing a digest mismatch
amounts to today.

**Nothing in the system commits to state content.** An archive node's answer to a
historical query is therefore unverifiable: there is no root to check it against
and no proof it can produce.

### 6. Consensus is protected by wiring, not by design

`Mint`/`Validate` build a `workingSetStoreWithSecondary`, whose `States`
delegates to the statedb reader; only `newReadOnlyWorkingSet` swaps in an
erigon-only store. So the era freeze in `freezePendingPoolDrainWork` — which does
depend on an ordered range scan — never meets erigon today.

That is a property of the current wiring, not a guarantee. An erigon-primary
mode, or any change that moves read-only working sets onto the execution path,
turns a query-time error into a consensus split.

---

## Erigon 3 — second-hand, needs confirmation against source

From release notes and docs, not read from source (network restrictions here
blocked github/pkg.go.dev/proxy.golang.org):

- State is split into `domain` (latest), `history` (historical values), `idx`
(inverted indices, searchable/filterable), `accessor` (Get-only).
- History granularity moved from block to a global `txNum`.
- `RangeAsOf` reportedly provides ordered key-range iteration as of a past point,
by merging history files over the domain snapshot — i.e. the thing finding 4
says Erigon 2 lacks.
- A prototype exists for pluggable custom domains/history/indexers.
- A `commitment` domain computes a real MPT root; historical `eth_getProof`
left experimental in v3.4.

If the pluggable-domain part holds, iotex state could become its own erigon
domain with native key ordering and its own history — which would remove the
Solidity layer entirely.

**Note this does not help on its own**: with state still in contract storage,
`RangeAsOf` iterates erigon's key space, and iotex keys are still hashed inside
it. Upgrading to E3 without moving state out of contracts changes nothing for
#4998.

---

## Options

### A. Give statedb archive capability, retire erigon

statedb's KV is bolt/pebble keyed by the **raw iotex key**, already ordered. Add
a changeset + inverted index layer (the Erigon 2 design, which this codebase
already understands).

| | today | statedb + archive |
|---|---|---|
| historical point read | via erigon | yes |
| historical ordered scan | structurally impossible | **yes — keys are ordered** |
| digest verification | statedb path only | one path, everywhere |
| Solidity storage layer | maintained | **deleted** |
| #4998 | needs a workaround | does not exist |

Cost: implement history and pruning; lose EVM-side visibility of iotex state
(`eth_getStorageAt` on the system contracts). Finding 1 shows that visibility is
not a consensus dependency.

### B. Adopt a real state commitment

Put a content-addressed state root (merkle or verkle) in the header, replacing or
alongside the delta digest.

This is the only option that answers "is the state I just read correct". It makes
any backend verifiable, makes historical proofs possible, makes light clients
possible, and removes the write-order sensitivity described in finding 5.

Cost: consensus change, hard fork, root computation on the hot path.

### C. Keep erigon, adopt E3's commitment domain

Gets a real MPT root, but it commits to the **EVM view**. Using it as iotex's
state commitment means adopting an EVM state root as canonical — still a
consensus change, and it permanently binds iotex state to contract storage,
which is the opposite direction from A.

### D. Verify by re-execution

Replay from a checkpoint and compare. No code change, cost per verification is a
chain replay. Fine for audit, not for serving queries.

---

## Suggested sequencing

**Now** — #5000 unblocks the archive read. It is a workaround: it makes one read
stop needing a guarantee the backend cannot give. It does not remove the hazard.

**Next, and independent of everything else** — remove the protocol's dependency
on ordered prefix enumeration by materialising the IIP-59 pending-pool index as
an ordinary state entry, and make scan capability explicit in `StateReader` so a
backend that cannot serve it fails at registration rather than at query or block
time.

This is a small protocol change with a fork gate, and it is the only item that
defuses finding 6. It is not wasted work under any of A/B/C.

**Then** — A. One storage path, ordered scans, no Solidity layer, and the
`not implemented` methods disappear. Node-internal; does not touch consensus.

**Then** — B. Do it after A, so the commitment is designed against one storage
path instead of two.

A and B are orthogonal and compose. A makes every backend behave the same; only B
makes the state verifiable. "Consistent" and "correct" are different properties
and today the system has neither guaranteed.

---

## Immediate risk

Independent of which direction is chosen: `freezePendingPoolDrainWork` depends on
an ordered range scan on the consensus path, and is protected only by finding 6's
wiring. Any storage refactor risks disturbing that. The "next" step above should
not wait for the long-term decision.

Related: #4998, #4999, #5000.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Start by reading state/factory/erigonstore/workingsetstore_erigon.go, api/web3server_marshal.go, and the statedb working-set and digest code cited in the findings. Trace #4998, #4999, and #5000, then verify the Erigon 3 claims against source; done requires an agreed, scoped architecture and acceptance criteria for ordered scans, historical reads, and state commitments.

Written by the indexing model from the issue text.