erigontech / erigontech/erigon

BlockStateCache vs StateCache

Open
#23,140 4 comments 0 reactions 1 assignee Claimed by @sudeepdino008 View on GitHub
ErigonDB
Dominant language
Go
Stars
3.6k
Forks
1.5k
Avg merge
1d 16h
Merged PRs (30d)
455

Description

Erigon has two caches over the same data: `BlockStateCache` and `StateCache`. Both hold Accounts, Storage and Code. They sit on top of each other in the read chain, and they work against each other.

### The two caches

| | `BlockStateCache` | `StateCache` |
|---|---|---|
| Where | `execution/state/rw_v3.go:987` | `execution/cache/state_cache.go:59` |
| Created | fresh for every block (`exec3.go:670`, `exec3_parallel.go:2451`) | once per process (`exec_module.go:317`) |
| Holds | pre-block values, plus this block's writes | committed values only |
| Freshness | none — entries carry no txNum | txNum-stamped, frontier-gated, unwind-invalidated |
| Size limit | none — plain maps | byte budget, registered in the shared memory envelope |

The read chain is: `BlockStateCache` → `sd.mem` → `parent.mem` → `StateCache` → `BranchCache` → files.

### What goes wrong

`BlockStateCache` sits above `StateCache`, so it absorbs the reads `StateCache` exists to serve — then throws them away at the end of the block.

1. **`StateCache` sees a decimated read stream.** Its Account/Storage tiers are `freelru.ShardedLRU`, which promotes an entry on every `Get`. A slot read 5000 times in a block reaches it once. Eviction then ranks keys by "how many blocks touched this key first", not by how hot they are. The hottest keys look the coldest.
2. **Hit-rate numbers are not comparable.** `UpdateStateCacheHit/Miss` (`domain_shared.go:1271`) only fires after both `BlockStateCache` and `sd.mem` miss. The reported rate is conditional on two upstream misses.
3. **Only one of them is budgeted.** `StateCache` has byte budgets and releases its reservation in `ExecModule.Close`. `BlockStateCache` is unbounded maps, sized by the block's working set, outside that accounting.
4. **Two freshness models for one problem.** `StateCache` stamps every entry with a txNum and invalidates by epoch and floor. `BlockStateCache` stamps nothing. Its correctness rests on a short life, so the two cannot be merged as they stand.

### The committed tier does not isolate anything

`BlockStateCache`'s docstring says the committed tier gives "a stable view for GetCommittedState that isn't affected by intra-block DomainPut calls" (`rw_v3.go:972`). In the parallel executor there are no such calls for these three domains:

- `sd.mem` changes only in `blockStateCache.Flush` during `completeBlock`, and block N+1 is not scheduled until block N's `completeBlock` returns (`exec3_parallel.go:1218-1233`).
- `StateCache` is written only at flush and only invalidated on unwind — "the executor does not touch it during forward execution" (`exec3_parallel.go:1264`).
- The other in-block writers, the commitment calculator and `ApplyTxIndexes`, write CommitmentDomain, ReceiptDomain and the inverted indices. `BlockStateCache` covers none of those.

So the whole chain below is already frozen for the length of a block. The committed tier is a plain read cache, not an isolation mechanism. The hazard the docstring names is real only when `blockCache == nil` — the serial executor, which does not use this cache (`rw_v3.go:401`).

The write buffer half of `BlockStateCache` is a different matter. It has to be per-block, and it is what freezes the chain in the first place.

### What the committed tier still earns

It is not dead weight today:

- It saves a chain walk per read: an `sd.mem` probe under `latestStateLock.RLock`, then a `StateCache` probe.
- Mid-batch there is a band of keys written by earlier blocks in the same batch. Those live in `sd.mem` only and are absent from `StateCache`, which is populated at flush. For that band the committed tier is not redundant at all.

### Suggested direction

Close the gap between `sd.mem` and `StateCache` first. If `StateCache` can serve values that are still sitting in `sd.mem`, the second reason above disappears, the first shrinks to a cheap probe, and the committed tier can go — which removes the redundancy and gives `StateCache` back its real read stream. This overlaps with #23139.

Worth measuring before changing anything: how often the committed tier hits, and how much of that would have hit `StateCache` instead.

### Adjacent cleanup

`execution/state/block_cache_multiblock_flush_test.go` opens with "The parallel executor reuses one BlockStateCache across all blocks in a batch". That was true before #20805, which made the cache per-block and removed the stale-committed dedup in `Flush` that caused a wrong trie root at block 24839762. The test still earns its place — it pins `Flush` behaviour whatever the allocation model — but the docstring reads as a description of current behaviour, and it is not.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.