erigontech / erigontech/erigon
BlockStateCache vs StateCache
- Dominant language
- Go
- Stars
- 3.6k
- Forks
- 1.5k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 455
Description
Erigon has two caches over the same data: `BlockStateCache` and `StateCache`. Both hold Accounts, Storage and Code. They sit on top of each other in the read chain, and they work against each other.
### The two caches
| | `BlockStateCache` | `StateCache` |
|---|---|---|
| Where | `execution/state/rw_v3.go:987` | `execution/cache/state_cache.go:59` |
| Created | fresh for every block (`exec3.go:670`, `exec3_parallel.go:2451`) | once per process (`exec_module.go:317`) |
| Holds | pre-block values, plus this block's writes | committed values only |
| Freshness | none — entries carry no txNum | txNum-stamped, frontier-gated, unwind-invalidated |
| Size limit | none — plain maps | byte budget, registered in the shared memory envelope |
The read chain is: `BlockStateCache` → `sd.mem` → `parent.mem` → `StateCache` → `BranchCache` → files.
### What goes wrong
`BlockStateCache` sits above `StateCache`, so it absorbs the reads `StateCache` exists to serve — then throws them away at the end of the block.
1. **`StateCache` sees a decimated read stream.** Its Account/Storage tiers are `freelru.ShardedLRU`, which promotes an entry on every `Get`. A slot read 5000 times in a block reaches it once. Eviction then ranks keys by "how many blocks touched this key first", not by how hot they are. The hottest keys look the coldest.
2. **Hit-rate numbers are not comparable.** `UpdateStateCacheHit/Miss` (`domain_shared.go:1271`) only fires after both `BlockStateCache` and `sd.mem` miss. The reported rate is conditional on two upstream misses.
3. **Only one of them is budgeted.** `StateCache` has byte budgets and releases its reservation in `ExecModule.Close`. `BlockStateCache` is unbounded maps, sized by the block's working set, outside that accounting.
4. **Two freshness models for one problem.** `StateCache` stamps every entry with a txNum and invalidates by epoch and floor. `BlockStateCache` stamps nothing. Its correctness rests on a short life, so the two cannot be merged as they stand.
### The committed tier does not isolate anything
`BlockStateCache`'s docstring says the committed tier gives "a stable view for GetCommittedState that isn't affected by intra-block DomainPut calls" (`rw_v3.go:972`). In the parallel executor there are no such calls for these three domains:
- `sd.mem` changes only in `blockStateCache.Flush` during `completeBlock`, and block N+1 is not scheduled until block N's `completeBlock` returns (`exec3_parallel.go:1218-1233`).
- `StateCache` is written only at flush and only invalidated on unwind — "the executor does not touch it during forward execution" (`exec3_parallel.go:1264`).
- The other in-block writers, the commitment calculator and `ApplyTxIndexes`, write CommitmentDomain, ReceiptDomain and the inverted indices. `BlockStateCache` covers none of those.
So the whole chain below is already frozen for the length of a block. The committed tier is a plain read cache, not an isolation mechanism. The hazard the docstring names is real only when `blockCache == nil` — the serial executor, which does not use this cache (`rw_v3.go:401`).
The write buffer half of `BlockStateCache` is a different matter. It has to be per-block, and it is what freezes the chain in the first place.
### What the committed tier still earns
It is not dead weight today:
- It saves a chain walk per read: an `sd.mem` probe under `latestStateLock.RLock`, then a `StateCache` probe.
- Mid-batch there is a band of keys written by earlier blocks in the same batch. Those live in `sd.mem` only and are absent from `StateCache`, which is populated at flush. For that band the committed tier is not redundant at all.
### Suggested direction
Close the gap between `sd.mem` and `StateCache` first. If `StateCache` can serve values that are still sitting in `sd.mem`, the second reason above disappears, the first shrinks to a cheap probe, and the committed tier can go — which removes the redundancy and gives `StateCache` back its real read stream. This overlaps with #23139.
Worth measuring before changing anything: how often the committed tier hits, and how much of that would have hit `StateCache` instead.
### Adjacent cleanup
`execution/state/block_cache_multiblock_flush_test.go` opens with "The parallel executor reuses one BlockStateCache across all blocks in a batch". That was true before #20805, which made the cache per-block and removed the stale-committed dedup in `Flush` that caused a wrong trie root at block 24839762. The test still earns its place — it pins `Flush` behaviour whatever the allocation model — but the docstring reads as a description of current behaviour, and it is not.
Contributor guide
Assessment
This issue has not been assessed yet.