erigontech / erigontech/erigon
Investigate how much RPC↔exec state sharing to keep: STATE_CACHE_FILLS default, RPC cache reads, and the SharedDomains overlay
- Dominant language
- Go
- Stars
- 3.6k
- Forks
- 1.5k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 455
Description
How much RPC↔execution state sharing should erigon keep on by default? This covers the `STATE_CACHE_FILLS` default and, prompted by https://github.com/erigontech/erigon/pull/22444#issuecomment-5193286206, the wider concern behind it: whether the embedded RPC daemon should use the shared `StateCache` and the global `SharedDomains` overlay at all.
The sharing has three separable layers, with different value claims and different risks:
| Layer | What it is | Value claim | Risk/cost |
|---|---|---|---|
| 1. RPC/reader **fills** into the shared cache | readers *write* the cache on miss (admission-gated since #22444) | warms read-mostly keys (hot storage, popular code) that applies can never cache | the entire stale-fill bug family (#22356, #22463, #23028); admission machinery on the hot read path (~48 B / 2 allocs per cold fill, exact-frontier lookups) |
| 2. RPC **reads** from the shared cache | RPC-originated getters consult the cache and serve hits (latest-wins, possibly newer than the request's tx) | skips the file-accessor/MDBX stack on repeated reads | latest-wins semantics for RPC; ties RPC read behavior to exec's cache contents |
| 3. The **SharedDomains overlay** in RPC | `execmodule.Cache.View` hands RPC a getter over exec's current/published SD | embedded-RPC `latest` sees the newest (even uncommitted, mid-FCU) state | RPC reads coupled to exec internals (published-SD lifetime, background-commit windows — the #21613/#23003 family); the "premature design" objection |
## Layer 1 — should `STATE_CACHE_FILLS` default to false (apply-only cache)?
Apply-only mode (introduced in #22444) disables every reader write: admission-gated fills, addr→codeHash seeds, code-size fills. Applies from `SharedDomains.Commit` remain the only writer.
Why consider it:
- **Every bug in this family is a fill bug.** #22356 (resurrection), #22463 (unwind readmission), #23028 (publication staleness) all poison the cache through reader fills; applies are authoritative by construction. Apply-only eliminates the class rather than gating it.
- **Fills may pay less than assumed.** Parallel-exec workers hold one read tx per run, so their fills self-starve after the first mid-run commit. With frozen-block catchup wired into the apply stream (#23033), execution keeps the cache warm for every *written* key. The residual value is warming for read-mostly keys — only fills can cache those.
This was the explicit open crux of the #22444 review (the admission gate vs. dropping reader fills entirely); the gate was built, the default deferred pending measurement.
## Layers 2+3 — should the RPC daemon use the shared cache and overlay at all?
The comment linked above argues the sharing is unnecessary and premature, and asks for a feature flag disabling global shared domains and caching in the RPC daemon. Mechanically that flag is small: `CacheView` already has fallback paths for the nil-SD case, so an RPC-isolation mode = bypass the overlay getter and the cache, read straight through the plain temporal tx.
The two halves need separate verdicts:
- **Cache reads (layer 2)** are a pure perf claim — measurable (see below). If RPC-originated cache hits are rare or cheap to lose, disabling cache use for RPC costs little and decouples RPC from exec's cache entirely.
- **The overlay (layer 3) is semantics, not perf**: without it, embedded-RPC `latest` lags execution until commit. That gap is exactly what the flush+commit-online simplification (the same PR-22444 discussion thread) shrinks — with synchronous per-FCU commits, `latest`≈committed most of the time, and disabling the overlay becomes semantically cheap. Verdict on layer 3 should therefore be coordinated with that work, not taken standalone.
## Measurements the decisions need
1. **Interleaved A/B on execution throughput** — eest benchmark shards and/or a tip-following node, `STATE_CACHE_FILLS` on vs off (an earlier single-pass attempt was discarded: run-order cold-cache artifact, env propagation unverified).
2. **RPC latency A/B at `latest`** on an RPC-serving node: (a) fills on vs off; (b) cache reads on vs off; (c) overlay on vs off (correctness caveat above).
3. **Fill-outcome counters over a real sync** (`PrintStatsAndReset`: admitted / rejected / no-frontier, #23033) plus hit ratios: if admitted fills are a small fraction of attempts, or fill-originated entries rarely produce hits, layer 1's value collapses.
4. **Read-ahead interaction**: `STATE_CACHE_FILLS=false` also silences `BlockReadAheader` warming; the #22146 lineage suggests warming mattered there — quantify.
## Possible outcomes (not mutually exclusive)
- Flip `STATE_CACHE_FILLS` default to false — the machinery stays as an opt-in lever; #22463 drops in urgency (guards an opt-in path); release lines can adopt the flipped default as a mitigation posture (3.4/3.5 lack the admission gate).
- Add an RPC-isolation flag (no overlay, no cache for RPC-originated reads), default per measurement; coordinate the overlay half with flush+commit-online.
- Keep defaults as-is with the measurements as evidence that the sharing pays.
Contributor guide
Assessment
This issue has not been assessed yet.