erigontech / erigontech/erigon

Investigate how much RPC↔exec state sharing to keep: STATE_CACHE_FILLS default, RPC cache reads, and the SharedDomains overlay

Open
#23,082 2 comments 0 reactions 2 assignees Claimed by @lupin012 View on GitHub
performance RPC
Dominant language
Go
Stars
3.6k
Forks
1.5k
Avg merge
1d 16h
Merged PRs (30d)
455

Description

How much RPC↔execution state sharing should erigon keep on by default? This covers the `STATE_CACHE_FILLS` default and, prompted by https://github.com/erigontech/erigon/pull/22444#issuecomment-5193286206, the wider concern behind it: whether the embedded RPC daemon should use the shared `StateCache` and the global `SharedDomains` overlay at all.

The sharing has three separable layers, with different value claims and different risks:

| Layer | What it is | Value claim | Risk/cost |
|---|---|---|---|
| 1. RPC/reader **fills** into the shared cache | readers *write* the cache on miss (admission-gated since #22444) | warms read-mostly keys (hot storage, popular code) that applies can never cache | the entire stale-fill bug family (#22356, #22463, #23028); admission machinery on the hot read path (~48 B / 2 allocs per cold fill, exact-frontier lookups) |
| 2. RPC **reads** from the shared cache | RPC-originated getters consult the cache and serve hits (latest-wins, possibly newer than the request's tx) | skips the file-accessor/MDBX stack on repeated reads | latest-wins semantics for RPC; ties RPC read behavior to exec's cache contents |
| 3. The **SharedDomains overlay** in RPC | `execmodule.Cache.View` hands RPC a getter over exec's current/published SD | embedded-RPC `latest` sees the newest (even uncommitted, mid-FCU) state | RPC reads coupled to exec internals (published-SD lifetime, background-commit windows — the #21613/#23003 family); the "premature design" objection |

## Layer 1 — should `STATE_CACHE_FILLS` default to false (apply-only cache)?

Apply-only mode (introduced in #22444) disables every reader write: admission-gated fills, addr→codeHash seeds, code-size fills. Applies from `SharedDomains.Commit` remain the only writer.

Why consider it:
- **Every bug in this family is a fill bug.** #22356 (resurrection), #22463 (unwind readmission), #23028 (publication staleness) all poison the cache through reader fills; applies are authoritative by construction. Apply-only eliminates the class rather than gating it.
- **Fills may pay less than assumed.** Parallel-exec workers hold one read tx per run, so their fills self-starve after the first mid-run commit. With frozen-block catchup wired into the apply stream (#23033), execution keeps the cache warm for every *written* key. The residual value is warming for read-mostly keys — only fills can cache those.

This was the explicit open crux of the #22444 review (the admission gate vs. dropping reader fills entirely); the gate was built, the default deferred pending measurement.

## Layers 2+3 — should the RPC daemon use the shared cache and overlay at all?

The comment linked above argues the sharing is unnecessary and premature, and asks for a feature flag disabling global shared domains and caching in the RPC daemon. Mechanically that flag is small: `CacheView` already has fallback paths for the nil-SD case, so an RPC-isolation mode = bypass the overlay getter and the cache, read straight through the plain temporal tx.

The two halves need separate verdicts:
- **Cache reads (layer 2)** are a pure perf claim — measurable (see below). If RPC-originated cache hits are rare or cheap to lose, disabling cache use for RPC costs little and decouples RPC from exec's cache entirely.
- **The overlay (layer 3) is semantics, not perf**: without it, embedded-RPC `latest` lags execution until commit. That gap is exactly what the flush+commit-online simplification (the same PR-22444 discussion thread) shrinks — with synchronous per-FCU commits, `latest`≈committed most of the time, and disabling the overlay becomes semantically cheap. Verdict on layer 3 should therefore be coordinated with that work, not taken standalone.

## Measurements the decisions need

1. **Interleaved A/B on execution throughput** — eest benchmark shards and/or a tip-following node, `STATE_CACHE_FILLS` on vs off (an earlier single-pass attempt was discarded: run-order cold-cache artifact, env propagation unverified).
2. **RPC latency A/B at `latest`** on an RPC-serving node: (a) fills on vs off; (b) cache reads on vs off; (c) overlay on vs off (correctness caveat above).
3. **Fill-outcome counters over a real sync** (`PrintStatsAndReset`: admitted / rejected / no-frontier, #23033) plus hit ratios: if admitted fills are a small fraction of attempts, or fill-originated entries rarely produce hits, layer 1's value collapses.
4. **Read-ahead interaction**: `STATE_CACHE_FILLS=false` also silences `BlockReadAheader` warming; the #22146 lineage suggests warming mattered there — quantify.

## Possible outcomes (not mutually exclusive)

- Flip `STATE_CACHE_FILLS` default to false — the machinery stays as an opt-in lever; #22463 drops in urgency (guards an opt-in path); release lines can adopt the flipped default as a mitigation posture (3.4/3.5 lack the admission gate).
- Add an RPC-isolation flag (no overlay, no cache for RPC-originated reads), default per measurement; coordinate the overlay half with flush+commit-online.
- Keep defaults as-is with the measurements as evidence that the sharing pays.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.