erigontech / erigontech/erigon
StateCache: fills can go stale when a snapshot download extends file visibility with never-applied state
- Dominant language
- Go
- Stars
- 3.6k
- Forks
- 1.5k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 455
Description
Found while qualifying the `Frontier` stale-low contract in #22444 (review follow-up). Same family as #22463: the cache's latest-value claim is anchored to the apply stream, and state that arrives outside that stream can invalidate fills without anything noticing. Code-inspection level — no reproduction built yet; the startup ordering needs verification.
## Mechanism
Fill admission (`frontier >= appliedEnd`) is sound as long as every txNum beyond the filling view's frontier eventually flows through `Applier.Apply`. Snapshot download breaks that assumption: `ForceReopenUnderlyingFilesTx` / files reopen after a download **extends visibility with state the process never applied**, so `appliedEnd` knows nothing about it and no cache entry is invalidated.
Concrete shape (negative fill, the sharpest case):
1. Node starts with an incomplete snapshot set; a fill-enabled `StateCache` is wired at `ExecModule` construction.
2. A read at `latest` (embedded RPC, read-ahead) misses account K — its data lives in a segment still being downloaded. The read legitimately returns "absent"; the negative is filled, admitted (cold cache: `appliedEnd = 0`).
3. The download completes; the files view extends; K now exists — via files, not via any `Apply`.
4. Every later `GetLatest(K)` hits the cached negative and serves "absent" for an existing account, until K's next in-process write.
Positive fills have the same shape (an old value cached as latest while the downloaded segment holds a newer one).
The dependency-clamp instance of this family was closed in #22444 (`DomainVisibleEnd` answers `ok=false` for clamped values views) — but the download case is not a clamped view; the filling view is perfectly coherent at fill time. The problem is purely that the future arrived outside the apply stream.
## Why the existing machinery doesn't catch it
- Admission compares against `appliedEnd`, which only applies advance; a files extension bypasses it.
- Unwind invalidation (epochs + floor) fires on unwinds, not on visibility raises.
- The visibility-lowering guard (`BindAggregator` → `recalcVisibleFiles` assert) forbids *lowering*; raising is allowed and is exactly this path.
- No production path calls `StateCache` `Clear`/invalidation on `ForceReopenUnderlyingFilesTx` or `OnFilesChange`.
## Exposure
Narrow in practice: fills at `latest` while segments are still downloading (initial sync, catch-up after downtime, new segment types). A freshly syncing node serves little meaningful `latest` traffic, and locally *built* segments cover data that already flowed through applies (safe). But the invariant violation is real, and negative entries make it user-visible (`eth_getBalance` returning 0/absent for an existing account after sync completes).
## Fix directions
1. **Invalidate on extension**: bump the cache epochs (or `Clear`) from the files-change hook when a reopen extends a cached state domain's visible end beyond `appliedEnd` — precise trigger, no cost in the steady state where extensions only cover applied ranges.
2. **Wire late**: attach the fill-enabled cache only once initial download is complete (apply-only until then). Simple, but doesn't cover later catch-up downloads.
3. **Advance `appliedEnd` on extension**: treat a download-driven extension as an authority advance (rejects the stale fills' *admission* window going forward, but does not evict entries already admitted — insufficient alone).
Direction 1 looks right; direction 3 alone is not enough.
Contributor guide
Assessment
This issue has not been assessed yet.