erigontech / erigontech/erigon

StateCache: fills can go stale when a snapshot download extends file visibility with never-applied state

Open
#23,028 1 comment 0 reactions 1 assignee Claimed by @yperbasis View on GitHub
ErigonDB
Dominant language
Go
Stars
3.6k
Forks
1.5k
Avg merge
1d 16h
Merged PRs (30d)
455

Description

Found while qualifying the `Frontier` stale-low contract in #22444 (review follow-up). Same family as #22463: the cache's latest-value claim is anchored to the apply stream, and state that arrives outside that stream can invalidate fills without anything noticing. Code-inspection level — no reproduction built yet; the startup ordering needs verification.

## Mechanism

Fill admission (`frontier >= appliedEnd`) is sound as long as every txNum beyond the filling view's frontier eventually flows through `Applier.Apply`. Snapshot download breaks that assumption: `ForceReopenUnderlyingFilesTx` / files reopen after a download **extends visibility with state the process never applied**, so `appliedEnd` knows nothing about it and no cache entry is invalidated.

Concrete shape (negative fill, the sharpest case):

1. Node starts with an incomplete snapshot set; a fill-enabled `StateCache` is wired at `ExecModule` construction.
2. A read at `latest` (embedded RPC, read-ahead) misses account K — its data lives in a segment still being downloaded. The read legitimately returns "absent"; the negative is filled, admitted (cold cache: `appliedEnd = 0`).
3. The download completes; the files view extends; K now exists — via files, not via any `Apply`.
4. Every later `GetLatest(K)` hits the cached negative and serves "absent" for an existing account, until K's next in-process write.

Positive fills have the same shape (an old value cached as latest while the downloaded segment holds a newer one).

The dependency-clamp instance of this family was closed in #22444 (`DomainVisibleEnd` answers `ok=false` for clamped values views) — but the download case is not a clamped view; the filling view is perfectly coherent at fill time. The problem is purely that the future arrived outside the apply stream.

## Why the existing machinery doesn't catch it

- Admission compares against `appliedEnd`, which only applies advance; a files extension bypasses it.
- Unwind invalidation (epochs + floor) fires on unwinds, not on visibility raises.
- The visibility-lowering guard (`BindAggregator` → `recalcVisibleFiles` assert) forbids *lowering*; raising is allowed and is exactly this path.
- No production path calls `StateCache` `Clear`/invalidation on `ForceReopenUnderlyingFilesTx` or `OnFilesChange`.

## Exposure

Narrow in practice: fills at `latest` while segments are still downloading (initial sync, catch-up after downtime, new segment types). A freshly syncing node serves little meaningful `latest` traffic, and locally *built* segments cover data that already flowed through applies (safe). But the invariant violation is real, and negative entries make it user-visible (`eth_getBalance` returning 0/absent for an existing account after sync completes).

## Fix directions

1. **Invalidate on extension**: bump the cache epochs (or `Clear`) from the files-change hook when a reopen extends a cached state domain's visible end beyond `appliedEnd` — precise trigger, no cost in the steady state where extensions only cover applied ranges.
2. **Wire late**: attach the fill-enabled cache only once initial download is complete (apply-only until then). Simple, but doesn't cover later catch-up downloads.
3. **Advance `appliedEnd` on extension**: treat a download-driven extension as an authority advance (rejects the stale fills' *admission* window going forward, but does not evict entries already admitted — insufficient alone).

Direction 1 looks right; direction 3 alone is not enough.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.