Timing Issues in the Statistics Cache: Lost Updates, Inconsistent Reads, and Stuck Operations

## Summary

On the current master, after statistics are written to the cache, subsequent reads may still fail to see the newly written data. When a full reload replaces the cache, operations still using the old cache may hang or trigger a panic (runtime exception) inside the cache API.

## Two Layers

- [Main cache — Checked first when reading a single table](https://github.com/pingcap/tidb/blob/c767f6fd8c01e9dcb459767611c0c1d4110d210d/pkg/statistics/handle/cache/internal/lfu/lfu_cache.go#L116). Updates to tables already present take effect immediately; tables not yet in the main cache are queued, and admission is handled in the background.

- [Fallback map — An in-memory map holding information for every cached table](https://github.com/pingcap/tidb/blob/c767f6fd8c01e9dcb459767611c0c1d4110d210d/pkg/statistics/handle/cache/internal/lfu/key_set_shard.go#L37); it is also the data source for "list all tables." It allows tables that have not yet been admitted to be read. After memory eviction, [it retains a copy with the detailed statistics stripped out](https://github.com/pingcap/tidb/blob/c767f6fd8c01e9dcb459767611c0c1d4110d210d/pkg/statistics/handle/cache/internal/lfu/lfu_cache.go#L177).

[Full reload builds a separate cache, swaps it in as the active cache, and closes the old one](https://github.com/pingcap/tidb/blob/c767f6fd8c01e9dcb459767611c0c1d4110d210d/pkg/statistics/handle/cache/statscache.go#L276). Both the initial load at startup and [`REFRESH STATS` without a table name](https://github.com/pingcap/tidb/blob/c767f6fd8c01e9dcb459767611c0c1d4110d210d/pkg/executor/simple.go#L3175) go through this replacement process.

## Category 1: In-flight background processing masks newer writes

### 1.1 The column from the later load is still missing after both writes return

Trigger condition: An earlier write to the same table has not yet completed its initial admission, while a later write has already published a new copy.

The refresh task first writes the old copy A. The load task reads A, fills in the column statistics, and writes back a new copy B. If both copies are waiting for initial admission, the following sequence can occur:

```mermaid
sequenceDiagram
participant R as Refresh task
participant L as Stats loader
participant F as Fallback map
participant P as Primary cache / worker
R->>F: Publish old copy A
R->>P: Queue A, write call returns
Note over R,P: A is not yet in the primary cache
L->>F: Read A, publish B with the loaded column
L->>P: Queue B, write call returns
P->>P: Admit A, reject B as a duplicate insert
L->>P: Read the table after the queue drains
P-->>L: Return A, the loaded column is missing
```

The same mechanism has two additional tested variants:

- Two loaded copies written back in succession: Before the earliest copy A is admitted, B and C are written, each filling in a different column. The primary cache still ends up with A, and both backfilled columns are missing.
- A new column is visible, then disappears: Before admission, a read from the fallback map returns C; after admission, the read is served from A instead. Even with no new user writes in between, the read result regresses.

### 1.2 Single-table reads and "list all tables" may see different versions

Trigger condition: The eviction callback for an old copy interleaves with the publication of a newer copy.

1. The cache begins evicting a table's old copy.
2. The load task publishes a newer, complete copy.
3. [The eviction callback for the old copy runs later and writes the stripped-down old copy into the fallback map](https://github.com/pingcap/tidb/blob/c767f6fd8c01e9dcb459767611c0c1d4110d210d/pkg/statistics/handle/cache/internal/lfu/lfu_cache.go#L167).
4. The new copy enters the primary cache, but the old copy in the fallback map is not updated accordingly.

At this point, even after background processing has finished, a single-table read returns the new copy while "list all tables" returns the old one. If the listing result is written back, the primary cache also regresses to the old copy. [The stats initialization code contains exactly this "list, then write back" pattern](https://github.com/pingcap/tidb/blob/c767f6fd8c01e9dcb459767611c0c1d4110d210d/pkg/statistics/handle/bootstrap.go#L766).

## Category 2: A swap or delete between the read and the write causes the old copy to be dropped or written to the wrong place

### 2.1 A table read from the old cache overwrites the table in the new cache
Trigger condition: The active cache is swapped after the read but before the write-back.

[The load task's read and write-back each look up whichever cache is active at that moment](https://github.com/pingcap/tidb/blob/c767f6fd8c01e9dcb459767611c0c1d4110d210d/pkg/statistics/handle/syncload/stats_syncload.go#L568). If a swap happens between the two steps, they straddle two different caches:

```mermaid
sequenceDiagram
participant L as Stats loader
participant C as Active cache
participant R as Full reload
L->>C: Read the table from the old cache
C-->>L: Return table metadata at version 1
R->>C: Install new cache with version 2, close old cache
L->>L: Prepare a write-back from the earlier snapshot
L->>C: Look up the active cache again, write the old copy
Note over L,C: The write reaches the new cache, table version regresses from 2 to 1
```

### 2.2 A stale write-back reinserts the cache entry after the delete has completed

Trigger condition: The load task reads the table before the delete and writes back only after it.

When a refresh finds that the table no longer exists in the schema, [it deletes the table's stats cache entry](https://github.com/pingcap/tidb/blob/c767f6fd8c01e9dcb459767611c0c1d4110d210d/pkg/statistics/handle/cache/statscache.go#L122); the old copy already read by the load task, however, is not invalidated as a result.

```mermaid
sequenceDiagram
participant L as Stats loader
participant R as Refresh task
participant C as Stats cache
L->>C: Read table snapshot
R->>C: Delete the table entry
Note over R,C: Deletion completes, Get and Values no longer contain the table
L->>C: Write back the earlier snapshot
Note over L,C: Get and Values contain the table again
```

A delete racing with a write-back can also leave a state where the entry exists only in the primary cache and not in the fallback map. This is because [the delete does not modify both layers atomically](https://github.com/pingcap/tidb/blob/c767f6fd8c01e9dcb459767611c0c1d4110d210d/pkg/statistics/handle/cache/internal/lfu/lfu_cache.go#L136):

```mermaid
sequenceDiagram
participant R as Refresh task
participant L as Stats loader
participant P as Primary cache / worker
participant F as Fallback map
Note over R,F: The loader already holds an earlier snapshot
R->>P: Remove entry, queue delete marker
L->>F: Publish the earlier snapshot
L->>P: Queue write-back after delete marker
R->>F: Finish deletion by removing fallback entry
P->>P: Process delete marker, then admit write-back
Note over P,F: Get finds the entry, Values does not
```

## Category 3: The cache an operation is using is closed before the operation finishes

The common condition is: an operation first obtains a pointer to the cache, and another party then swaps out and closes that cache.

### 3.1 A write during close triggers a panic

1. A write has started but has not yet sent data to the write buffer.
2. [Another party closes the write buffer](https://github.com/dgraph-io/ristretto/blob/v0.1.1/cache.go#L352).
3. The write proceeds with the send, [triggering `send on closed channel`](https://github.com/dgraph-io/ristretto/blob/v0.1.1/cache.go#L290).

### 3.2 A write retries endlessly while holding a closed cache

1. [`StatsCache.Put` obtains a pointer to the old cache](https://github.com/pingcap/tidb/blob/c767f6fd8c01e9dcb459767611c0c1d4110d210d/pkg/statistics/handle/cache/statscacheinner.go#L80).
2. The cache is swapped and the old cache is closed.
3. The write fails, and [`Put` retries every 5 ms](https://github.com/pingcap/tidb/blob/c767f6fd8c01e9dcb459767611c0c1d4110d210d/pkg/statistics/handle/cache/statscacheinner.go#L114); a closed cache never accepts writes again, and the loop does not check for a cancellation signal.

### 3.3 Waiters never receive the completion notification

1. A wait call interleaves with the close.
2. The close stops background processing, and [a wait marker is left unprocessed](https://github.com/dgraph-io/ristretto/blob/v0.1.1/cache.go#L211).
3. `Close` has already returned, but the waiter remains blocked.

Start by tracing cache ownership and swap behavior through pkg/statistics/handle/cache/statscache.go, statscacheinner.go, internal/lfu/lfu_cache.go, and syncload/stats_syncload.go, then inspect the linked Ristretto close and write paths. Reproduce the reported interleavings around admission, eviction, deletion, reload, and close. Done means preventing lost or regressed versions, inconsistent layers, stale reinsertion, panics, endless retries, and blocked waiters.

Written by the indexing model from the issue text.