statistics: ANALYZE shortly after committed bulk INSERT can publish incomplete statistics as healthy
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal reproduce step (Required)
[tidb_recent_insert_analyze_repro.sql](https://github.com/user-attachments/files/30162702/tidb_recent_insert_analyze_repro.sql)
[tidb_recent_insert_analyze_repro_result.txt](https://github.com/user-attachments/files/30162721/tidb_recent_insert_analyze_repro_result.txt)
Run the attached `tidb_recent_insert_analyze_repro.sql` on a dedicated test cluster.
The script creates two identical tables. Each table initially contains 200,000 rows:
```text
100,000 non-NULL rows
100,000 NULL rows
```
After exact baseline statistics are created, another 400,000 committed rows are inserted into each table. Both tables therefore contain:
```text
600,000 total rows
500,000 NULL rows
100,000 non-NULL rows
```
The two paths differ only in when statistics are refreshed:
```text
t_immediate:
bulk INSERT
COMMIT
FLUSH STATS_DELTA ... CLUSTER
ANALYZE TABLE ... ALL COLUMNS WITH 1 SAMPLERATE immediately
t_waited:
bulk INSERT
COMMIT
wait 30 seconds
FLUSH STATS_DELTA ... CLUSTER
ANALYZE TABLE ... ALL COLUMNS WITH 1 SAMPLERATE
```
For both paths, the script prints:
```sql
SHOW ANALYZE STATUS;
SHOW STATS_META;
SHOW STATS_HEALTHY;
SHOW STATS_HISTOGRAMS;
```
In the reproduced run, ordinary queries immediately after COMMIT returned the correct data shape:
```text
actual rows = 600000
actual NULL rows = 500000
```
For `t_immediate`, the cluster-wide flush started about 0.76 seconds after COMMIT and finished successfully. ANALYZE started about 1.82 seconds after COMMIT.
For `t_waited`, ANALYZE started about 30.74 seconds after COMMIT.
### 2. What did you expect to see? (Required)
After the bulk INSERT commits, a manual cluster-wide stats-delta flush followed by a full-sample manual ANALYZE should include all committed rows.
Both tables contain exactly the same data, so both ANALYZE jobs should publish:
```text
Processed_rows = 600000
Row_count = 600000
Null_count = 500000
Modify_count = 0
Healthy = 100
```
If the complete recently committed statistics delta cannot be made visible safely, ANALYZE should not publish incomplete statistics as fully healthy, clear the modification count, and return without a warning.
### 3. What did you see instead (Required)
For `t_immediate`, the explicit cluster-wide flush returned successfully, but `SHOW STATS_META` remained unchanged immediately after the flush:
```text
before FLUSH:
Row_count = 200000
Modify_count = 0
after FLUSH:
Row_count = 200000
Modify_count = 0
```
The subsequent full-sample ANALYZE also returned successfully, but published statistics that contained only the old data plus a very small fraction of the 400,000 newly inserted rows:
```text
actual rows = 600000
actual NULL rows = 500000
SHOW ANALYZE STATUS:
State = finished
Processed_rows = 200031
SHOW STATS_META:
Row_count = 200031
Modify_count = 0
SHOW STATS_HEALTHY:
Healthy = 100
SHOW STATS_HISTOGRAMS:
Null_count = 100031
```
The immediate path therefore underestimated:
```text
total row count by about 3x
NULL row count by about 5x
```
while reporting the statistics as fully healthy.
For the otherwise identical `t_waited` table, after waiting 30 seconds, `SHOW STATS_META` had already become:
```text
Row_count = 600000
Modify_count = 400000
```
Running the same cluster-wide flush and full-sample ANALYZE then produced the correct statistics:
```text
Processed_rows = 600000
Row_count = 600000
Null_count = 500000
Modify_count = 0
Healthy = 100
```
#### Timing-window observations
The following commit-to-ANALYZE intervals were measured in repeated isolated tests using the same 400,000-row bulk INSERT:
| Commit-to-ANALYZE interval | Explicit action before ANALYZE | Published Row_count | Published Null_count | Result |
|---:|---|---:|---:|---|
| 0.77 seconds | none | 200031 | 100031 | incomplete |
| 1.26 seconds | local `FLUSH STATS_DELTA` | 200031 | 100031 | incomplete |
| 1.25 seconds | `FLUSH STATS_DELTA ... CLUSTER` | 200031 | 100031 | incomplete |
| 2.81 seconds | local flush after writer disconnect | 200031 | 100031 | incomplete |
| 30.72 seconds | wait before ANALYZE | 600000 | 500000 | correct |
The final minimal reproduction independently observed:
```text
1.82 seconds after COMMIT:
Row_count = 200031
Null_count = 100031
30.74 seconds after COMMIT:
Row_count = 600000
Null_count = 500000
```
These observations establish that the statistics are still incomplete when ANALYZE starts within approximately three seconds of COMMIT, while they are correct by approximately 30 seconds in this environment.
The exact transition point has not been determined. It may depend on the statistics lease and the internal stats-delta publication cycle. The issue is not that ANALYZE is required to finish within a specific number of
seconds; the issue is that ANALYZE can return successfully and publish incomplete statistics as fully healthy during this window.
At the end of the script, both tables still contained the same 600,000 rows, but their published statistics remained different:
```text
t_immediate:
Row_count = 200031
Null_count = 100031
Healthy = 100
t_waited:
Row_count = 600000
Null_count = 500000
Healthy = 100
```
Therefore, the incorrect statistics on `t_immediate` were not merely a temporary display delay that repaired itself while the waited control ran.
The issue was reproduced on a single-TiDB-node deployment. It appears to be a synchronization gap between recently committed DML statistics deltas and the stats-delta flush performed before ANALYZE. The committed table data is already visible to ordinary queries, but the complete delta is temporarily unavailable to `FLUSH STATS_DELTA ... CLUSTER` and ANALYZE.
#### Suggested fix direction
This issue appears closely related to [#22934](https://github.com/pingcap/tidb/issues/22934) and its follow-up fix [#67939](https://github.com/pingcap/tidb/pull/67939).
Issue #22934 identified that `ANALYZE` could capture the base `Row_count` and `Modify_count` before previously generated statistics deltas had been processed. PR #67939 attempted to close this synchronization gap by adding `flushStatsDeltaForAnalyze()`, which issues a table-scoped `FLUSH STATS_DELTA ... CLUSTER` before Version 2 column ANALYZE captures its base count and modification count from `mysql.stats_meta`.
However, this reproduction suggests that the current pre-ANALYZE flush is not a sufficient synchronization barrier. Shortly after COMMIT, all of the following operations can return successfully while most of the newly committed delta is still unavailable:
```text
FLUSH STATS_DELTA db.table
FLUSH STATS_DELTA db.table CLUSTER
ANALYZE TABLE db.table ALL COLUMNS WITH 1 SAMPLERATE
```
The relevant statistics flow appears to be:
```text
transaction COMMIT
-> transaction table delta
-> SessionStatsCollector
-> SweepSessionStatsList()
-> global table-delta map
-> DumpStatsDeltaToKV()
-> mysql.stats_meta
-> ANALYZE captures base count / modify count
```
The observed behavior suggests that `flushStatsDeltaForAnalyze()` can complete before the full committed delta has become visible to `SweepSessionStatsList()` or `DumpStatsDeltaToKV()`. ANALYZE then uses an outdated base count, publishes incomplete statistics, resets `Modify_count` to zero, and reports `Healthy=100`.
One possible fix is to introduce an explicit stats-delta generation barrier for ANALYZE:
1. When ANALYZE starts, capture the latest committed statistics-delta generation for each target table.
2. Require every TiDB instance to sweep all relevant session collectors through at least that generation.
3. Persist the target tables' deltas to `mysql.stats_meta`.
4. Wait for acknowledgements from all required instances.
5. Only after the barrier succeeds, read the base `Row_count` and `Modify_count` and start building statistics.
The local TiDB instance should also participate explicitly in this barrier instead of relying only on the cluster broadcast path.
If the barrier cannot be completed, ANALYZE should fail or return a visible warning rather than publishing incomplete statistics with:
```text
Modify_count = 0
Healthy = 100
```
Regression tests should cover an immediate bulk INSERT followed by ANALYZE, with the writer session both alive and disconnected, and verify that the following three paths all include the complete committed delta:
```text
local FLUSH STATS_DELTA
cluster-wide FLUSH STATS_DELTA
the internal pre-ANALYZE stats-delta flush
```
### 4. What is your TiDB version? (Required)
Output of:
```sql
SELECT TIDB_VERSION();
```
```text
Release Version: v8.5.7
Edition: Community
Git Commit Hash: 202b7f47286a1109b5c957401d34c9358d130ae0
Git Branch: HEAD
UTC Build Time: 2026-07-15 02:06:00
GoVersion: go1.25.10
Race Enabled: false
Check Table Before Drop: false
Store: tikv
```
The reproduction used one TiDB node and:
```text
performance.stats-lease = 3s
```
Contributor guide
Research direction
Start by reproducing the attached SQL scenario, then trace flushStatsDeltaForAnalyze(), SessionStatsCollector, SweepSessionStatsList(), and DumpStatsDeltaToKV(). Add synchronization so ANALYZE sees the complete committed delta, or reports a visible failure instead of healthy incomplete statistics. Regression coverage should include local and cluster-wide FLUSH STATS_DELTA and internal pre-ANALYZE flushing with the writer session alive and disconnected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100