ClickHouse / ClickHouse/ClickHouse
Statistics-based part pruning wrongly prunes on a monotonic DateTime wrapper when the Date range crosses 2106 (silent empty result)
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
### Describe what's wrong
`use_statistics_for_part_pruning` (default 1) with auto-collected statistics (`materialize_statistics_on_merge = 1`) prunes a part that contains matching rows when the predicate wraps a `Date` column in a `DateTime`-producing monotonic function (`toStartOfDay`, `toDateTime`, ...) and the column's value range extends past the `DateTime` overflow boundary (2106-02-07): the function is no longer monotonic over the part's `[min, max]` range, so the range-transformed bound is wrong and the whole part is skipped — a silent empty/short result at default settings.
This is the same non-monotonic-past-overflow hole that KeyCondition had in #101814, now in the statistics part-pruning path. Before the part has statistics (fresh insert), the result is correct; the first merge builds the auto statistics and flips the result.
### How to reproduce
```sql
CREATE TABLE statmin (d Date, v Int64) ENGINE = MergeTree ORDER BY tuple();
INSERT INTO statmin SELECT toDate('2020-01-01') + number * 40, number FROM numbers(1300);
-- d range: 1970-01-07 .. 2149-05-04 (crosses the DateTime 2106 boundary)
SELECT count() FROM statmin WHERE toStartOfDay(d) > toDateTime('2020-11-06 00:00:00');
-- 779 (correct — no statistics yet)
OPTIMIZE TABLE statmin FINAL; -- materialize_statistics_on_merge builds the stats
SELECT count() FROM statmin WHERE toStartOfDay(d) > toDateTime('2020-11-06 00:00:00');
-- 0 (wrong)
SELECT count() FROM statmin WHERE toStartOfDay(d) > toDateTime('2020-11-06 00:00:00')
SETTINGS use_statistics_for_part_pruning = 0;
-- 779 (correct)
```
`EXPLAIN indexes = 1` shows the part dropped by the `Statistics` entry (`Parts: 0/1`). A plain `WHERE d > toDate('2020-11-06')` (no function wrapper) stays correct, and the same data with all dates before 2106 stays correct — the overflow crossing is load-bearing.
Verified on `26.8.1.32` and same-day master `26.8.1.63` (`18de9dbe6d4a`).
### Expected behavior
Statistics-based pruning must not treat a function as monotonic across a range where its result type overflows — same guard KeyCondition received in #101814.
Related: https://github.com/ClickHouse/ClickHouse/issues/101814
Found by a query-invariance tester (a pristine table vs the same logical rows after a no-op mutation + `OPTIMIZE FINAL` diverged — the merge was the first thing to build statistics).
Contributor guide
Assessment
This issue has not been assessed yet.