ClickHouse / ClickHouse/ClickHouse
Column statistics
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
RFC and the first PR: https://github.com/ClickHouse/ClickHouse/pull/53240
This issue is for discussing what we will do in the future.
Use cases of column statistics:
- join reordering
- filter by decreasing selectivity in PREWHERE
- [automatic low-cardinality](https://github.com/ClickHouse/ClickHouse/issues/72489)
Related proposal: https://github.com/ClickHouse/ClickHouse/issues/64210
### Usability
- [x] cache of statistics
- [x] more grammar suger
- `ADD STATISTIC column_name TYPE ALL` to create all kinds of useful statistics as we can
- `DROP/CLEAR/MATERIALIZE STATISTIC column_name` to drop/clear/materialize all statistics if we omit `TYPE ...`
- [x] support more condition pattern for selectivity estimation
- [x] `a between 100 and 200` && `a > 100 && a < 200`
- [x] `a < 100 or a > 200`
- [x] system tables
- reveal statistics information in `system.parts` and `system.parts_columns`
- [x] support more data types
- support decimal type for tdigest
- [x] compact statistics files into single file per part
### functionality
- [x] support `hyperloglog`
- [x] then automatically decide if a string column can be stored as `low cardinality` format
- [x] support `cmsketch`
- to estimate `a = 1`
- [ ] support `equi-depth histograms` -> https://github.com/ClickHouse/clickhouse-private/issues/62912
- [ ] support heavy hitter (e.g. top 20 frequency) -> https://github.com/ClickHouse/clickhouse-private/issues/65801
- to estimate `a = 1` better
- [x] support min_max
- [ ] support sample: store a configurable number of values
- [x] support more counters
- NULL values
- Default values
- Deleted values for LWD
- [ ] estimate by combinator of above statistics
- e.g. `a = 1` will at first see if 1 is top 20 of column a.
- [ ] statistic for other tables / materialized views / projections ...
- [x] automatically create & maintain statistic
- [x] for cheap statistic like `hyperloglog` & `min_max` & `null_count`
- [ ] for frequently queried columns
- [ ] support statistics name aliases, e.g. `min_max` and `MinMax` statistics shall mean the same statistics type
Contributor guide
Assessment
This issue has not been assessed yet.