ClickHouse / ClickHouse/ClickHouse

Column statistics

Open
#55,065 4 comments 6 reactions 0 assignees View on GitHub
comp-mergetree feature
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

RFC and the first PR: https://github.com/ClickHouse/ClickHouse/pull/53240

This issue is for discussing what we will do in the future.

Use cases of column statistics:
- join reordering
- filter by decreasing selectivity in PREWHERE
- [automatic low-cardinality](https://github.com/ClickHouse/ClickHouse/issues/72489)

Related proposal: https://github.com/ClickHouse/ClickHouse/issues/64210

### Usability

- [x] cache of statistics
- [x] more grammar suger
- `ADD STATISTIC column_name TYPE ALL` to create all kinds of useful statistics as we can
- `DROP/CLEAR/MATERIALIZE STATISTIC column_name` to drop/clear/materialize all statistics if we omit `TYPE ...`
- [x] support more condition pattern for selectivity estimation
- [x] `a between 100 and 200` && `a > 100 && a < 200`
- [x] `a < 100 or a > 200`
- [x] system tables
- reveal statistics information in `system.parts` and `system.parts_columns`
- [x] support more data types
- support decimal type for tdigest
- [x] compact statistics files into single file per part

### functionality
- [x] support `hyperloglog`
- [x] then automatically decide if a string column can be stored as `low cardinality` format
- [x] support `cmsketch`
- to estimate `a = 1`
- [ ] support `equi-depth histograms` -> https://github.com/ClickHouse/clickhouse-private/issues/62912
- [ ] support heavy hitter (e.g. top 20 frequency) -> https://github.com/ClickHouse/clickhouse-private/issues/65801
- to estimate `a = 1` better
- [x] support min_max
- [ ] support sample: store a configurable number of values
- [x] support more counters
- NULL values
- Default values
- Deleted values for LWD
- [ ] estimate by combinator of above statistics
- e.g. `a = 1` will at first see if 1 is top 20 of column a.
- [ ] statistic for other tables / materialized views / projections ...
- [x] automatically create & maintain statistic
- [x] for cheap statistic like `hyperloglog` & `min_max` & `null_count`
- [ ] for frequently queried columns
- [ ] support statistics name aliases, e.g. `min_max` and `MinMax` statistics shall mean the same statistics type

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.