ClickHouse / ClickHouse/ClickHouse
Idea: stratified sampling
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
### Company or project name
ClickHouse
### Use case
For data visualization, instead of selecting a uniformly random subset of data, select a subset of the same size, but non-uniformly, so the resulting data maximizes the number of unique values in every column.
### Describe the solution you'd like
The new operator in the query pipeline should process and filter data in a stream, like the DISTINCT operator does, but do the following:
Collect sets of unique values, or the frequencies of values in every column encountered in the result so far. It can use approximate sets, such as counting bloom filters, count-min sketches.
For every subsequent record, estimate their uniqueness - the number of columns with values not seen before, or seen rarely, or not seen recently.
Whether the record passes the filter and added to the result is determined by the sample rate, but the probability of returning a record is adjusted for its uniqueness - more unique records will have a higher chance of being returned.
### Describe alternatives you've considered
_No response_
### Additional context
As a result, visualizations like this: https://adsb.exposed/?zoom=6&lat=27.7030&lng=-81.7163&query=0b75769d7eec8e82590be03a42b48d38 can work well even on lower sample rates.
Contributor guide
Assessment
This issue has not been assessed yet.