ClickHouse / ClickHouse/ClickHouse

Idea: stratified sampling

Open
#79,382 3 comments 0 reactions 0 assignees View on GitHub
comp-query-execution feature research
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

### Company or project name

ClickHouse

### Use case

For data visualization, instead of selecting a uniformly random subset of data, select a subset of the same size, but non-uniformly, so the resulting data maximizes the number of unique values in every column.

### Describe the solution you'd like

The new operator in the query pipeline should process and filter data in a stream, like the DISTINCT operator does, but do the following:

Collect sets of unique values, or the frequencies of values in every column encountered in the result so far. It can use approximate sets, such as counting bloom filters, count-min sketches.

For every subsequent record, estimate their uniqueness - the number of columns with values not seen before, or seen rarely, or not seen recently.

Whether the record passes the filter and added to the result is determined by the sample rate, but the probability of returning a record is adjusted for its uniqueness - more unique records will have a higher chance of being returned.

### Describe alternatives you've considered

_No response_

### Additional context

As a result, visualizations like this: https://adsb.exposed/?zoom=6&lat=27.7030&lng=-81.7163&query=0b75769d7eec8e82590be03a42b48d38 can work well even on lower sample rates.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.