ClickHouse / ClickHouse/ClickHouse

Specialize sparse and low cardinality behaviour for `equals` and `notEquals`

Open
#81,039 1 comment 2 reactions 0 assignees View on GitHub
comp-regular-function performance
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

### Company or project name

CH

### Describe the situation

When processing data in filters with a sparse column, converting to and from full columns is quite expensive. This is from a query executing `!=` over a sparse column:

```
- 11.33% DB::FilterTransform::doTransform(DB::Chunk&) ▒
- 8.62% DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const ▒
- 4.99% DB::IExecutableFunction::execute(std::__1::vector > const&, std::__1::shared_ptr const&, unsigned long, bool) const ▒
- 2.67% DB::IExecutableFunction::executeWithoutSparseColumns(std::__1::vector > const&, std::__1::shared_ptr const&, unsigned long, bool) const ▒
- 1.48% DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::__1::vector > const&, std::__1::shared_ptr const&, unsigned long, bool) const ▒
- 1.22% DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::__1::vector > const&, std::__1::shared_ptr const&, unsigned long) const ▒
- 1.09% DB::FunctionComparison::executeImpl(std::__1::vector > const&, std::__1::shared_ptr const&, unsigned long) const ▒
0.51% COW::immutable_ptr DB::FunctionComparison::executeNumLeftType(DB::IColumn const*, DB::IColumn const*) const ▒
0.54% DB::(anonymous namespace)::convertLowCardinalityColumnsToFull(std::__1::vector >&) ▒
0.64% DB::IColumnHelper, DB::ColumnFixedSizeHelper>::getNumberOfDefaultRows() const ▒
0.72% std::__1::vector >::__destroy_vector::operator()[abi:ne190107]() ▒
0.62% DB::Block::insert(DB::ColumnWithTypeAndName) ▒
0.53% DB::ColumnSparse::~ColumnSparse() ▒
```

This is the default behaviour for most functions because it doesn't make sense to specialize each function to work with sparse / low cardinality columns, as the complexity of the code would be too much.

On the other hand, there are some functions where making use of the dictionaries directly would make sense, such as `equals` / `=` or `notEquals`.

### Which ClickHouse versions are affected?

All

### How to reproduce

Clickbench Q1

### Expected performance

Hopefully it should be faster. If not, don't merge the change.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.