ClickHouse / ClickHouse/ClickHouse
Specialize sparse and low cardinality behaviour for `equals` and `notEquals`
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
### Company or project name
CH
### Describe the situation
When processing data in filters with a sparse column, converting to and from full columns is quite expensive. This is from a query executing `!=` over a sparse column:
```
- 11.33% DB::FilterTransform::doTransform(DB::Chunk&) ▒
- 8.62% DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const ▒
- 4.99% DB::IExecutableFunction::execute(std::__1::vector > const&, std::__1::shared_ptr const&, unsigned long, bool) const ▒
- 2.67% DB::IExecutableFunction::executeWithoutSparseColumns(std::__1::vector > const&, std::__1::shared_ptr const&, unsigned long, bool) const ▒
- 1.48% DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::__1::vector > const&, std::__1::shared_ptr const&, unsigned long, bool) const ▒
- 1.22% DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::__1::vector > const&, std::__1::shared_ptr const&, unsigned long) const ▒
- 1.09% DB::FunctionComparison::executeImpl(std::__1::vector > const&, std::__1::shared_ptr const&, unsigned long) const ▒
0.51% COW::immutable_ptr DB::FunctionComparison::executeNumLeftType(DB::IColumn const*, DB::IColumn const*) const ▒
0.54% DB::(anonymous namespace)::convertLowCardinalityColumnsToFull(std::__1::vector >&) ▒
0.64% DB::IColumnHelper, DB::ColumnFixedSizeHelper>::getNumberOfDefaultRows() const ▒
0.72% std::__1::vector >::__destroy_vector::operator()[abi:ne190107]() ▒
0.62% DB::Block::insert(DB::ColumnWithTypeAndName) ▒
0.53% DB::ColumnSparse::~ColumnSparse() ▒
```
This is the default behaviour for most functions because it doesn't make sense to specialize each function to work with sparse / low cardinality columns, as the complexity of the code would be too much.
On the other hand, there are some functions where making use of the dictionaries directly would make sense, such as `equals` / `=` or `notEquals`.
### Which ClickHouse versions are affected?
All
### How to reproduce
Clickbench Q1
### Expected performance
Hopefully it should be faster. If not, don't merge the change.
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.