apache / apache/arrow

[C++] Investigate reporting filter selectivity for filter order optimization

Open
#30,995 0 comments 0 reactions 0 assignees View on GitHub
Component: C++ Type: task
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

If we have highly selective filters we can reduce the total amount of work we need to do if we run those filters first. For example, in TPC-H query 6 we are either casting decimals to doubles and running comparisons or we are performing somewhat costly double comparisons. At the same time we have a highly selective date filter. If we can evaluate that date filter first then we can reduce the amount of decimal arithmetic we need to do.

It's a little unclear to me at the moment how much of this is something we want to back into the execution engine versus how much of this should be getting provided to us by some kind of intelligent runtime optimizing planner.

**Reporter**: [Weston Pace](https://issues.apache.org/jira/browse/ARROW-15522) / @westonpace

**Note**: *This issue was originally created as [ARROW-15522](https://issues.apache.org/jira/browse/ARROW-15522). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start with TPC-H query 6 and investigate how filter evaluation order affects the decimal and double comparisons versus the selective date filter. Determine whether filter ordering belongs in the execution engine or an intelligent runtime-optimizing planner, and quantify the resulting reduction in work.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.