[C++] Investigate reporting filter selectivity for filter order optimization
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
If we have highly selective filters we can reduce the total amount of work we need to do if we run those filters first. For example, in TPC-H query 6 we are either casting decimals to doubles and running comparisons or we are performing somewhat costly double comparisons. At the same time we have a highly selective date filter. If we can evaluate that date filter first then we can reduce the amount of decimal arithmetic we need to do.
It's a little unclear to me at the moment how much of this is something we want to back into the execution engine versus how much of this should be getting provided to us by some kind of intelligent runtime optimizing planner.
**Reporter**: [Weston Pace](https://issues.apache.org/jira/browse/ARROW-15522) / @westonpace
**Note**: *This issue was originally created as [ARROW-15522](https://issues.apache.org/jira/browse/ARROW-15522). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start with TPC-H query 6 and investigate how filter evaluation order affects the decimal and double comparisons versus the selective date filter. Determine whether filter ordering belongs in the execution engine or an intelligent runtime-optimizing planner, and quantify the resulting reduction in work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100