apache / apache/doris

[Enhancement] Selectivity-aware conjunct execution for ORC/Parquet readers (opt-in)

Open
#66,108 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/doris/issues) and found no similar issues.

### Description

The ORC and Parquet readers evaluate every filter conjunct over all rows of a block and AND the results together, regardless of how cheap/selective each predicate is or the order the planner emitted them. An expensive predicate (e.g. a string function like `split_by_string(col)[n]='x'`) is therefore run over the full block even when a cheap, highly-selective predicate on the same block would have filtered almost everything first.

### Proposal

Add three **opt-in (default off)** session variables that only change the order/method of filter execution, never which rows pass:

1. `enable_scan_conjunct_reorder` — static cost-based conjunct reorder (cheap predicates first).
2. `enable_scan_selective_filter` — selection-vector execution: a later predicate is evaluated only on the rows earlier ones let through, producing the identical full-width `result_filter`/`can_filter_all` contract.
3. `enable_scan_adaptive_reorder` — runtime reorder by measured cost/selectivity, correcting the static estimate for skewed data or unknown UDF cost.

All default **off**, so with the flags unset the reader path is byte-identical to today. When enabled, they change execution order/method only — results are identical.

### Use case

Faster ORC/Parquet scans on queries mixing cheap selective predicates with expensive ones (e.g. string/UDF predicates).

### Related issues

_No response_

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start by tracing the ORC and Parquet reader paths that evaluate filter conjuncts, then inspect how the three session variables are defined and consumed. Verify that each flag is default-off, preserves the full-width result_filter/can_filter_all contract, and returns identical rows while improving execution for selective and expensive predicates.

Written by the indexing model from the issue text.

Assessment

Domain
databases, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.