apache / apache/doris

[Enhancement] Evaluate deterministic single-slot string expression predicates on the ORC/Parquet column dictionary

Open
#66,183 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.

### Description

Today ORC/Parquet dictionary filtering only applies when a filter conjunct is a bare column reference compared to constants. A predicate whose value side is a deterministic single-slot string expression — e.g. `split_by_string(col, sep)[n] = 'x'`, `element_at(...)`, `substring_index(...)`, `regexp_extract(...)` — is evaluated once per row even though it depends only on the column value.

Such predicates can instead be evaluated once per distinct dictionary value. The FE (Nereids) is the authoritative admission check: it stamps a conjunct root as dict-filterable when the predicate is an equality/IN over a single slot, is deterministic, and preserves NULL semantics (null-in implies null-out — the dictionary carries no NULL entry, so a NULL source must map to a NULL result to match per-row semantics). The verdict is carried to BE via `TExprNode.can_dict_filter`; when unset (older FE) BE falls back to a conservative allowlist. A session variable `enable_dict_filter_for_expr` (default true, forwarded as a query option) gates the expression case.

### Use case

Scans that filter on a heavy string expression of a low-cardinality dictionary-encoded ORC/Parquet column evaluate the expression per row today; evaluating per distinct dictionary value is a large reduction for low-NDV columns.

### Related issues

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start by tracing dictionary filtering from the FE (Nereids) admission check through TExprNode.can_dict_filter into the BE, and inspect how enable_dict_filter_for_expr is forwarded as a query option. Compare the existing bare-column path with the conservative older-FE fallback, then verify deterministic single-slot string expressions and NULL behavior. Done means eligible equality/IN predicates are evaluated per distinct dictionary value without changing per-row semantics.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
backend, databases, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.