[Enhancement] Evaluate deterministic single-slot string expression predicates on the ORC/Parquet column dictionary
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Description
Today ORC/Parquet dictionary filtering only applies when a filter conjunct is a bare column reference compared to constants. A predicate whose value side is a deterministic single-slot string expression — e.g. `split_by_string(col, sep)[n] = 'x'`, `element_at(...)`, `substring_index(...)`, `regexp_extract(...)` — is evaluated once per row even though it depends only on the column value.
Such predicates can instead be evaluated once per distinct dictionary value. The FE (Nereids) is the authoritative admission check: it stamps a conjunct root as dict-filterable when the predicate is an equality/IN over a single slot, is deterministic, and preserves NULL semantics (null-in implies null-out — the dictionary carries no NULL entry, so a NULL source must map to a NULL result to match per-row semantics). The verdict is carried to BE via `TExprNode.can_dict_filter`; when unset (older FE) BE falls back to a conservative allowlist. A session variable `enable_dict_filter_for_expr` (default true, forwarded as a query option) gates the expression case.
### Use case
Scans that filter on a heavy string expression of a low-cardinality dictionary-encoded ORC/Parquet column evaluate the expression per row today; evaluating per distinct dictionary value is a large reduction for low-NDV columns.
### Related issues
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Start by tracing dictionary filtering from the FE (Nereids) admission check through TExprNode.can_dict_filter into the BE, and inspect how enable_dict_filter_for_expr is forwarded as a query option. Compare the existing bare-column path with the conservative older-FE fallback, then verify deterministic single-slot string expressions and NULL behavior. Done means eligible equality/IN predicates are evaluated per distinct dictionary value without changing per-row semantics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- backend, databases, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100