apache / apache/doris

[Enhancement](nereids) derive bare-column range predicate from monotonic function predicates for storage-layer pruning

Open
#66,293 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.

### Description

A predicate that wraps a column in a monotonic (or piecewise floor/ceil) function cannot be consumed by any storage-layer min/max statistics today, because those statistics are kept on the **bare column**, not on the wrapped expression. Examples:

- `date_trunc(dt_col, 'day') = '2026-03-26'`
- `year(dt_col) >= 2024`
- `*_ceil` / `*_floor` families
- `from_unixtime(epoch_col) >= '...'`
- `to_date(dt_col)`, `to_monday(dt_col)`
- binary-collation prefix: `substring(s_col, 1, 3) = 'abc'` / `left(s_col, 3) >= 'abc'`

As a result ORC/Parquet row-group pruning, OLAP zonemap pruning, and partition pruning all miss these predicates and scan far more data than necessary.

### Solution

Add a Nereids expression-rewrite rule that, for a predicate over a monotonic function of a column, derives an **additional** bare-column range predicate consumable by any min/max statistic, while always preserving the original predicate (the derived one is a one-way / relaxed implication, so the original still does the exact filtering and query results are unchanged). Gate it behind a session variable defaulting to on.

Soundness gates:
- **from_unixtime:** a DST/zone-offset guard — the epoch→local map is only strictly monotonic on a stretch with no UTC-offset transition, so the rule derives nothing when a transition falls in the guarded interval.
- **prefix functions:** emitted only under binary collation, where lexicographic byte order matches string order.
- **date_trunc / floor units:** the `[c, c + 1 unit)` bound is built from the exact trunc unit via the reused `*Add` scalar functions, so leap-year / month-end clamp reversibility is preserved.

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

The issue names no files or tests. Start by locating the Nereids expression-rewrite rule entry point and the session-variable definition, then trace how derived predicates reach storage and partition pruning. Done means the original predicate is preserved, safe bare-column ranges are added, the session setting defaults on, and the listed timezone, collation, and date-bound soundness gates hold.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
databases, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.