apache / apache/doris

[Enhancement](zonemap) Prune segments and row groups with column-vs-column comparisons

Open
#67,771 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.

### Description

Zone map pruning currently only fires when a predicate compares a column against a constant. A
predicate that compares two columns of the same table, such as `WHERE a != b` or `WHERE a < b`,
never participates, so every segment and every Parquet row group is read even when the two columns' min/max
ranges make the predicate unsatisfiable.

The expression-level framework for this already exists. `ZoneMapEvalContext`
(`be/src/storage/index/zone_map/zonemap_eval_context.h`) is keyed per slot and holds an entry for
every slot a compound expression references, and the segment-level and Parquet row-group-level
callers already populate it that way:

- `be/src/storage/segment/segment.cpp:94` `build_segment_zonemap_context`
- `be/src/format/parquet/vparquet_reader.cpp:1651` `_process_expr_zonemap_filter`
- `be/src/format_v2/parquet/parquet_statistics.cpp:1149` `check_native_statistics`

Cross-column AND / OR compounds therefore already prune today: `a = 1 OR b = 2` and
`AND(a > 1, b < 2)` both work, because `VCompoundPred` passes the same multi-slot context down to
each child. What is missing is only the leaf case. The comparison operator's capability gate
(`be/src/exprs/function/functions_comparison.h:355`) accepts one slot plus one literal and nothing
else, so a two-slot comparison is dropped before evaluation at `segment.cpp:104`.

DuckDB added the equivalent feature in duckdb/duckdb#24805, with follow-ups #25087 (temporal, bool),
#25233 (string) and #25194 (monotone interval). This issue tracks the Doris side.

### Solution

Four PRs, risk increasing, each independently verifiable. The shared acceptance criterion is that a
given query returns row-for-row identical results with pruning enabled and disabled. The counters
only show that pruning fired; the comparison is what shows it fired correctly.

1. **Segment and row-group level.** Add a slot-vs-slot extractor and a separate capability
predicate, and evaluate the six comparison operators against the two columns' bounds. Lights up
the native segment path and both Parquet readers at once.
2. **Native segment page level.** Page zone maps are per column and their row boundaries do not line
up between two columns, so this needs the common refinement of the two page partitions rather
than a per-column prune-and-intersect.
3. **Parquet page index level**, same shape as 2.
4. *(optional)* **Monotone functions**, e.g. `date_trunc(a) < date_trunc(b)`. Needs derived
statistics, which Doris does not have as a general mechanism today.

PR 1 is both the first observable improvement and the point where we decide whether to continue: PRs
2 and 3 refine the same interval test to a finer granularity and do not change whether two columns'
ranges can be separated at all. If PR 1 shows no benefit on a real workload, PRs 2 and 3 almost
certainly will not either. So the sequencing is: land PR 1, measure, then decide.

The expected benefit is narrow. `<` and `>` prune only
when the two ranges are fully separated; `!=` only when both columns collapse to the same single
value within the zone. TPC-H Q12 and Q4 contain the ideal shape, `l_commitdate < l_receiptdate` and
`l_shipdate < l_commitdate`: three DATE columns of one table with no cast in between. But dbgen
generates those dates as tightly correlated offsets, so their per-segment ranges overlap heavily and
little or nothing will be pruned. Q12 is therefore a regression baseline, not a benefit
demonstration. The shapes that do pay are column pairs whose ranges separate naturally: a
batch-increasing column against a fixed-range one, or an "expected vs actual" pair that is equal in
most load batches.

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

---

Contributor guide

Open the contributing guide

Research direction

Start with ZoneMapEvalContext in be/src/storage/index/zone_map/zonemap_eval_context.h and the capability gate in be/src/exprs/function/functions_comparison.h:355. Trace how contexts are built in be/src/storage/segment/segment.cpp:94 and the Parquet callers at vparquet_reader.cpp:1651 and parquet_statistics.cpp:1149. The first milestone is segment and row-group pruning for slot-vs-slot comparisons, verified by identical results with pruning enabled and disabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, sql
Domain
databases, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.