[Enhancement](zonemap) Prune segments and row groups with column-vs-column comparisons
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Description
Zone map pruning currently only fires when a predicate compares a column against a constant. A
predicate that compares two columns of the same table, such as `WHERE a != b` or `WHERE a < b`,
never participates, so every segment and every Parquet row group is read even when the two columns' min/max
ranges make the predicate unsatisfiable.
The expression-level framework for this already exists. `ZoneMapEvalContext`
(`be/src/storage/index/zone_map/zonemap_eval_context.h`) is keyed per slot and holds an entry for
every slot a compound expression references, and the segment-level and Parquet row-group-level
callers already populate it that way:
- `be/src/storage/segment/segment.cpp:94` `build_segment_zonemap_context`
- `be/src/format/parquet/vparquet_reader.cpp:1651` `_process_expr_zonemap_filter`
- `be/src/format_v2/parquet/parquet_statistics.cpp:1149` `check_native_statistics`
Cross-column AND / OR compounds therefore already prune today: `a = 1 OR b = 2` and
`AND(a > 1, b < 2)` both work, because `VCompoundPred` passes the same multi-slot context down to
each child. What is missing is only the leaf case. The comparison operator's capability gate
(`be/src/exprs/function/functions_comparison.h:355`) accepts one slot plus one literal and nothing
else, so a two-slot comparison is dropped before evaluation at `segment.cpp:104`.
DuckDB added the equivalent feature in duckdb/duckdb#24805, with follow-ups #25087 (temporal, bool),
#25233 (string) and #25194 (monotone interval). This issue tracks the Doris side.
### Solution
Four PRs, risk increasing, each independently verifiable. The shared acceptance criterion is that a
given query returns row-for-row identical results with pruning enabled and disabled. The counters
only show that pruning fired; the comparison is what shows it fired correctly.
1. **Segment and row-group level.** Add a slot-vs-slot extractor and a separate capability
predicate, and evaluate the six comparison operators against the two columns' bounds. Lights up
the native segment path and both Parquet readers at once.
2. **Native segment page level.** Page zone maps are per column and their row boundaries do not line
up between two columns, so this needs the common refinement of the two page partitions rather
than a per-column prune-and-intersect.
3. **Parquet page index level**, same shape as 2.
4. *(optional)* **Monotone functions**, e.g. `date_trunc(a) < date_trunc(b)`. Needs derived
statistics, which Doris does not have as a general mechanism today.
PR 1 is both the first observable improvement and the point where we decide whether to continue: PRs
2 and 3 refine the same interval test to a finer granularity and do not change whether two columns'
ranges can be separated at all. If PR 1 shows no benefit on a real workload, PRs 2 and 3 almost
certainly will not either. So the sequencing is: land PR 1, measure, then decide.
The expected benefit is narrow. `<` and `>` prune only
when the two ranges are fully separated; `!=` only when both columns collapse to the same single
value within the zone. TPC-H Q12 and Q4 contain the ideal shape, `l_commitdate < l_receiptdate` and
`l_shipdate < l_commitdate`: three DATE columns of one table with no cast in between. But dbgen
generates those dates as tightly correlated offsets, so their per-segment ranges overlap heavily and
little or nothing will be pruned. Q12 is therefore a regression baseline, not a benefit
demonstration. The shapes that do pay are column pairs whose ranges separate naturally: a
batch-increasing column against a fixed-range one, or an "expected vs actual" pair that is equal in
most load batches.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
---
Contributor guide
Research direction
Start with ZoneMapEvalContext in be/src/storage/index/zone_map/zonemap_eval_context.h and the capability gate in be/src/exprs/function/functions_comparison.h:355. Trace how contexts are built in be/src/storage/segment/segment.cpp:94 and the Parquet callers at vparquet_reader.cpp:1651 and parquet_statistics.cpp:1149. The first milestone is segment and row-group pruning for slot-vs-slot comparisons, verified by identical results with pruning enabled and disabled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, sql
- Domain
- databases, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100