[Enhancement](zonemap) Support column-vs-column comparisons in expression zone map pruning
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 522
Description
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Description
First step of #67771. Today a comparison between two columns of the same table is
rejected by the comparison operator's zone map capability gate, so it never reaches evaluation:
`comparison_zonemap_detail::can_evaluate` (`be/src/exprs/function/functions_comparison.h:355`) calls
`extract_slot_and_literal`, which requires exactly one `VSlotRef` and one literal.
Nothing else on the path needs changing. FE already ships every conjunct verbatim through
`TPlanNode.conjuncts` (`fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java:487`), and BE
already routes a predicate it cannot turn into a `ColumnPredicate` into
`_common_expr_ctxs_push_down`, which is the vector the zone map evaluator consumes.
### Solution
Add a slot-vs-slot shape alongside the existing slot-vs-literal one, and give the six always-false tests a
single implementation shared by both.
**Interval algebra.** With the left operand's non-null values in `[lmin, lmax]` and the right
operand's in `[rmin, rmax]`, the comparison cannot be TRUE for any row when:
| operator | condition |
| --- | --- |
| `=` | `lmin > rmax \|\| rmin > lmax` |
| `!=` | `lmin == lmax && rmin == rmax && lmin == rmin` |
| `<` | `lmin >= rmax` |
| `<=` | `lmin > rmax` |
| `>` | `rmin >= lmax` |
| `>=` | `rmin > lmax` |
Substituting the degenerate range `[literal, literal]` for the right operand reduces each row to the
slot-vs-literal test that is in the tree today, so both shapes share one implementation and the
single-column path keeps its exact current behaviour. The existing tests passing unchanged is the
mechanical check on that claim.
**Why NULL does not weaken these rules.** `kNoMatch` means "no row in this zone can make the
conjunct TRUE", not "every row is FALSE", because a WHERE conjunct discards both FALSE and NULL
rows. Rows where either side is NULL yield NULL and so are not matches; rows where both sides are
non-null are bounded by the two ranges, because zone map min/max summarize exactly the non-null
values. The test only has to hold on the second group, which is the group the bounds cover. A column
with no non-null value at all makes the comparison NULL on every row, so it returns `kNoMatch`
unconditionally.
**Guards, in this order.** Both slots have a compatible type in the context; both have a zone map;
either side has no non-null value → `kNoMatch`; both pass `range_stats_usable_for_zonemap`; neither
side has an unknown floating NaN count. The third and fourth cannot be swapped: an all-null zone map
leaves min/max default-constructed as `TYPE_NULL`, which the range check would fatal on.
One trap for reviewers: `can_evaluate` must not be widened. It is shared by
dictionary filtering (`can_evaluate_dictionary_filter`) and by `can_evaluate_equality`, and both
dereference `extract_slot_and_literal` behind a `DORIS_CHECK`, so accepting a slot-vs-slot expression
there would abort. This PR adds a separate `can_evaluate_slot_slot` and ORs it into
`can_evaluate_zonemap_filter` only. A unit test pins this down by asserting that a two-slot
comparison is accepted by the zone map gate and rejected by the dictionary and bloom gates.
Two deliberate limitations: a cast around either operand is rejected, because the bounds are raw
stored values and the optimizer inserts a cast whenever the two column types differ; and on Parquet
a column pair of floating type does not prune at all, because Parquet bounds omit NaN without
reporting how many were skipped, which would flip the `!=` rule.
This work also uncovered a bug on the v1 reader, fixed in the same PR. The flag that carries "these
bounds came from Parquet, so a hidden NaN is possible" was only ever set in the format_v2 reader. The
v1 reader assigned the slot's data type and left the flag at its default, so the NaN guard never
fired on that path. Two DOUBLE
columns whose Parquet bounds both read `[1.0, 1.0]` would then be pruned by the `!=` rule even when
one of them hid a NaN, dropping a row that should have come back. The fix moves the rule into a
single setter on `SlotZoneMap` that assigns the type and the flag together, and routes all three
Parquet call sites through it. The root cause was one rule written in two places, so the remedy is
to leave it in one.
That fix has a cost: on the v1 reader, single-column float and double predicates using `>`, `>=`,
`!=` against a non-NaN literal, or `=` against NaN, no longer prune. That trades pruning for
correctness, and it was already the behaviour on format_v2. No existing test covers it either way.
The two tests named for it, `test_expr_push_down_gt_float` and `test_expr_push_down_ge_double` in
`be/test/format/parquet/parquet_expr_test.cpp`, build a statistics callback and an expression context
and then assert only that the predicate is not constant, so the reader is never invoked. Reviving
them needs a reader fixture, so that is filed separately rather than folded in here.
**Testing.** Unit tests cover the six operators against an interval matrix, both fully separated and
partially overlapping, plus the guard matrix: slot missing from the context, zone map absent,
incompatible types, `pass_all`, an all-null column on either side, the same slot on both sides, and
an unknown floating NaN count. The regression suite
(`regression-test/suites/query_p0/expr_zonemap/test_expr_zonemap_pruning.groovy`) adds three loads of
one segment each holding a constant in both columns, which is the shape that forces a per-segment
decision: `a != b` must drop exactly two of the three segments and `a = b` exactly one. Every query
there runs with `enable_expr_zonemap_filter` on and off and the row counts are compared, since the
counter only shows that pruning fired.
That suite passes on a one-FE one-BE cluster: `Test 1 suites, failed 0 suites`.
One gap remains: the row-group level on both Parquet readers has unit coverage of the rule but no
end-to-end case, because building Parquet input in the regression suite means the
external-table Hive setup rather than an OLAP table.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
---
Contributor guide
Research direction
Start in be/src/exprs/function/functions_comparison.h at comparison_zonemap_detail::can_evaluate and review the related zone-map tests and Parquet SlotZoneMap call sites. Run the interval and guard unit tests plus regression-test/suites/query_p0/expr_zonemap/test_expr_zonemap_pruning.groovy; done means slot-vs-slot pruning matches the existing slot-vs-literal behavior without widening dictionary or bloom gates.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, sql
- Domain
- backend, databases, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 42/100