apache / apache/datafusion

Pass over udfs/physical expr that `strictly_order_preserving` and mark them as such

Closed
#23,920 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

After this is merged:
- #23807

we now have a new property `strictly_order_preserving` that allow for more optimizations if the expression marked as keeping the same ordering

this property means that given expression `f` and 2 values from the input column `a` and `b` the following variants are kept:
1. `a.cmp(b) == f(a).cmp(f(b))`
2. nulls maps to nulls

Example of satisfying expression:
`cast(col_a as BIGINT)` where `col_a` is `INT` it is keeping the properties

Example of not satesfing:

`floor` - floor can not

`array_repeat(my_col, 2)` which might look like at first glance as keeping the property as well but in fact it does not.

the reason is that `array_repeat(null, 2)` will output list of 2 nulls which breaks the 2nd property that nulls must be kept as nulls

for more details on the property meaning and difference from `preserves_lex_ordering` see:
https://github.com/apache/datafusion/blob/1c3232ce2262523fb4043e7264185b2f8f929b0d/datafusion/expr-common/src/sort_properties.rs#L151-L198

Contributor guide

Open the contributing guide

Research direction

Start with the definition and explanation of `strictly_order_preserving` in `datafusion/expr-common/src/sort_properties.rs` at the referenced lines, then locate the UDF and physical-expression handling described in the issue. Check which expressions satisfy both ordering and null-preservation rules; done means qualifying expressions are marked without incorrectly including cases such as `floor` or `array_repeat`.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.