Pass over udfs/physical expr that `strictly_order_preserving` and mark them as such
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
After this is merged:
- #23807
we now have a new property `strictly_order_preserving` that allow for more optimizations if the expression marked as keeping the same ordering
this property means that given expression `f` and 2 values from the input column `a` and `b` the following variants are kept:
1. `a.cmp(b) == f(a).cmp(f(b))`
2. nulls maps to nulls
Example of satisfying expression:
`cast(col_a as BIGINT)` where `col_a` is `INT` it is keeping the properties
Example of not satesfing:
`floor` - floor can not
`array_repeat(my_col, 2)` which might look like at first glance as keeping the property as well but in fact it does not.
the reason is that `array_repeat(null, 2)` will output list of 2 nulls which breaks the 2nd property that nulls must be kept as nulls
for more details on the property meaning and difference from `preserves_lex_ordering` see:
https://github.com/apache/datafusion/blob/1c3232ce2262523fb4043e7264185b2f8f929b0d/datafusion/expr-common/src/sort_properties.rs#L151-L198
Contributor guide
Research direction
Start with the definition and explanation of `strictly_order_preserving` in `datafusion/expr-common/src/sort_properties.rs` at the referenced lines, then locate the UDF and physical-expression handling described in the issue. Check which expressions satisfy both ordering and null-preservation rules; done means qualifying expressions are marked without incorrectly including cases such as `floor` or `array_repeat`.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100