[FEA] Support segmented_sorted_order using an AST function for comparison or something similar
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
Spark supports an `array_sort` SQL method that will sort arrays, but it takes a lambda function that calculates and returns the comparison between the elements of the array instead of going off of a config for doing the ordering.
The docs for this are at https://spark.apache.org/docs/latest/api/sql/index.html#array_sort
```
The comparator will take two arguments representing two elements of the array. It returns -1, 0, or 1 as the first element is less than, equal to, or greater than the second element. If the comparator function returns other values (including null), the function will fail and raise an error.
```
**Describe the solution you'd like**
I would love a segmented_sorted_order that takes an AST function similar to an inequality join and follows the rules described above. I am fine if it is more lenient than Spark requires and does not throw exceptions if the values do not fully conform to exactly what is requested here.
We may also need some extensions to what the AST can do. Because of the tight rules that Spark has the implementations often use case/when and if/else expressions. We probably need to file a separate issue for this, but I wanted to call it out here because without it, there isn't a lot that we can do.
**Describe alternatives you've considered**
We don't really have any other alternative. We could do some code analysis of the function and see if it matches some known patterns for ascending/descending and nulls preceding or following. But this is very limiting, and we would prefer to do it as a performance improvement instead.
Contributor guide
Assessment
This issue has not been assessed yet.