[FEA] Allow SQL-like 3VL for segmented_reduce
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
I wish I could use libcudf's segmented_reduce without much pre/post-processing to implement SQL 3VL
Current implementation is "null begets null" with https://docs.rapids.ai/api/libcudf/stable/group__aggregation__reduction.html#gae36b126703c20e1836f5eb02adaa965d
>
null_handling | If INCLUDE, the reduction is valid if all elements in a segment are valid, otherwise null
-- | --
**Describe the solution you'd like**
In SQL 3VL, a valid result is returned when the result does not depend on the "unknowns" . E.g. given in Spark
```
>>> sql("select null or true").show()
+--------------+
|(NULL OR true)|
+--------------+
| true|
+--------------+
```
by extension one would expect `ANY` aggregation which is just OR-ing to be valid `true` if one of the segment values is 1 regardless of presence of NULLs.
Similar argument can be made for min/max aggregations if the segment contains a minimum/maximum value for the type such as INT_MIN/INT_MAX
**Describe alternatives you've considered**
Run multiple reductions https://github.com/NVIDIA/spark-rapids/blob/branch-22.06/sql-plugin/src/main/scala/com/nvidia/spark/rapids/higherOrderFunctions.scala#L371-L384 to implement 3VL
**Additional context**
[spark.sql.legacy.followThreeValuedLogicInArrayExists](https://spark.apache.org/docs/latest/sql-migration-guide.html)
Contributor guide
Assessment
This issue has not been assessed yet.