NVIDIA / NVIDIA/cudf

[FEA] Allow SQL-like 3VL for segmented_reduce

Open
#10,977 2 comments 0 reactions 0 assignees View on GitHub
feature request libcudf Spark
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**
I wish I could use libcudf's segmented_reduce without much pre/post-processing to implement SQL 3VL

Current implementation is "null begets null" with https://docs.rapids.ai/api/libcudf/stable/group__aggregation__reduction.html#gae36b126703c20e1836f5eb02adaa965d
>

null_handling | If INCLUDE, the reduction is valid if all elements in a segment are valid, otherwise null
-- | --

**Describe the solution you'd like**
In SQL 3VL, a valid result is returned when the result does not depend on the "unknowns" . E.g. given in Spark
```
>>> sql("select null or true").show()
+--------------+
|(NULL OR true)|
+--------------+
| true|
+--------------+
```
by extension one would expect `ANY` aggregation which is just OR-ing to be valid `true` if one of the segment values is 1 regardless of presence of NULLs.

Similar argument can be made for min/max aggregations if the segment contains a minimum/maximum value for the type such as INT_MIN/INT_MAX

**Describe alternatives you've considered**
Run multiple reductions https://github.com/NVIDIA/spark-rapids/blob/branch-22.06/sql-plugin/src/main/scala/com/nvidia/spark/rapids/higherOrderFunctions.scala#L371-L384 to implement 3VL

**Additional context**
[spark.sql.legacy.followThreeValuedLogicInArrayExists](https://spark.apache.org/docs/latest/sql-migration-guide.html)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.