apache / apache/datafusion-comet
Fix nullability report for spark expression
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### Describe the bug
As https://github.com/apache/datafusion/issues/19144 indicates, most of DF's Spark expression do not report the nullability correctly.
Spark 4.0 has a `nullIntolerant` method defined in [Expression](https://github.com/apache/spark/blob/branch-4.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala#L143-L148) and the comment is
```Scala
/**
* When an expression inherits this, meaning the expression is null intolerant (i.e. any null
* input will result in null output). We will use this information during constructing IsNotNull
* constraints.
*/
```
I think Comet should have similar fix.
### Steps to reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.