apache / apache/datafusion-comet

Fix nullability report for spark expression

Open
#3,362 1 comment 0 reactions 0 assignees View on GitHub
area:expressions bug priority:medium
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### Describe the bug

As https://github.com/apache/datafusion/issues/19144 indicates, most of DF's Spark expression do not report the nullability correctly.

Spark 4.0 has a `nullIntolerant` method defined in [Expression](https://github.com/apache/spark/blob/branch-4.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala#L143-L148) and the comment is

```Scala
/**
* When an expression inherits this, meaning the expression is null intolerant (i.e. any null
* input will result in null output). We will use this information during constructing IsNotNull
* constraints.
*/
```

I think Comet should have similar fix.

### Steps to reproduce

_No response_

### Expected behavior

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.