apache / apache/datafusion

Unexpected `Null` behavior for `ScalarValue::partial_cmp()`

Open
#19,579 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

For `partial_cmp()` function for `ScalarValue`

https://github.com/apache/datafusion/blob/79f67b8ef2d3016c866ae92d7d2f56d1a7830045/datafusion/common/src/scalar/mod.rs#L568

The current behavior is unexpected:

```rust
// this test can be added to the above file that `partial_cmp()` lives
#[test]
fn scalar_partial_ordering_nulls() {
use ScalarValue::*;

assert_eq!(
Int32(Some(3)).partial_cmp(&Int32(None)),
Some(Ordering::Greater)
);
assert_eq!(
Int32(None).partial_cmp(&Int32(Some(3))),
Some(Ordering::Less)
);
}
```
The comparison between `ScalarValue { type: Int32, val: 3 }` and `ScalarValue { type: Int32, val: Null }` returns `Some(_)`, whereas I expected `None`, indicating that the values are not comparable, consistent with SQL’s three-valued logic.

Additionally, comparing `Null` with `Null` returns `Some(Ordering::Equal)` instead of `None`.

We should either:
1. Fix this behavior to follow SQL’s null comparison semantics, or
2. If this non-standard null behavior is intentional for specific reasons, explicitly document it in the comments.

### To Reproduce

_No response_

### Expected behavior

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start in datafusion/common/src/scalar/mod.rs at the partial_cmp implementation referenced in the issue, then review the surrounding tests and comments. Add the proposed scalar_partial_ordering_nulls test to observe the current behavior, and confirm whether the intended result is SQL-style incomparability or documented existing behavior before making the issue's outcome explicit.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.