apache / apache/datafusion-comet

[Variant] Support native is_variant_null and Spark 4.2 is_valid_variant

Open
#5,429 1 comment 0 reactions 1 assignee Claimed by @peterxcli View on GitHub
area:expressions enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

Comet has no native predicate for distinguishing Variant JSON null from SQL NULL. Spark 4.0+ provides `is_variant_null`, and Spark 4.2 adds `is_valid_variant` for validating the Variant value/metadata pair.

```sql
SELECT is_variant_null(v) FROM t;
SELECT is_valid_variant(v) FROM t; -- Spark 4.2+
```

Spark's [`is_variant_null`](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L80-L119) returns true only for Variant null and false for SQL NULL and all other values; malformed physical data raises an error in its [evaluation helper](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionEvalUtils.scala#L58-L70). Spark 4.2's [`is_valid_variant`](https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L957-L987) returns true/false for a non-null Variant and SQL NULL for SQL NULL.

### Describe the potential solution

Add expression-specific serializers and native evaluators over the canonical `[value, metadata]` representation:

- implement Spark's exact JSON-null, SQL-NULL, malformed-value, and nullability behavior;
- validate both Variant value and metadata bytes for `is_valid_variant`;
- expose `is_valid_variant` only in the Spark 4.2 shim/profile while leaving Spark 4.0/4.1 compilation and registry behavior unchanged; and
- keep Variant rejected for unrelated predicates/operators.

Add focused parity and native-plan tests for every Variant scalar/container kind, Variant null, SQL NULL, malformed value bytes, malformed metadata bytes, and the Spark 4.2 version boundary.

### Additional context

Spark 4.2 registers `is_valid_variant` alongside the existing Variant functions in its [tagged function registry](https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala#L973-L983). It is not present in Spark 4.0/4.1.

Related work: #4295, #5407, #5424, and #5425.

Non-goals: comparisons, hash functions, parsing, casts, Variant mutation functions, C2R, shuffle/spill, writing, and Python transport.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.