apache / apache/datafusion-comet
[Variant] Support native is_variant_null and Spark 4.2 is_valid_variant
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
Comet has no native predicate for distinguishing Variant JSON null from SQL NULL. Spark 4.0+ provides `is_variant_null`, and Spark 4.2 adds `is_valid_variant` for validating the Variant value/metadata pair.
```sql
SELECT is_variant_null(v) FROM t;
SELECT is_valid_variant(v) FROM t; -- Spark 4.2+
```
Spark's [`is_variant_null`](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L80-L119) returns true only for Variant null and false for SQL NULL and all other values; malformed physical data raises an error in its [evaluation helper](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionEvalUtils.scala#L58-L70). Spark 4.2's [`is_valid_variant`](https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L957-L987) returns true/false for a non-null Variant and SQL NULL for SQL NULL.
### Describe the potential solution
Add expression-specific serializers and native evaluators over the canonical `[value, metadata]` representation:
- implement Spark's exact JSON-null, SQL-NULL, malformed-value, and nullability behavior;
- validate both Variant value and metadata bytes for `is_valid_variant`;
- expose `is_valid_variant` only in the Spark 4.2 shim/profile while leaving Spark 4.0/4.1 compilation and registry behavior unchanged; and
- keep Variant rejected for unrelated predicates/operators.
Add focused parity and native-plan tests for every Variant scalar/container kind, Variant null, SQL NULL, malformed value bytes, malformed metadata bytes, and the Spark 4.2 version boundary.
### Additional context
Spark 4.2 registers `is_valid_variant` alongside the existing Variant functions in its [tagged function registry](https://github.com/apache/spark/blob/v4.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala#L973-L983). It is not present in Spark 4.0/4.1.
Related work: #4295, #5407, #5424, and #5425.
Non-goals: comparisons, hash functions, parsing, casts, Variant mutation functions, C2R, shuffle/spill, writing, and Python transport.
Contributor guide
Assessment
This issue has not been assessed yet.