apache / apache/datafusion-comet

[Variant] Support VariantType in native columnar-to-row conversion

Open
#5,436 1 comment 0 reactions 1 assignee Claimed by @peterxcli View on GitHub
area:ffi enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

Comet keeps native columnar-to-row conversion on Spark fallback when a schema contains `VariantType`. This prevents a native scan or projection that carries Variant from feeding row-based Spark operators through Comet's native C2R path.

Spark does not encode Variant in `UnsafeRow` as an ordinary nested Struct. [`VariantVal`](https://github.com/apache/spark/blob/v4.1.3/common/unsafe/src/main/java/org/apache/spark/unsafe/types/VariantVal.java#L28-L46) defines a dedicated payload, and [`UnsafeWriter.write(VariantVal)`](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/codegen/UnsafeWriter.java#L164-L182) writes `4-byte value length + value bytes + metadata bytes`.

Comet's current [`supportsSchema`](https://github.com/peterxcli/datafusion-comet/blob/c355fefd9c0b7e96d86523a7214bb2cdd47e1a55/spark/src/main/scala/org/apache/spark/sql/comet/CometNativeColumnarToRowExec.scala#L230-L259) does not admit Variant, while the native [C2R initialization path](https://github.com/peterxcli/datafusion-comet/blob/c355fefd9c0b7e96d86523a7214bb2cdd47e1a55/native/core/src/execution/jni_api.rs#L1018-L1034) currently derives only the physical Arrow datatype. Treating the `[value, metadata]` storage as a normal Struct would produce the wrong UnsafeRow representation.

### Describe the potential solution

- Preserve explicit Variant identity when initializing native C2R.
- Add a top-level Variant writer that consumes Comet's canonical `[value: Binary, metadata: Binary]` array and emits Spark's exact UnsafeRow Variant payload.
- Preserve the distinction between SQL NULL and Variant JSON null, and validate malformed child/null combinations rather than panicking.
- Keep Variant out of generic Struct handling.

Focused parity tests should cover objects, arrays/scalars, Variant JSON null, SQL NULL, nullable parents, empty batches, and fields before/after Variant. The resulting row should expose Spark `VariantType` and round-trip through `UnsafeRow.getVariant` with the same value and metadata bytes.

### Additional context

Related work: #5407 establishes the native columnar `[value, metadata]` representation; #5425 tracks Variant-valued native expression output. This issue is only C2R transport.

Nested Variant, shuffle/spill, Python transport, writes, and general Variant expressions remain separate. Spark 3.x behavior must remain unchanged.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.