apache / apache/datafusion-comet

[Variant] Support native `variant_explode` and `variant_explode_outer`

Open
#5,432 1 comment 0 reactions 1 assignee Claimed by @peterxcli View on GitHub
area:expressions enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

Spark 4.x provides `variant_explode` and `variant_explode_outer` to turn a Variant object or array into rows. Comet does not support these generators, so queries must leave native execution:

```sql
SELECT id, e.* FROM t LATERAL VIEW variant_explode(v) e AS pos, key, value;
SELECT id, e.* FROM t LATERAL VIEW variant_explode_outer(v) e AS pos, key, value;
```

Spark defines the generated columns as `pos INT NOT NULL`, `key STRING`, and `value VARIANT NOT NULL`, and implements object/array iteration in [`VariantExplode`](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L664-L775). The exact SQL names are registered as generators in Spark's [4.1.3 function registry](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala#L1233-L1240).

### Describe the potential solution

Add a Spark-4-specific generator path that:

- iterates arrays by index and objects by their stored field order;
- emits Spark-compatible `pos`, nullable `key`, and Variant `value` columns;
- preserves the Variant result Field marker and canonical `[value, metadata]` storage for each emitted value;
- matches Spark behavior for scalar values, Variant null, SQL NULL, empty containers, and the outer form; and
- keeps unrelated Variant-bearing generators/operators on explicit fallback.

Add focused parity and native-plan tests for objects, arrays, nested values, empty containers, scalars, Variant/SQL nulls, Unicode keys, aliases, lateral-view syntax, and outer versus non-outer cardinality.

### Additional context

Related work: #4295, #5407, #5425, and the general generator execution path.

Non-goals: Variant mutation, subfield pruning or pushdown, shuffle/spill, C2R, writing, Python transport, and Iceberg-specific work.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.