apache / apache/datafusion-comet

[Variant] Native top-level Variant projection from ordinary Parquet

Open
#5,546 1 comment 0 reactions 1 assignee Claimed by @peterxcli View on GitHub
area:scan enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What problem does this solve?

#5407 implements the ordinary-Parquet phase of #4295, but it now combines several independently reviewable concerns: shared Arrow FFI schema export, Variant logical-type transport, native whole-value normalization, Spark-compatible Parquet physical semantics, and scan admission/schema-evolution defaults.

This issue tracks re-landing that work as five atomic pull requests under the broader Variant roadmap in #5438, following the maintainer request in https://github.com/apache/datafusion-comet/pull/5407#issuecomment-5463550828.

### Scope

Support direct, top-level, whole-value `VariantType` projection from ordinary Parquet:

```sql
SELECT v FROM parquet_table;
SELECT id, v, tail FROM parquet_table;
```

The completed series preserves Variant identity through:

```text
Spark VariantType
-> Comet protobuf
-> marked Arrow Field
-> ordinary native Parquet scan
-> whole-value normalization
-> Arrow C Data Interface
-> CometStructVector
-> ColumnVector.getVariant
```

This tracker does not include Iceberg, nested Variant, Variant expressions, subfield pruning, predicate pushdown, writes, columnar-to-row, shuffle/spill, or Python transport.

### Ordered subtasks

- [x] #5547 — Export complete Arrow Fields through the native FFI boundary (#5552)
- [x] #5548 — Carry Spark VariantType identity through Comet schema serialization (#5631)
- [x] #5549 — Normalize marked Variant arrays at the native Parquet boundary (#5715, in review)
- [ ] #5550 — Match Spark physical Parquet semantics for projected Variant columns
- [ ] #5551 — Enable direct top-level Variant scan admission, defaults, and fallbacks

Each pull request must include focused tests and retain explicit Spark fallback until the complete scan boundary is ready.

### Definition of done

- Direct top-level Variant projection remains a Comet native Parquet scan and matches Spark.
- Fields before and after Variant retain their order.
- Objects, arrays, scalars, Variant JSON null, SQL NULL, and nullable parents round-trip.
- Canonical unshredded and shredded inputs reconstruct whole values.
- Spark receives logical `VariantType` with exactly `[value, metadata]` children.
- Unread Variant pruning from #5377 remains native.
- Unsupported expressions, nested Variant, pushed `VariantStruct`, native writes, C2R, shuffle/spill, Python operators, and Iceberg retain explicit fallback.
- Spark 3.x compilation and behavior remain unchanged.

### Related work

- #3983 — shredded writing, pruning, and predicate pushdown
- #5474 — removal of legacy Spark UTF-16 output rewriting
- #5477 — removal or reassessment of upstream compatibility paths
- #5495 — completed by #5602; the Variant series reuses its shared Spark-compatible Unicode Parquet field matcher
- #5519 — pushed one-field `VariantStruct`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.