apache / apache/datafusion-comet
[Variant] Native top-level Variant projection from ordinary Parquet
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What problem does this solve?
#5407 implements the ordinary-Parquet phase of #4295, but it now combines several independently reviewable concerns: shared Arrow FFI schema export, Variant logical-type transport, native whole-value normalization, Spark-compatible Parquet physical semantics, and scan admission/schema-evolution defaults.
This issue tracks re-landing that work as five atomic pull requests under the broader Variant roadmap in #5438, following the maintainer request in https://github.com/apache/datafusion-comet/pull/5407#issuecomment-5463550828.
### Scope
Support direct, top-level, whole-value `VariantType` projection from ordinary Parquet:
```sql
SELECT v FROM parquet_table;
SELECT id, v, tail FROM parquet_table;
```
The completed series preserves Variant identity through:
```text
Spark VariantType
-> Comet protobuf
-> marked Arrow Field
-> ordinary native Parquet scan
-> whole-value normalization
-> Arrow C Data Interface
-> CometStructVector
-> ColumnVector.getVariant
```
This tracker does not include Iceberg, nested Variant, Variant expressions, subfield pruning, predicate pushdown, writes, columnar-to-row, shuffle/spill, or Python transport.
### Ordered subtasks
- [x] #5547 — Export complete Arrow Fields through the native FFI boundary (#5552)
- [x] #5548 — Carry Spark VariantType identity through Comet schema serialization (#5631)
- [x] #5549 — Normalize marked Variant arrays at the native Parquet boundary (#5715, in review)
- [ ] #5550 — Match Spark physical Parquet semantics for projected Variant columns
- [ ] #5551 — Enable direct top-level Variant scan admission, defaults, and fallbacks
Each pull request must include focused tests and retain explicit Spark fallback until the complete scan boundary is ready.
### Definition of done
- Direct top-level Variant projection remains a Comet native Parquet scan and matches Spark.
- Fields before and after Variant retain their order.
- Objects, arrays, scalars, Variant JSON null, SQL NULL, and nullable parents round-trip.
- Canonical unshredded and shredded inputs reconstruct whole values.
- Spark receives logical `VariantType` with exactly `[value, metadata]` children.
- Unread Variant pruning from #5377 remains native.
- Unsupported expressions, nested Variant, pushed `VariantStruct`, native writes, C2R, shuffle/spill, Python operators, and Iceberg retain explicit fallback.
- Spark 3.x compilation and behavior remain unchanged.
### Related work
- #3983 — shredded writing, pruning, and predicate pushdown
- #5474 — removal of legacy Spark UTF-16 output rewriting
- #5477 — removal or reassessment of upstream compatibility paths
- #5495 — completed by #5602; the Variant series reuses its shared Spark-compatible Unicode Parquet field matcher
- #5519 — pushed one-field `VariantStruct`
Contributor guide
Assessment
This issue has not been assessed yet.