apache / apache/datafusion-comet
[Variant] Enable direct top-level Variant projection in ordinary native Parquet scans
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### Problem
After the schema, normalization, and physical-reader layers are ready, Comet still needs a deliberately narrow JVM admission path for direct top-level Variant projection and Spark-compatible behavior for missing columns and unsupported consumers.
Parent: #5546
Depends on: #5550
### Solution
- Admit only direct top-level `VariantType` fields in ordinary native Parquet scans.
- Reuse Unicode Parquet name matching from #5602 (#5495 is complete); do not port #5407's non-ASCII-name fallback.
- Build the native data schema from Spark's pruned required schema so unread Variant roots remain removable.
- Preserve existence-default value/index pairing and transport a Variant default as a scan-only constant `[value, metadata]` struct.
- Apply a default only when the physical column is absent.
- Export the marked Field and reuse `CometStructVector` so inherited `ColumnVector.getVariant` works.
- Keep nested Variant, pushed `VariantStruct` (#5519), strict malformed reads, expressions, writes, C2R, shuffle/spill, Python operators, encryption, and Iceberg on explicit Spark fallback where applicable.
- Document the supported surface and remaining fallbacks.
### Tests
- `SELECT v` and `SELECT id, v, tail` remain native and match Spark.
- Objects, arrays, scalars, JSON null, SQL NULL, nullable parents, and entirely-null columns round-trip.
- Spark sees logical `VariantType` with exact `[value, metadata]` Binary children.
- A missing Variant column receives its existence default without shifting later defaults.
- Unread Variant pruning from #5377 remains native.
- Case-insensitive Unicode Variant projection remains native and matches Spark, covering a non-ASCII Variant root, an ordinary sibling beside Variant, and physical Kelvin sign `K` matched to logical `k`. Preserve Spark's missing-column behavior for physical `ſ` read as logical `s`.
- Unsupported consumers above have focused fallback assertions.
- Spark 4.0/4.1 profiles pass and Spark 3.5 still compiles unchanged.
Completing this issue closes the ordinary-Parquet phase of #4295. Iceberg and the broader shredded optimization scope in #3983 remain separate.
Contributor guide
Assessment
This issue has not been assessed yet.