apache / apache/datafusion-comet

[Variant] Enable direct top-level Variant projection in ordinary native Parquet scans

Open
#5,551 1 comment 0 reactions 1 assignee Claimed by @peterxcli View on GitHub
area:scan enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### Problem

After the schema, normalization, and physical-reader layers are ready, Comet still needs a deliberately narrow JVM admission path for direct top-level Variant projection and Spark-compatible behavior for missing columns and unsupported consumers.

Parent: #5546
Depends on: #5550

### Solution

- Admit only direct top-level `VariantType` fields in ordinary native Parquet scans.
- Reuse Unicode Parquet name matching from #5602 (#5495 is complete); do not port #5407's non-ASCII-name fallback.
- Build the native data schema from Spark's pruned required schema so unread Variant roots remain removable.
- Preserve existence-default value/index pairing and transport a Variant default as a scan-only constant `[value, metadata]` struct.
- Apply a default only when the physical column is absent.
- Export the marked Field and reuse `CometStructVector` so inherited `ColumnVector.getVariant` works.
- Keep nested Variant, pushed `VariantStruct` (#5519), strict malformed reads, expressions, writes, C2R, shuffle/spill, Python operators, encryption, and Iceberg on explicit Spark fallback where applicable.
- Document the supported surface and remaining fallbacks.

### Tests

- `SELECT v` and `SELECT id, v, tail` remain native and match Spark.
- Objects, arrays, scalars, JSON null, SQL NULL, nullable parents, and entirely-null columns round-trip.
- Spark sees logical `VariantType` with exact `[value, metadata]` Binary children.
- A missing Variant column receives its existence default without shifting later defaults.
- Unread Variant pruning from #5377 remains native.
- Case-insensitive Unicode Variant projection remains native and matches Spark, covering a non-ASCII Variant root, an ordinary sibling beside Variant, and physical Kelvin sign `K` matched to logical `k`. Preserve Spark's missing-column behavior for physical `ſ` read as logical `s`.
- Unsupported consumers above have focused fallback assertions.
- Spark 4.0/4.1 profiles pass and Spark 3.5 still compiles unchanged.

Completing this issue closes the ordinary-Parquet phase of #4295. Iceberg and the broader shredded optimization scope in #3983 remain separate.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.