apache / apache/datafusion-comet
Support Spark-compatible duplicate Parquet field resolution
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
## Description
Implement Spark-compatible duplicate Parquet sibling-name resolution in the native reader, replacing the explicit rejection proposed in PR #5786 for issue #5783.
As reported in [the PR review](https://github.com/apache/datafusion-comet/pull/5786#pullrequestreview-5187242277), Spark's behavior depends on where the duplicate occurs:
- Root group: with `message spark_schema { optional int64 a; optional int64 a; optional int64 b; }` and one raw row `a=1, a=2, b=3`, an explicit `a bigint` read schema returns `[1]` (first child wins).
- Nested structs: issue #5783 demonstrates last-wins resolution through `caseSensitiveParquetFieldMap`.
The native Arrow decoder can combine distinct leaves into one column before schema adaptation, multiplying rows or failing with a decoder synchronization error. Resolution must select the correct physical leaf before decoding, rather than dropping duplicate logical fields afterward.
## Acceptance criteria
Match Spark results, row counts, and error behavior for root, nested struct, array-element, and map-value duplicates using explicit read schemas. Cover case sensitivity and field-ID reads; preserve unrelated-column projection, filter pruning, and metadata-cache behavior. Replace the scan compatibility limitation once supported.
Related: #5783, #5786.
### Additional reproduction: reader-dependent nested behavior
Independent testing on Spark 4.1.3 with Comet disabled, `spark.sql.caseSensitive=true`, and Spark's vectorized Parquet reader enabled (`FileScan parquet Batched:true`) did not produce a simple last-wins result. The fixture:
```scala
spark.range(3).coalesce(1)
.selectExpr("named_struct('dup', id, 'dup', id + 100) as s")
.write.parquet(path)
spark.read.schema("s struct").parquet(path)
```
returned the multiset `{0, 100, 1}`, rather than `{100, 101, 102}`. Result order is not asserted. The root duplicate regression separately confirmed first-wins under case-sensitive resolution.
Treat the first/last-wins observations above as specific reader/source observations, not a universal resolution contract. Characterize Spark behavior across versions and vectorized/non-vectorized readers before choosing semantics; preserve the fail-fast guard for selected ambiguous groups until correct resolution is supported.
Contributor guide
Research direction
Start with issue #5783, PR #5786, and the native Parquet reader, then reproduce the root and nested duplicate-field cases against Spark 4.1.3 with vectorized and non-vectorized readers. Characterize root, struct, array-element, and map-value behavior under case sensitivity and field-ID reads before choosing semantics. Done means matching Spark results, row counts, errors, pruning, caching, and scan compatibility.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100