apache / apache/datafusion-comet

Support Spark-compatible duplicate Parquet field resolution

Open
#5,884 0 comments 0 reactions 0 assignees View on GitHub
requires-triage
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

## Description

Implement Spark-compatible duplicate Parquet sibling-name resolution in the native reader, replacing the explicit rejection proposed in PR #5786 for issue #5783.

As reported in [the PR review](https://github.com/apache/datafusion-comet/pull/5786#pullrequestreview-5187242277), Spark's behavior depends on where the duplicate occurs:

- Root group: with `message spark_schema { optional int64 a; optional int64 a; optional int64 b; }` and one raw row `a=1, a=2, b=3`, an explicit `a bigint` read schema returns `[1]` (first child wins).
- Nested structs: issue #5783 demonstrates last-wins resolution through `caseSensitiveParquetFieldMap`.

The native Arrow decoder can combine distinct leaves into one column before schema adaptation, multiplying rows or failing with a decoder synchronization error. Resolution must select the correct physical leaf before decoding, rather than dropping duplicate logical fields afterward.

## Acceptance criteria

Match Spark results, row counts, and error behavior for root, nested struct, array-element, and map-value duplicates using explicit read schemas. Cover case sensitivity and field-ID reads; preserve unrelated-column projection, filter pruning, and metadata-cache behavior. Replace the scan compatibility limitation once supported.

Related: #5783, #5786.

### Additional reproduction: reader-dependent nested behavior

Independent testing on Spark 4.1.3 with Comet disabled, `spark.sql.caseSensitive=true`, and Spark's vectorized Parquet reader enabled (`FileScan parquet Batched:true`) did not produce a simple last-wins result. The fixture:

```scala
spark.range(3).coalesce(1)
.selectExpr("named_struct('dup', id, 'dup', id + 100) as s")
.write.parquet(path)
spark.read.schema("s struct").parquet(path)
```

returned the multiset `{0, 100, 1}`, rather than `{100, 101, 102}`. Result order is not asserted. The root duplicate regression separately confirmed first-wins under case-sensitive resolution.

Treat the first/last-wins observations above as specific reader/source observations, not a universal resolution contract. Characterize Spark behavior across versions and vectorized/non-vectorized readers before choosing semantics; preserve the fail-fast guard for selected ambiguous groups until correct resolution is supported.

Contributor guide

Open the contributing guide

Research direction

Start with issue #5783, PR #5786, and the native Parquet reader, then reproduce the root and nested duplicate-field cases against Spark 4.1.3 with vectorized and non-vectorized readers. Characterize root, struct, array-element, and map-value behavior under case sensitivity and field-ID reads before choosing semantics. Done means matching Spark results, row counts, errors, pruning, caching, and scan compatibility.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.