apache / apache/datafusion-comet

Duplicate field ids inside a struct are not validated when the file schema equals the requested schema and no predicate is pushed

Open
#5,801 1 comment 0 reactions 0 assignees View on GitHub
requires-triage
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 6h
Merged PRs (30d)
190

Description

### Describe the bug

Comet validates Parquet field id lookups in the physical expression adapter, which resolves each file's mapping once per file (#5654). DataFusion 55's Parquet opener skips the adapter entirely when the file's physical schema compares equal to the logical schema and there is no pushed predicate. In that case a file whose struct carries two children with the same field id is read positionally instead of failing with Spark's duplicate field id error.

Spark-written files always carry key-value metadata that arrow-rs folds into the physical schema, so they never compare equal and always reach the adapter. The gap is limited to files with no key-value metadata at all, read with field id matching enabled and no data filter.

### Steps to reproduce

1. Write a Parquet file without key-value metadata whose schema is `s` (parquet-mr with a hand-built MessageType).
2. Read it with the native scan, `spark.sql.parquet.fieldId.read.enabled=true`, and a requested schema identical to the file schema, with no filter.
3. Spark rejects the duplicate id during schema clipping; Comet returns the rows.

### Expected behavior

The duplicate field id error, as when a cast or predicate is present.

### Additional context

Found in review of #5654, where the once-per-file validation was added. The short circuit is in DataFusion's opener, so the fix is either a Comet-side check before the opener decides, or an upstream option to always run the adapter.

Contributor guide

Open the contributing guide

Research direction

Start by tracing Comet's physical expression adapter and DataFusion's Parquet opener, focusing on the short circuit when the physical and logical schemas compare equal and no predicate is pushed. Use the metadata-free duplicate-id reproduction described here, and compare it with the validation path from #5654. Done means the native scan rejects duplicate struct field IDs with field-id matching enabled, even for an identical schema without a filter.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, scala
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.