apache / apache/datafusion-comet
Duplicate field ids inside a struct are not validated when the file schema equals the requested schema and no predicate is pushed
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 190
Description
### Describe the bug
Comet validates Parquet field id lookups in the physical expression adapter, which resolves each file's mapping once per file (#5654). DataFusion 55's Parquet opener skips the adapter entirely when the file's physical schema compares equal to the logical schema and there is no pushed predicate. In that case a file whose struct carries two children with the same field id is read positionally instead of failing with Spark's duplicate field id error.
Spark-written files always carry key-value metadata that arrow-rs folds into the physical schema, so they never compare equal and always reach the adapter. The gap is limited to files with no key-value metadata at all, read with field id matching enabled and no data filter.
### Steps to reproduce
1. Write a Parquet file without key-value metadata whose schema is `s` (parquet-mr with a hand-built MessageType).
2. Read it with the native scan, `spark.sql.parquet.fieldId.read.enabled=true`, and a requested schema identical to the file schema, with no filter.
3. Spark rejects the duplicate id during schema clipping; Comet returns the rows.
### Expected behavior
The duplicate field id error, as when a cast or predicate is present.
### Additional context
Found in review of #5654, where the once-per-file validation was added. The short circuit is in DataFusion's opener, so the fix is either a Comet-side check before the opener decides, or an upstream option to always run the adapter.
Contributor guide
Research direction
Start by tracing Comet's physical expression adapter and DataFusion's Parquet opener, focusing on the short circuit when the physical and logical schemas compare equal and no predicate is pushed. Use the metadata-free duplicate-id reproduction described here, and compare it with the validation path from #5654. Done means the native scan rejects duplicate struct field IDs with field-id matching enabled, even for an identical schema without a filter.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, scala
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100