apache / apache/arrow-rs

Parquet reader does not support unknown physical/logical type combinations per GH-607

Open
#10,941 5 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

### Describe the bug

parquet-format [GH-607](https://github.com/apache/parquet-format/pull/606) adds clarification that in addition to unknown logical type enum variants, implementations must also handle unknown combinations of physical type and logical type (e.g. an `INT32` column annotated with `UUID`). This crate currently checks for unsupported combinations and errors when encountering them.

### To Reproduce

Try reading the file added in https://github.com/apache/parquet-testing/pull/122 (parquet-testing/data/int32_with_uuid_logical_type.parquet).

### Expected behavior

Unknown combinations should now be treated as having an unknown logical type i.e. on read they should be provided to users as the physical type with no type annotation, and statistics should be ignored. It's not clear (to me) what should be done on round trip.

It might be good to also log a warning when an unknown type is encountered to let users know of the degraded experience.

### Additional context

We could continue to error on unknown combination, or add a gate to control behavior upon encountering unknown types.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the Parquet reader failure with parquet-testing/data/int32_with_uuid_logical_type.parquet and trace how unsupported physical/logical combinations are handled. Done means the file reads with its physical type unannotated and statistics are ignored; round-trip behavior and whether to warn or gate the behavior still need a project decision.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.