Parquet reader does not support unknown physical/logical type combinations per GH-607
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
### Describe the bug
parquet-format [GH-607](https://github.com/apache/parquet-format/pull/606) adds clarification that in addition to unknown logical type enum variants, implementations must also handle unknown combinations of physical type and logical type (e.g. an `INT32` column annotated with `UUID`). This crate currently checks for unsupported combinations and errors when encountering them.
### To Reproduce
Try reading the file added in https://github.com/apache/parquet-testing/pull/122 (parquet-testing/data/int32_with_uuid_logical_type.parquet).
### Expected behavior
Unknown combinations should now be treated as having an unknown logical type i.e. on read they should be provided to users as the physical type with no type annotation, and statistics should be ignored. It's not clear (to me) what should be done on round trip.
It might be good to also log a warning when an unknown type is encountered to let users know of the degraded experience.
### Additional context
We could continue to error on unknown combination, or add a gate to control behavior upon encountering unknown types.
Contributor guide
Research direction
Start by reproducing the Parquet reader failure with parquet-testing/data/int32_with_uuid_logical_type.parquet and trace how unsupported physical/logical combinations are handled. Done means the file reads with its physical type unannotated and statistics are ignored; round-trip behavior and whether to warn or gate the behavior still need a project decision.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100