Duplication between `ParquetFormat` and `ParquetReadOptions` is confusing
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
While working on https://github.com/apache/arrow-datafusion/pull/2985 I noticed I had to repeat the `skip_metadata` configuration option on both `ParquetReadOptions` (public API) and the `ParquetFormat`.
It also looks like the 'metadata_hint' size option was not added to the public API.
I believe the split is due to the fact that some of the values on `ParquetReadOptions` are used to make a `ListingOptions` and some are used on the actual `ParquetFormat` itself.
**Describe the solution you'd like**
I would like to have the options in a single place and avoid the redundancy
Oneapproach might be to pull the options that are shared into its own sub-struct shared by `ParquetReadOptions` and `ParquetFormat, but since those fields are pub that would be backwards incompatible
**Describe alternatives you've considered**
Another approach would be to add a `ParquetReadOptions` field to the `ParquetFormat` but this has issues that then several of these fields will be ignored.
**Additional context**
Add any other context or screenshots about the feature request here.
Contributor guide
Research direction
Start by locating the definitions and uses of ParquetReadOptions and ParquetFormat, then compare their fields and how ListingOptions is constructed. The work is done when shared configuration has one consistent representation, skip_metadata is not duplicated, and metadata_hint is exposed through the public API without silently ignored fields.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend-api-design
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100