apache / apache/datafusion

Duplication between `ParquetFormat` and `ParquetReadOptions` is confusing

Open
#2,987 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

While working on https://github.com/apache/arrow-datafusion/pull/2985 I noticed I had to repeat the `skip_metadata` configuration option on both `ParquetReadOptions` (public API) and the `ParquetFormat`.

It also looks like the 'metadata_hint' size option was not added to the public API.

I believe the split is due to the fact that some of the values on `ParquetReadOptions` are used to make a `ListingOptions` and some are used on the actual `ParquetFormat` itself.

**Describe the solution you'd like**
I would like to have the options in a single place and avoid the redundancy

Oneapproach might be to pull the options that are shared into its own sub-struct shared by `ParquetReadOptions` and `ParquetFormat, but since those fields are pub that would be backwards incompatible

**Describe alternatives you've considered**
Another approach would be to add a `ParquetReadOptions` field to the `ParquetFormat` but this has issues that then several of these fields will be ignored.

**Additional context**
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

Research direction

Start by locating the definitions and uses of ParquetReadOptions and ParquetFormat, then compare their fields and how ListingOptions is constructed. The work is done when shared configuration has one consistent representation, skip_metadata is not duplicated, and metadata_hint is exposed through the public API without silently ignored fields.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.