apache / apache/datafusion

parquet: Add support for user-provided metadata loaders

Open
#12,592 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

This allows users to implement #12547 (caching metadata, especially the Page Index) themselves without reimplementing all of `ParquetOpener` and `ParquetExec`

### Describe the solution you'd like

Add a method to the object returned by `ParquetFileReaderFactory` that loads metadata, so that users who overwrite the `DefaultParquerFileReaderFactory` with `SessionState::register_file_format(..., true)` can provide their own implementation.

### Describe alternatives you've considered

Adding caching support directly in Datafusion. I tried in #12548 but it went nowhere.

Support for caching directly in Datafusion can be added later, based on this feature.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Trace ParquetFileReaderFactory, DefaultParquerFileReaderFactory, ParquetOpener, and ParquetExec, then follow the SessionState::register_file_format(..., true) registration path. Done means a custom factory can provide metadata-loading behavior, including caching, without reimplementing the opener and execution components.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.