apache / apache/datafusion

Option to *always* enable page index for `ParquetOpener`

Open
#15,179 3 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

We are using DataFusion as described in #13456 – i.e. querying Parquet files directly from object storage (S3). I've recently been exploring the possibility of introducing cached parquet metadata into query planning and execution (related: #12592).

After a bit of a journey, I was able to get a pre-deserialized `ParquetMetaData` plumbed into a custom `TableProvider`, which would then refer to the ready-set metadata throughout planning and execution (for execution, a custom `ParquetFileReaderFactory` and `AsyncFileReader` are also required).

For queries with a filter (e.g. `column = value`), this had the expected effect of removing the metadata calls from our traces – success!

However, I was surprised to see that traces for queries *without* a filter (only a limit) showed dramatically different object store calls. Specifically:

- Using the default setup, one query would make **6** object store calls (5 for metadata and 1 for data), and would scan **~137Mb** of data.
- Using the caching setup, the same query would make **29** object store calls (all for data), and would scan **~4.2Mb** of data.

I had thought my changes to be fairly surgical, and would not have introduced this kind of difference in the query execution. After picking through the implementation again, I think this is caused by this logic in `ParquetOpener`:

https://github.com/apache/datafusion/blob/efb75f3fe34b0de51ff7aed1959451202e7733a4/datafusion/datasource-parquet/src/opener.rs#L119-L122

This calls `should_enable_page_index`, which is defined as:

https://github.com/apache/datafusion/blob/efb75f3fe34b0de51ff7aed1959451202e7733a4/datafusion/datasource-parquet/src/mod.rs#L537-L547

The resulting `enable_page_index` seems to ultimately be what controls whether the page index is read and used inside `ParquetOpener`.

In my caching implementation, the whole `ParquetMetaData` is always available and so presumably some of the range pruning logic inside `ParquetOpener` has an effect, where otherwise that metadata would not be loaded.

For the example query above, this leads to a significant (~30%) reduction in latency.

### Describe the solution you'd like

The latency reduction we have observed comes almost entirely from using the page index even though there's no predicate.

I.e., although caching does reduce the number of object calls for otherwise like-for-like execution plans (e.g. with a predicate), it has a small impact on latency.

For our use-case, we would get most of the benefit if there was some way to "force" the use of the page index, even without a pruning predicate (and/or for this decision to made heuristically based on the likely savings).

And/or, the documentation around [`enable_page_index`](https://docs.rs/datafusion/latest/datafusion/config/struct.ParquetOptions.html#structfield.enable_page_index) could clarify that actual use of the metadata will be heuristic.

### Describe alternatives you've considered

Even without caching, it still takes quite a lot of plumbing to get the page index to be used. The smallest implementation I could come up with uses:

- A `FileFormat`, whose `create_physical_plan` injects a custom...
- `ParquetFileReaderFactory`, whose `create_reader`:
1. Builds a `ParquetObjectReader` and sets `with_preload_column_index`/`with_preload_offset_index` to `true`
1. Returns a custom...
- `AsyncFileReader` (not necessary for functionality, but `ParquetFileReader` used by `DefaultParquetFileReaderFactory` is private).

This is ~200 lines of boilerplate consisting mostly of trait forwarding and copied implementations in order to add `.with_preload_column_index(true).with_preload_offset_index(true)` at the right point.

I've also not done a thorough investigation into the performance – it's possible that not using the page index is the best choice in the general case.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reading datafusion/datasource-parquet/src/opener.rs around the ParquetOpener logic and the should_enable_page_index definition in datafusion/datasource-parquet/src/mod.rs. Review ParquetOptions.enable_page_index documentation and the reported object-store behavior, then determine how an always-enable option or heuristic should be exposed. Done means the behavior is configurable and the documentation explains when page-index metadata is used.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.