apache / apache/datafusion

parquet: Add an option to not parse the Page Index on each query

Open
#12,547 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

`CREATE TABLE` does not parse the Page Index, and `SELECT` does not cache it. This can make requests on large Parquet datasets take a significant time for queries that have a small number of results.

For example, with a simple `SELECT int_column, other_int_column WHERE int_column=123456` on a table with 184 billion rows (so about 9 million Page Index items, given the default 20k page size)

> output_rows=0, elapsed_compute=96ns, num_predicate_creation_errors=0, page_index_rows_filtered=0, predicate_evaluation_errors=0, row_groups_pruned_bloom_filter=21050, row_groups_matched_bloom_filter=0, file_open_errors=0, file_scan_errors=0, bytes_scanned=25023432248, row_groups_matched_statistics=21050, pushdown_rows_filtered=0, row_groups_pruned_statistics=173576, time_elapsed_scanning_total=16.763964ms, page_index_eval_time=3.153918ms, time_elapsed_scanning_until_data=16.745759ms, time_elapsed_processing=61.531313027s, **time_elapsed_opening=96.012649352s**, pushdown_eval_time=382ns

### Describe the solution you'd like

Parse it once and for all, either on `CREATE TABLE` or lazily as `SELECT` queries read the files. (Note that in the case of partitioned tables, not all files may be read by the first `SELECT`)

### Describe alternatives you've considered

https://github.com/apache/datafusion/blob/3b93cc952b889cec2364ad2490ae18ecddb3ca49/datafusion-examples/examples/advanced_parquet_index.rs

but it requires using the low-level API, and is not available through the SQL or Python interfaces.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with datafusion-examples/examples/advanced_parquet_index.rs, the low-level example linked in the issue, and trace how Parquet page indexes are handled for CREATE TABLE and SELECT. Define and validate a SQL- or Python-accessible option that avoids parsing the same index on every query, including the partitioned-table case where files may be read lazily.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust, sql
Domain
databases, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.