apache / apache/arrow-rs

`parquet::column::reader::GenericColumnReader::skip_records` still decompresses most data

Open
#6,454 13 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 16h
Merged PRs (30d)
168

Description

**Describe the bug**

I noticed this while investigating https://github.com/apache/datafusion/issues/7845#issuecomment-2370455772.

The suggestion from @jayzhan211 and @alamb was that `datafusion.execution.parquet.pushdown_filters true` should improve performance of queries like this, but it seems to make them slower.

I think the reason is that data is being decompressed twice (or data is being decompressed that shouldn't be), here's a screenshot from samply running on [this code](https://github.com/samuelcolvin/batson-perf):

image

(You can view this flamegraph properly [here](https://share.firefox.dev/3zrdUpN))

You can see that there are two blocks of decompression work, the second one is associated with `parquet::column::reader::GenericColumnReader::skip_records` and happens after the first decompression chunk and running the query has completed.

In particular you can se that there's a `read_new_page()` cal in `
parquet::column::reader::GenericColumnReader::skip_records` (line 335) that's taking a lot of time:

image

My question is - could this second run of compression be avoided?

**To Reproduce**

Clone https://github.com/samuelcolvin/batson-perf, comment out one of the modes, compile with profiling enabled `cargo build --profile profiling`, run with samply `samply record ./target/profiling/batson-perf`

**Expected behavior**

I would expect that `datafusion.execution.parquet.pushdown_filters true` was faster, I think the reason it's not is decompressing the data twice.

**Additional context**

https://github.com/apache/datafusion/issues/7845#issuecomment-2370455772

Contributor guide

Open the contributing guide

Research direction

Start with parquet::column::reader::GenericColumnReader::skip_records, especially the read_new_page() call at line 335, and reproduce the profile using the batson-perf commands with profiling enabled. Compare the two decompression blocks when datafusion.execution.parquet.pushdown_filters is enabled. Done means establishing whether the second decompression can be avoided and confirming the performance behavior in the reproduction.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.