`parquet::column::reader::GenericColumnReader::skip_records` still decompresses most data
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 168
Description
**Describe the bug**
I noticed this while investigating https://github.com/apache/datafusion/issues/7845#issuecomment-2370455772.
The suggestion from @jayzhan211 and @alamb was that `datafusion.execution.parquet.pushdown_filters true` should improve performance of queries like this, but it seems to make them slower.
I think the reason is that data is being decompressed twice (or data is being decompressed that shouldn't be), here's a screenshot from samply running on [this code](https://github.com/samuelcolvin/batson-perf):
(You can view this flamegraph properly [here](https://share.firefox.dev/3zrdUpN))
You can see that there are two blocks of decompression work, the second one is associated with `parquet::column::reader::GenericColumnReader::skip_records` and happens after the first decompression chunk and running the query has completed.
In particular you can se that there's a `read_new_page()` cal in `
parquet::column::reader::GenericColumnReader::skip_records` (line 335) that's taking a lot of time:
My question is - could this second run of compression be avoided?
**To Reproduce**
Clone https://github.com/samuelcolvin/batson-perf, comment out one of the modes, compile with profiling enabled `cargo build --profile profiling`, run with samply `samply record ./target/profiling/batson-perf`
**Expected behavior**
I would expect that `datafusion.execution.parquet.pushdown_filters true` was faster, I think the reason it's not is decompressing the data twice.
**Additional context**
https://github.com/apache/datafusion/issues/7845#issuecomment-2370455772
Contributor guide
Research direction
Start with parquet::column::reader::GenericColumnReader::skip_records, especially the read_new_page() call at line 335, and reproduce the profile using the batson-perf commands with profiling enabled. Compare the two decompression blocks when datafusion.execution.parquet.pushdown_filters is enabled. Done means establishing whether the second decompression can be avoided and confirming the performance behavior in the reproduction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100