apache / apache/datafusion

Filtered Parquet scans issue additional object-store requests after ParquetPushDecoder migration

Open
#24,527 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

## Describe the bug

After [PR #20839](https://github.com/apache/datafusion/pull/20839) replaced the pull-based Parquet reader with `ParquetPushDecoder`, a controlled comparison found substantially more object-store requests for a large filtered Parquet scan.

The leading explanation is that predicate and projected-column ranges are discovered in separate `NeedsData` rounds. DataFusion can coalesce ranges within one `get_byte_ranges` call, but not with ranges discovered in a later decoder round. Current `main` still follows this sequence:

```text
try_next_reader()
-> NeedsData(ranges)
-> get_byte_ranges(ranges).await
-> push_ranges(...)
-> repeat
```

In architectures where compute and storage are decoupled, each additional object-store round trip adds directly to scan latency. This is especially significant when scans touch many small Parquet files.

The request-level attribution to predicate versus projection rounds is inferred from the source and the A/B results below. The additional requests themselves are directly measured.

## To reproduce

Compare a selective filter with a narrow projection over many remote Parquet files while instrumenting `AsyncFileReader` request counts. Our controlled comparison upgraded only DataFusion from 53.1.0 to 54.1.0 while retaining `parquet` 58.3.0 and `object_store` 0.13.2:

| Configuration | Object-store requests | Bytes read |
|---|---:|---:|
| DataFusion 53.1.0 | 708K-709K | ~139 GB |
| DataFusion 54.1.0 with filter pushdown | 894K-898K | ~86.6 GB |
| DataFusion 54.1.0 without filter pushdown | 583K | ~186.3 GB |

The DF54 request increase closely matched one additional request per filtered file.

Disabling filter pushdown is not a viable workaround. It reduces request count but more than doubles bytes read and produced a 164-second runtime in this workload.

## Expected behavior

Retain the byte savings from predicate pushdown without requiring a separate network round trip for predicate and projected-column ranges.

Possible directions include:

- Coalesce ranges across consecutive decoder rounds.
- Speculatively fetch projected-column ranges where appropriate.
- Extend the arrow-rs decoder API to expose upcoming optional ranges.
- Add request-count coverage using a counting, delayed object store.

## Additional context

- PR #20839 explicitly identified coalescing and prefetching as goals of the push-decoder migration.
- Closed [PR #21370](https://github.com/apache/datafusion/pull/21370) coalesced adjacent ranges within one `NeedsData` result, but did not address ranges discovered across separate rounds.
- A search did not find an existing DataFusion issue covering this cross-round request amplification.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the ParquetPushDecoder flow from try_next_reader through NeedsData, get_byte_ranges, and push_ranges, using PRs #20839 and #21370 as context. Reproduce the request amplification with a counting, delayed object store and compare filtered scans. Done means predicate and projected-column ranges avoid unnecessary cross-round object-store requests while retaining predicate-pushdown byte savings.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.