apache / apache/datafusion

feat: OFFSET pushdown for multi-file parquet scans

Open
#21,915 0 comments 0 reactions 1 assignee Claimed by @zhuqi-lucas View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

## Background

#21828 implements OFFSET pushdown for single-file parquet queries. Multi-file queries still use `GlobalLimitExec` for offset handling.

## Problem

For multi-file queries like `SELECT * FROM directory/ LIMIT 5 OFFSET 1000000`, the offset is handled by `GlobalLimitExec` which reads all rows then discards the first 1M. With multiple files, we could skip entire files whose cumulative row count falls within the offset.

## Challenge

File read order is non-deterministic with `target_partitions > 1` and dynamic scheduling (#21351). A shared counter (`Arc`) across file openers could work for single-partition sequential reads, but multi-partition ordering is undefined.

## Proposed approach

1. Single partition (`preserve_order=true`): files read in deterministic order → shared counter tracks consumed offset across files → skip entire files + RGs
2. Multi-partition: keep `GlobalLimitExec` (order undefined without ORDER BY)
3. Use file-level statistics (`PartitionedFile.statistics.num_rows`) to skip entire files before opening

## Related

- #21828 — Single-file OFFSET pushdown (parent PR)
- #19654 — Original issue for OFFSET performance

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.