apache / apache/datafusion

Re-sort file groups in FileScanConfig to satisfy ordering requirements

Open
#19,724 8 comments 2 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

`FileScanConfig::try_pushdown_sort` could support re-sorting or re-arranging the `FileGroup`s themselves using min/max statistics to satisfy the queries preferred sort order.

This is described in section 5.3 of [Pruning in Snowflake: Working Smarter, Not Harder](https://arxiv.org/pdf/2504.11540).

Some considerations are:
- If we start re-building groups what should the parallelism be? One the one hand it would make sense to try to match the original parallelism, on the other hand that may not be possible (e.g. if we can only satisfy the sort ordering by making groups `[[f1, f2, f3], [f4]]` maybe it's worth it to have lopsided groups, less or more groups) or even optimal (in a TopK query reduced parallelism can lead to faster queries if we end up only scanning 1 group or even 1 file; all of the work opening the others is wasted effort; this is also known as `ProgressiveEval` and discussed in https://github.com/apache/datafusion/issues/15191).
-

Contributor guide

Open the contributing guide

Research direction

Start at FileScanConfig::try_pushdown_sort and read section 5.3 of the linked Snowflake pruning paper, then review the ProgressiveEval discussion in issue #15191. The work needs an agreed design for reordering FileGroups and choosing parallelism; done means that behavior is specified and implemented with validation for the preferred sort order.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.