Re-sort file groups in FileScanConfig to satisfy ordering requirements
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
`FileScanConfig::try_pushdown_sort` could support re-sorting or re-arranging the `FileGroup`s themselves using min/max statistics to satisfy the queries preferred sort order.
This is described in section 5.3 of [Pruning in Snowflake: Working Smarter, Not Harder](https://arxiv.org/pdf/2504.11540).
Some considerations are:
- If we start re-building groups what should the parallelism be? One the one hand it would make sense to try to match the original parallelism, on the other hand that may not be possible (e.g. if we can only satisfy the sort ordering by making groups `[[f1, f2, f3], [f4]]` maybe it's worth it to have lopsided groups, less or more groups) or even optimal (in a TopK query reduced parallelism can lead to faster queries if we end up only scanning 1 group or even 1 file; all of the work opening the others is wasted effort; this is also known as `ProgressiveEval` and discussed in https://github.com/apache/datafusion/issues/15191).
-
Contributor guide
Research direction
Start at FileScanConfig::try_pushdown_sort and read section 5.3 of the linked Snowflake pruning paper, then review the ProgressiveEval discussion in issue #15191. The work needs an agreed design for reordering FileGroups and choosing parallelism; done means that behavior is specified and implemented with validation for the preferred sort order.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100