apache / apache/datafusion

[EPIC] Sort Pushdown: remaining improvements

Open
#24,270 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

Follow-up to #23036, which tracked the broader **Sort Pushdown** effort across DataFusion v52 → v55. The bulk of that epic is now complete — the optimizer-rule framework and `Exact / Inexact / Unsupported` classifier, reverse iteration, statistics-based file/RG reorder, multi-partition `BufferExec`, and the runtime RG-level early stop (#22450, merged) have all landed. This issue tracks the remaining open follow-ups so #23036 can be closed as substantially done.

See #23036 for the full architecture, phase history, and benchmark results.

## Remaining open items

### Statistics-based reorder follow-ups
- [ ] #22198 — Follow-ups for stats-based RG / file reorder: multi-column, function-wrapped sort expressions, compound `ORDER BY`
- [ ] #21169 — Bug: `EnforceDistribution` loses `fetch` (LIMIT) from `CoalescePartitionsExec` / `SortPreservingMergeExec`
- [ ] #19394 — Add more `ExecutionPlan` nodes to support sort pushdown

### Multi-partition / BufferExec follow-ups
- [ ] #21381 — Parallel merge in `SortPreservingMergeExec` after sort elimination
- [ ] #21440 — Dynamic `BufferExec` sizing: row limit + memory cap
- [ ] #21780 — Refactor: make `DynamicFilterPhysicalExpr.fetch` a shared reference to `SortExec.fetch`

### Runtime RG-level early stop follow-ups (Phase 4 landed via #22450)
- [ ] #23067 — Per-RG `fully_matched` `RowFilter` skip on top of #22450 (PR #23696, in review)
- [ ] #22949 — Bug: `PhysicalExprSimplifier` folds `DynamicFilterPhysicalExpr` wrapper, breaking the live link to TopK updates

### Future — `Exact` reverse path (blocked on arrow-rs page-level reverse)
- [ ] #19486 — Support reverse page for sort pushdown (phase 1 follow-up)
- [ ] #19048 — Reverse parquet OOM mitigation for huge row groups
- [ ] Cross-repo: [apache/arrow-rs#9937](https://github.com/apache/arrow-rs/issues/9937) — POC `ReverseSerializedPageReader` (unlocks strictly-DESC output → `Exact` sort pushdown)

Contributor guide

Open the contributing guide

Research direction

Start with issue #23036 for the Sort Pushdown architecture and phase history, then review the individually listed follow-up issues (#22198, #21169, #19394, #21381, #21440, #21780, #23067, #22949, #19486, and #19048). Done means resolving the applicable checklist items and closing this tracking epic; several items depend on other work or cross-repository arrow-rs#9937.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.