[EPIC] Sort Pushdown: remaining improvements
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
Follow-up to #23036, which tracked the broader **Sort Pushdown** effort across DataFusion v52 → v55. The bulk of that epic is now complete — the optimizer-rule framework and `Exact / Inexact / Unsupported` classifier, reverse iteration, statistics-based file/RG reorder, multi-partition `BufferExec`, and the runtime RG-level early stop (#22450, merged) have all landed. This issue tracks the remaining open follow-ups so #23036 can be closed as substantially done.
See #23036 for the full architecture, phase history, and benchmark results.
## Remaining open items
### Statistics-based reorder follow-ups
- [ ] #22198 — Follow-ups for stats-based RG / file reorder: multi-column, function-wrapped sort expressions, compound `ORDER BY`
- [ ] #21169 — Bug: `EnforceDistribution` loses `fetch` (LIMIT) from `CoalescePartitionsExec` / `SortPreservingMergeExec`
- [ ] #19394 — Add more `ExecutionPlan` nodes to support sort pushdown
### Multi-partition / BufferExec follow-ups
- [ ] #21381 — Parallel merge in `SortPreservingMergeExec` after sort elimination
- [ ] #21440 — Dynamic `BufferExec` sizing: row limit + memory cap
- [ ] #21780 — Refactor: make `DynamicFilterPhysicalExpr.fetch` a shared reference to `SortExec.fetch`
### Runtime RG-level early stop follow-ups (Phase 4 landed via #22450)
- [ ] #23067 — Per-RG `fully_matched` `RowFilter` skip on top of #22450 (PR #23696, in review)
- [ ] #22949 — Bug: `PhysicalExprSimplifier` folds `DynamicFilterPhysicalExpr` wrapper, breaking the live link to TopK updates
### Future — `Exact` reverse path (blocked on arrow-rs page-level reverse)
- [ ] #19486 — Support reverse page for sort pushdown (phase 1 follow-up)
- [ ] #19048 — Reverse parquet OOM mitigation for huge row groups
- [ ] Cross-repo: [apache/arrow-rs#9937](https://github.com/apache/arrow-rs/issues/9937) — POC `ReverseSerializedPageReader` (unlocks strictly-DESC output → `Exact` sort pushdown)
Contributor guide
Research direction
Start with issue #23036 for the Sort Pushdown architecture and phase history, then review the individually listed follow-up issues (#22198, #21169, #19394, #21381, #21440, #21780, #23067, #22949, #19486, and #19048). Done means resolving the applicable checklist items and closing this tracking epic; several items depend on other work or cross-repository arrow-rs#9937.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100