Benchmark / program to test Spilling Sorts
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
- Part of https://github.com/apache/datafusion/issues/15271
There are many interesting ideas on how to improve DataFusion while spilling for example https://github.com/apache/datafusion/issues/15271 from @2010YOUY01 and others.
What I think we really need next to make progress in this area is a benchmark / agreed upon way of measuring our progress so that we can improve and
### Describe the solution you'd like
I would like a documented command / set of commands that is:
1. Easy to run (and thus fast to test / iterate on)
2. Exercises the spilling feature at different levels of memory pressure
3. Spends most of its time sorting/spilling/merging (not generating output for example)
### Describe alternatives you've considered
idea 1: can use some `datafusion-cli` features / flags and document them
Idea 2: Add a new suite to bench.sh / `dfbench`: https://github.com/apache/datafusion/tree/main/benchmarks
As for what to do I suggest something relatively simple like sorting the TPCH lineitem table with 200MB, 500MB, 1GB, 5GB and 10GB of memory for example
### Additional context
_No response_
Contributor guide
Research direction
Start by comparing the datafusion-cli approach with the benchmarks/bench.sh and dfbench options mentioned in the issue. Use the TPCH lineitem sorting example and test memory limits from 200MB through 10GB, ensuring the benchmark spends its time on sorting, spilling, and merging rather than output generation. Done means a documented, repeatable command or command set for measuring spilling-sort progress.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- data-engineering, performance, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100