apache / apache/datafusion

Benchmark / program to test Spilling Sorts

Open
#15,664 0 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

- Part of https://github.com/apache/datafusion/issues/15271

There are many interesting ideas on how to improve DataFusion while spilling for example https://github.com/apache/datafusion/issues/15271 from @2010YOUY01 and others.

What I think we really need next to make progress in this area is a benchmark / agreed upon way of measuring our progress so that we can improve and

### Describe the solution you'd like

I would like a documented command / set of commands that is:
1. Easy to run (and thus fast to test / iterate on)
2. Exercises the spilling feature at different levels of memory pressure
3. Spends most of its time sorting/spilling/merging (not generating output for example)

### Describe alternatives you've considered

idea 1: can use some `datafusion-cli` features / flags and document them

Idea 2: Add a new suite to bench.sh / `dfbench`: https://github.com/apache/datafusion/tree/main/benchmarks

As for what to do I suggest something relatively simple like sorting the TPCH lineitem table with 200MB, 500MB, 1GB, 5GB and 10GB of memory for example

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by comparing the datafusion-cli approach with the benchmarks/bench.sh and dfbench options mentioned in the issue. Use the TPCH lineitem sorting example and test memory limits from 200MB through 10GB, ensuring the benchmark spends its time on sorting, spilling, and merging rather than output generation. Done means a documented, repeatable command or command set for measuring spilling-sort progress.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
data-engineering, performance, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.