apache / apache/datafusion

Remove old `dfbench` runner once benchmarks are migrated to the new SQL benchmark framework

Open
#24,312 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

- Part of https://github.com/apache/datafusion/issues/21706
- Follow up to https://github.com/apache/datafusion/issues/21937

Most benchmark suites have been ported to the new SQL based benchmark framework (`benchmark_runner`, see https://github.com/apache/datafusion/pull/21707 and follow on PRs). However, the old rust-based `dfbench` runner (`benchmarks/src/bin/dfbench.rs` and the per-suite modules in `benchmarks/src/`) still exists in parallel.

Maintaining two benchmark implementations means duplicated code and the risk that the two runners drift apart and report different results for the same suite.

### Describe the solution you'd like

Once `bench.sh` has been migrated to the new framework (https://github.com/apache/datafusion/issues/24311), remove the old `dfbench` runner and the now-unused suite code, keeping `dfbench` only for benchmarks that cannot be expressed as SQL (if any remain).

### Describe alternatives you've considered

Keep both runners indefinitely.

### Additional context

@Omega359 mentioned in https://github.com/apache/datafusion/issues/21937#issuecomment-5196560905 that they planned to file/work on this cleanup.

Contributor guide

Open the contributing guide

Research direction

Start with issue 24311 and inspect benchmarks/src/bin/dfbench.rs, the per-suite modules in benchmarks/src/, and bench.sh after its migration. Identify which suites are covered by the SQL benchmark framework and remove the old runner and unused suite code, retaining dfbench only for benchmarks that cannot be expressed as SQL.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.