apache / apache/datafusion

[EPIC] Add SQL based benchmarking

Open
#21,706 3 comments 3 reactions 1 assignee Claimed by @Omega359 View on GitHub
enhancement EPIC
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

As mentioned in https://github.com/apache/datafusion/issues/21165 having sql based benchmarking would make creating some types of benchmarks much easier and would remove the necessity of writing rust code just to benchmark some sql. It could also allow us to migrate away from manual benchmarking towards using [criterion](https://github.com/bheisler/criterion.rs) or [Divan](https://github.com/nvzqz/divan) for running the benchmarks with proper statistics.

### Describe the solution you'd like

I would suggest that we should take inspiration from [DuckDB's benchmarking tool](https://github.com/duckdb/duckdb/tree/main/benchmark) and create a similar solution for Datafusion.

### Describe alternatives you've considered

Leave things as they are.

### Additional context

### Tasks
- [x] Framework: https://github.com/apache/datafusion/pull/21707
- [x] Port tpch benchmarks: https://github.com/apache/datafusion/pull/21707
- [x] Port clickbench benchmarks: https://github.com/apache/datafusion/pull/22633
- [x] Port h2o benchmark: https://github.com/apache/datafusion/pull/22660
- [x] Port hj benchmark: https://github.com/apache/datafusion/pull/22802
- [x] Port imdb benchmark: https://github.com/apache/datafusion/pull/22680
- [x] Port smj benchmark: https://github.com/apache/datafusion/pull/22803
- [x] Port nlj benchmark: https://github.com/apache/datafusion/pull/22805
- [x] Port tpcds benchmark: https://github.com/apache/datafusion/pull/22801
- [ ] Port sort_pushdown (4 variants)
- [ ] Add taxi benchmark
- [ ] https://github.com/apache/datafusion/issues/24312
- [ ] https://github.com/apache/datafusion/issues/24311
- [ ] https://github.com/apache/datafusion/issues/21935
- [ ] https://github.com/apache/datafusion/issues/21936
- [x] https://github.com/apache/datafusion/issues/21937

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.