[EPIC] Add SQL based benchmarking
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
As mentioned in https://github.com/apache/datafusion/issues/21165 having sql based benchmarking would make creating some types of benchmarks much easier and would remove the necessity of writing rust code just to benchmark some sql. It could also allow us to migrate away from manual benchmarking towards using [criterion](https://github.com/bheisler/criterion.rs) or [Divan](https://github.com/nvzqz/divan) for running the benchmarks with proper statistics.
### Describe the solution you'd like
I would suggest that we should take inspiration from [DuckDB's benchmarking tool](https://github.com/duckdb/duckdb/tree/main/benchmark) and create a similar solution for Datafusion.
### Describe alternatives you've considered
Leave things as they are.
### Additional context
### Tasks
- [x] Framework: https://github.com/apache/datafusion/pull/21707
- [x] Port tpch benchmarks: https://github.com/apache/datafusion/pull/21707
- [x] Port clickbench benchmarks: https://github.com/apache/datafusion/pull/22633
- [x] Port h2o benchmark: https://github.com/apache/datafusion/pull/22660
- [x] Port hj benchmark: https://github.com/apache/datafusion/pull/22802
- [x] Port imdb benchmark: https://github.com/apache/datafusion/pull/22680
- [x] Port smj benchmark: https://github.com/apache/datafusion/pull/22803
- [x] Port nlj benchmark: https://github.com/apache/datafusion/pull/22805
- [x] Port tpcds benchmark: https://github.com/apache/datafusion/pull/22801
- [ ] Port sort_pushdown (4 variants)
- [ ] Add taxi benchmark
- [ ] https://github.com/apache/datafusion/issues/24312
- [ ] https://github.com/apache/datafusion/issues/24311
- [ ] https://github.com/apache/datafusion/issues/21935
- [ ] https://github.com/apache/datafusion/issues/21936
- [x] https://github.com/apache/datafusion/issues/21937
Contributor guide
Assessment
This issue has not been assessed yet.