apache / apache/datafusion-comet

Benchmark automation

Open
#1,636 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

I have been spending significant time manually running benchmarks, both during development and when preparing to release Comet. I have been using a mix of open-source scripts and custom scripts. I want other contributors to be able to run the same benchmarks, both locally and in the cloud.

I recently documented [how to run benchmarks in AWS](https://datafusion.apache.org/comet/contributor-guide/benchmarking_aws_ec2.html) towards this goal.

My goals for this issue are:

1. Ensure that all scripts, configs, and documentation are located in the datafusion-comet repo so that anyone can run the same benchmarks by just cloning the repo and running a script
2. Provide recommended configs for different environments and scale factors
3. Have some subset of benchmarks run nightly against the main branch, with the results posted somewhere for later analysis and reporting (this could be as simple as pushing to a GitHub repo)

### Describe the potential solution

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.