[EPIC] CI benchmarking MVP
- Dominant language
- C++
- Stars
- 2.5k
- Forks
- 486
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 295
Description
## Overview
### Main Goals
- Allow a subset of regression benchmarks to be run as part of our PR workflow.
- Perform full benchmark regression testing as part of our weekly CI.
- Initially get everything working for CUB, then extend to python.
- Add SASS extraction + hashing to CCCL benchmarking framework / databases. Skip running benchmarks if SASS is unchanged.
### Stretch Goals
- Expose a github action that be used to run an arbitrary number of benchmarks on two SHAs and compare them.
- Integrate with the Git Bisect scripting / action for locating regression SHA.
- Allow comparison of python results against C++ results.
## Considerations
- Benchmark results are not stable enough to reliably detect regressions from one system to the next, even with "identical" hardware. It is important that the baseline and testing benchmarks are run back-to-back on the same node within the same job.
- Locked clocks are not necessary -- NVBench's throttle detection provides a better representation of 'real' performance characteristics.
- Comparing SASS is faster and more accurate than running a benchmark. The benchmark harness should extract the SASS from the benchmark executables, generate hashes, and compare these to determine if new benchmark measurements are needed.
## Implementation Plan
- [ ] Add SASS extraction / hashing to the CCCL benchmarking framework. The hash should be stored in the database, and screened to determine if benchmarks need to run.
- [ ] A "pre-check" mode will be added that compiles and compares the SASS hashes and prints out a list of all benchmarks that have changes to the hash. This mode may be used in CI job on non-GPU machines to build benchmarking artifacts and determine which, if any, benchmarks need to be executed in a followup GPU job.
- [ ] Write a script to compare two SHAs: `ci/bench_cub.sh [common options] [-base ] [-test ] [-file [-- ]`. Use [last release] if base is omitted, and `HEAD` if test is omitted. Run the full suite if file is missing. This will checkout `base` and run the selected benchmarks, storing the full jsonbin output, repeat for `test`, and compare the two. Include the pre-check mode and execution mode.
- [ ] Add 'bench' jobs to `ci/matrix.yaml` that will invoke the `ci/bench_*` scripts, using the existing `args:` tag to specify customizations. This will allow PRs to trigger benchmarks using the existing `override:` machinery, and allow us to add benchmarking jobs to the nightly/weekly matrices easily.
- [ ] Provide documentation and examples for all.
## Stretch
- [ ] Interface `git_bisect.sh` efficiently with the benchmarking scripts.
- [ ] Incorporate SASS checking as a pre-benchmarking step.
- [ ] Add a 'prebuild' step that will compile all needed benchmark executables, then pass them to the GPU runner as an artifact or via sccache.
Contributor guide
Assessment
This issue has not been assessed yet.