Some benchmarks take really long
- Dominant language
- C++
- Stars
- 2.5k
- Forks
- 486
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 295
Description
Running the entire CUB+Thrust benchmarks using `run.py` takes >4h on a B200 and thus requires at least 2 batch jobs. We should reduce this time a bit, since it also helps with producing regression benchmarks more quickly.
Here are the top offenders, where each compile-time workload took longer than 100s to run:
```
cub.bench.segmented_radix_sort.keys.base
cub.bench.segmented_sort.keys.base
cub.bench.topk.keys.base
cub.bench.topk.pairs.base
```
Full table: https://docs.google.com/spreadsheets/d/131YcBo2XpZXVn8CE7OAdh16cd16Ui6CfgL05V-BWrf4/edit?usp=sharing
The cumulative runtime of all segmented_radix_sort benchmarks is for example 136 minutes.
Sadly, the database does not keep around which compile-time workloads took the longest.
I think we should reduce the number of variants for those benchmarks.
Contributor guide
Assessment
This issue has not been assessed yet.