[Story] Package our TPC benchmarks and run them in CI
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
We are doing significant work improving the performance of cudf_polars and cudf.pandas. That work needs to be reproducible. Internal teams like perflab need to run the benchmarks to validate results. The numbers show up in blogs, marketing material, conference talks, and customer conversations. When we publish results externally, people will want to reproduce them. We need a place to point them to, the same way the Velox team points people to rapidsai/velox-testing.
The benchmarks currently live inside the library packages:
```
python/cudf_polars/cudf_polars/streaming/benchmarks/
python/cudf/cudf/pandas/_benchmarks/
```
Not all benchmarks should move. libcudf has NVBenchmarks and cudf_polars has pytest-benchmark tests that measure individual functions. Those belong next to the code they test. These end-to-end workload benchmarks (TPC-H, TPC-DS) are different: they pull in tpchgen-cli and duckdb, run at scale, compare whole engines, and produce results for external audiences. They don't belong inside the library.
The story has four parts:
1. Add TPC-H instructions so people can reproduce our benchmarks today (#23025, #23087 done).
2. Add TPC-DS instructions for polars.
3. Move the benchmarks into a standalone `cudf_benchmarks` package so they can run without importing CUDA libraries (#23124).
4. Add a CI job so regressions are caught before they ship (#22868).
The CI job is the destination. We will not close this story until the benchmarks are running in CI.
Contributor guide
Assessment
This issue has not been assessed yet.