[DOC][cudf-polars] Repeatability of benchmarks at scale
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
It would be good to provide repeatable benchmarks at scale. For a myriad of reasons, it is difficult to ensure "perfect repeatability", including:
- Exact hardware may vary including, for example, storage setup.
- Software versions, in specific Kernel and CUDA driver versions, are sometimes beyond user's permissions to change, as is the case with shared clusters.
- Similarly, permissions for certain operations (such as dropping system cache) is often limited.
- Exact datasets used, and how to generate them when using synthetic data.
- Software configurations used.
At a minimum, we should provide the following:
- As much system information as possible
- Software: Kernel, CUDA and full RapidsMPF/cudf-polars stack (e.g., `conda list`/`pip list`).
- Specific hardware utilized: number of GPU and their exact models, interconnects available, storage used (NVMe, shared filesystems, etc.).
- Instructions to build the exact software stack used
- When possible, a complete Dockerfile using stable versions (i.e., avoid nightlies that are not reproducible); or
- How to build/install the exact environment (generally using instructions from [RAPIDS Installation Guide](https://docs.rapids.ai/install/)) for baremetal usage.
- Dataset retrieval or generation
- Provide sources/links where datasets can be retrieved from; or
- Scripts with exact parameters and minimum requirements to generate datasets with exact specifications used -- ideally derived from the instructions to generate image/environment.
- Spinning up the cluster and running benchmarks:
- Scripts with exact configurations (e.g., environment variables, where possible and relevant indicating _why_ the specific values were chosen) and command-lines with the exact arguments -- ideally derived from the instructions to generate image/environment.
Contributor guide
Assessment
This issue has not been assessed yet.