NVIDIA / NVIDIA/cudf

[DOC][cudf-polars] Repeatability of benchmarks at scale

Open
#22,892 1 comment 0 reactions 0 assignees View on GitHub
cudf-polars doc
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

It would be good to provide repeatable benchmarks at scale. For a myriad of reasons, it is difficult to ensure "perfect repeatability", including:
- Exact hardware may vary including, for example, storage setup.
- Software versions, in specific Kernel and CUDA driver versions, are sometimes beyond user's permissions to change, as is the case with shared clusters.
- Similarly, permissions for certain operations (such as dropping system cache) is often limited.
- Exact datasets used, and how to generate them when using synthetic data.
- Software configurations used.

At a minimum, we should provide the following:
- As much system information as possible
- Software: Kernel, CUDA and full RapidsMPF/cudf-polars stack (e.g., `conda list`/`pip list`).
- Specific hardware utilized: number of GPU and their exact models, interconnects available, storage used (NVMe, shared filesystems, etc.).
- Instructions to build the exact software stack used
- When possible, a complete Dockerfile using stable versions (i.e., avoid nightlies that are not reproducible); or
- How to build/install the exact environment (generally using instructions from [RAPIDS Installation Guide](https://docs.rapids.ai/install/)) for baremetal usage.
- Dataset retrieval or generation
- Provide sources/links where datasets can be retrieved from; or
- Scripts with exact parameters and minimum requirements to generate datasets with exact specifications used -- ideally derived from the instructions to generate image/environment.
- Spinning up the cluster and running benchmarks:
- Scripts with exact configurations (e.g., environment variables, where possible and relevant indicating _why_ the specific values were chosen) and command-lines with the exact arguments -- ideally derived from the instructions to generate image/environment.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.