NVIDIA / NVIDIA/cudf

[FEA] cudf-polars test suite CLI arguments for `NUM_RANKS` and `allow_gpu_sharing`

Open
#22,407 0 comments 0 reactions 0 assignees View on GitHub
cudf-polars feature request
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

`python/cudf_polars/tests/conftest.py` currently hardcodes both the rank count and the GPU-sharing flag for multi-rank streaming engines:

```python
# Number of ranks for multi-rank streaming engines that share one GPU
# (currently ``RayEngine``). Single-GPU dev hosts and CI runners require
# ``allow_gpu_sharing=True`` to oversubscribe one device across actors.
NUM_RANKS = 2
```

…and:

```python
engines["ray"] = RayEngine(
num_ranks=NUM_RANKS,
engine_options={"allow_gpu_sharing": True},
ray_init_options={"include_dashboard": False},
)
```

This configuration works for the initial single-GPU CI lane, but it hardcodes assumptions that belong to the test runner environment rather than library-level test setup.

### Proposed

Make both `NUM_RANKS` and `allow_gpu_sharing` configurable from outside `conftest.py`, while preserving the current behavior as the default.

Register pytest CLI options via `pytest_addoption`, expose them as session-scoped fixtures, and consume them from `streaming_engines`:

```bash
pytest --num-ranks=4 --no-gpu-sharing ...
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.