NVIDIA / NVIDIA/cudf

Refactor cudf-polars test structure to avoid repeated test runs

Open
#19,708 3 comments 0 reactions 0 assignees View on GitHub
cudf-polars Python tests
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

tl/dr: I'd like to reorganize our cudf-polars tests to avoid running tests multiple times and be explicit about which tests are specific to some engine / executor / executor parameters.

Currently, we run cudf-polars' test suite multiple times with different configurations: https://github.com/rapidsai/cudf/blob/df567254b81fc8d8c6d8c4f636d1c288717b77c2/ci/run_cudf_polars_pytests.sh#L11-L24 (the in-memory executor and the streaming executor with different parameters (blocksize, scheduler)).

Broadly speaking: we have two types of unit tests in cudf-polars:

1. Engine-agnostic tests, which are insensitive to the particular engine they use. We have some polars expressions, and we expect to get the same output as polars regardless of the specific engine used. For example, [`test_non_coalesce_join`](https://github.com/rapidsai/cudf/blob/df567254b81fc8d8c6d8c4f636d1c288717b77c2/python/cudf_polars/tests/test_join.py#L71-L75).
2. Engine-specific tests, which are extremely sensitive to some engine parameters. These might be polars operations targeting coverage for a specific operation that are difficult or inappropriate to run with an arbitrary engine, or they might test specific implementation details of cudf-polars. For example

- [`test_unsupported_config_raises`](https://github.com/rapidsai/cudf/blob/df567254b81fc8d8c6d8c4f636d1c288717b77c2/python/cudf_polars/tests/test_config.py#L55-L59), which specifies a specific (invalid) `engine` to test some behavior of cudf-polars
- [`test_profile_basic`](https://github.com/rapidsai/cudf/blob/df567254b81fc8d8c6d8c4f636d1c288717b77c2/python/cudf_polars/tests/test_profile.py#L13-L28) which tests a behavior that's only implemented for the in-memory executor
- [`tests/experimental/test_join.py::test_join`](https://github.com/rapidsai/cudf/blob/branch-25.10/python/cudf_polars/tests/experimental/test_join.py#L43) which tests some streaming-specific options

Currently, my understanding is that we run essentially all of the tests multiple times, even the engine-specific tests. Here are the four times we ran that `test_join` test from a recent run, despite each run being identical (because it sets an `engine` within the test):

1. https://github.com/rapidsai/cudf/actions/runs/16995451715/job/48186665155#step:11:429 (from `python -m pytest --cache-clear "$@" tests --executor in-memory`)
2. https://github.com/rapidsai/cudf/actions/runs/16995451715/job/48186665155#step:11:610 (from `python -m pytest --cache-clear "$@" tests`)
3. https://github.com/rapidsai/cudf/actions/runs/16995451715/job/48186665155#step:11:791 (from `python -m pytest --cache-clear "$@" tests --executor streaming --blocksize-mode small`)
4. https://github.com/rapidsai/cudf/actions/runs/16995451715/job/48186665155#step:11:912 (from `python -m pytest --cache-clear "$@" "tests/experimental" --executor streaming --scheduler distributed`)

Concretely, I'd propose two changes:

1. Reorganize the test files: Move all the engine-agnostic tests to `python/cudf_polars/tests/common/...` and the engine-specific tests to `python/cudf_polars/tests/{engine}/...`. The streaming-engine-with-distributed-scheduler can be `tests/experimental` still, if we want.
2. Update how we invoke the tests to pass the paths that are relevant for that engine (the streaming executor would pass `tests/common tests/streaming`, the in-memory executor would pass `tests/common tests/in_memory`).

As an alternative to 2, we could go all-in on parametrized fixtures, and have all the `common` tests take an `engine` fixture, which would be parametrized overall all the engines we test. Then we'd invoke pytest just once. This has some consequences on things like coverage reporting (some of which are positive, others negative). If we want that we can pursue it, but for now I'd recommend keeping the multiple `pytest` invocations.

Contributor guide

Open the contributing guide

Research direction

Start with ci/run_cudf_polars_pytests.sh and the test files under python/cudf_polars/tests, including test_join.py, test_config.py, test_profile.py, and tests/experimental/test_join.py. Map engine-agnostic tests to tests/common and engine-specific tests to the relevant executor directories, then update each pytest invocation to use the appropriate paths. Done means the suite still covers the listed behaviors without rerunning engine-specific tests under unrelated executors.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, shell
Domain
ci-cd, testing
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.