NVIDIA / NVIDIA/cudf

[FEA] Expand stream safety testing

Open
#21,922 0 comments 0 reactions 0 assignees View on GitHub
feature request libcudf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**
cudf's stream testing utilities are powerful, but insufficient to cover all the possible ways in which we could have hidden stream-ordering bugs. #21920 is one example of such a failure. We need to extend our existing testing to capture more possible failure modes.

**Describe the solution you'd like**
There are multiple things that we could do next. A couple of options:
1. We could run compute-sanitizer (with `track-stream-ordered-races=all`) while simultaneously setting `LD_PRELOAD=libcudf_identify_stream_usage_mode_cudf.so`. That would replace `cudf::get_default_stream()` with a version that returns a custom `rmm::cuda_stream`. This solution would help us catch various stream-ordering races that could be currently hidden by implicit synchronizations inside CUDA APIs like `cudaMallocAsync` that occur when using the CUDA default stream (the normal return value of `cudf::get_default_stream()`).
2. We could systematically update all libcudf tests to pass `cudf::test::get_default_stream()` to cudf APIs instead of relying on the default value of the `stream` parameter, then switch to `LD_PRELOAD=libcudf_identify_stream_usage_mode_testing.so`. That would provide even stricter validation of all our APIs since [the default check can only verify that we haven't passed one of CUDA's default streams](https://github.com/rapidsai/cudf/blob/4c1db9b874c9ab413c68007d21b8f735c7598e6b/cpp/tests/utilities/identify_stream_usage.cpp#L102). Using this approach everywhere would require us to generalize the test suite to support specifying exactly which streams should be allowed, but if we did so it would allow us to ensure that absolutely no work is queued on unexpected streams.
3. We could start adding more multistream tests. That would provide us testing of a case for which we have no significant test coverage today; we mostly rely on our downstream engines implemented on top of libcudf to find these errors.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.