[FEA] Expand stream safety testing
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
cudf's stream testing utilities are powerful, but insufficient to cover all the possible ways in which we could have hidden stream-ordering bugs. #21920 is one example of such a failure. We need to extend our existing testing to capture more possible failure modes.
**Describe the solution you'd like**
There are multiple things that we could do next. A couple of options:
1. We could run compute-sanitizer (with `track-stream-ordered-races=all`) while simultaneously setting `LD_PRELOAD=libcudf_identify_stream_usage_mode_cudf.so`. That would replace `cudf::get_default_stream()` with a version that returns a custom `rmm::cuda_stream`. This solution would help us catch various stream-ordering races that could be currently hidden by implicit synchronizations inside CUDA APIs like `cudaMallocAsync` that occur when using the CUDA default stream (the normal return value of `cudf::get_default_stream()`).
2. We could systematically update all libcudf tests to pass `cudf::test::get_default_stream()` to cudf APIs instead of relying on the default value of the `stream` parameter, then switch to `LD_PRELOAD=libcudf_identify_stream_usage_mode_testing.so`. That would provide even stricter validation of all our APIs since [the default check can only verify that we haven't passed one of CUDA's default streams](https://github.com/rapidsai/cudf/blob/4c1db9b874c9ab413c68007d21b8f735c7598e6b/cpp/tests/utilities/identify_stream_usage.cpp#L102). Using this approach everywhere would require us to generalize the test suite to support specifying exactly which streams should be allowed, but if we did so it would allow us to ensure that absolutely no work is queued on unexpected streams.
3. We could start adding more multistream tests. That would provide us testing of a case for which we have no significant test coverage today; we mostly rely on our downstream engines implemented on top of libcudf to find these errors.
Contributor guide
Assessment
This issue has not been assessed yet.