[BUG]: libcu++ tests are flaky due to oversubscribing the GPU
- Dominant language
- C++
- Stars
- 2.5k
- Forks
- 487
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 296
Description
### Is this a duplicate?
- [X] I confirmed there appear to be no [duplicate issues](https://github.com/NVIDIA/cccl/issues) for this bug and that I agree to the [Code of Conduct](CODE_OF_CONDUCT.md)
### Type of Bug
Something else
### Component
libcu++
### Describe the bug
The libcu++ tests are executed in parallel as determined by `libcudacxx_LIT_PARALELL_LEVEL`: https://github.com/NVIDIA/cccl/blob/a833153c740a7ce208ae0bb07a4d16c8cd95e24e/libcudacxx/.upstream-tests/test/CMakeLists.txt#L136
This leads to issuing many kernels to the GPU concurrently which can oversubscribe the GPU and cause tests to fail. `lit` will attempt to rerun the test 5 times before it ultimately gives up and fails the test as `UNRESOLVED`, e.g., https://github.com/NVIDIA/cccl/actions/runs/6618224125/job/17976533031?pr=504#step:7:1608
Due to the nature of the failure, it is intermittent and effectively random which tests actually fail. This leads to a frustrating experience of CI runs randomly failing for no reason.
It's worth noting that I believe lit is using multiple processes for the build parallelism as opposed to multiple threads. If we're not using CUDA MPS, this can lead to significant overheads, so depending on the solution we land on, we should also investigate the impact of using multiple processes without MPS.
### How to Reproduce
Cannot be deterministically reproduced, but here is an example of the failure: https://github.com/NVIDIA/cccl/actions/runs/6618224125/job/17976533031?pr=504#step:7:1608
### Expected behavior
libcudacxx tests should consistently pass.
### Reproduction link
_No response_
### Operating System
_No response_
### nvidia-smi output
_No response_
### NVCC version
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.