NVIDIA / NVIDIA/cccl

[BUG]: libcu++ tests are flaky due to oversubscribing the GPU

Open
#603 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
2.5k
Forks
487
Avg merge
2d 7h
Merged PRs (30d)
296

Description

### Is this a duplicate?

- [X] I confirmed there appear to be no [duplicate issues](https://github.com/NVIDIA/cccl/issues) for this bug and that I agree to the [Code of Conduct](CODE_OF_CONDUCT.md)

### Type of Bug

Something else

### Component

libcu++

### Describe the bug

The libcu++ tests are executed in parallel as determined by `libcudacxx_LIT_PARALELL_LEVEL`: https://github.com/NVIDIA/cccl/blob/a833153c740a7ce208ae0bb07a4d16c8cd95e24e/libcudacxx/.upstream-tests/test/CMakeLists.txt#L136

This leads to issuing many kernels to the GPU concurrently which can oversubscribe the GPU and cause tests to fail. `lit` will attempt to rerun the test 5 times before it ultimately gives up and fails the test as `UNRESOLVED`, e.g., https://github.com/NVIDIA/cccl/actions/runs/6618224125/job/17976533031?pr=504#step:7:1608

Due to the nature of the failure, it is intermittent and effectively random which tests actually fail. This leads to a frustrating experience of CI runs randomly failing for no reason.

It's worth noting that I believe lit is using multiple processes for the build parallelism as opposed to multiple threads. If we're not using CUDA MPS, this can lead to significant overheads, so depending on the solution we land on, we should also investigate the impact of using multiple processes without MPS.

### How to Reproduce

Cannot be deterministically reproduced, but here is an example of the failure: https://github.com/NVIDIA/cccl/actions/runs/6618224125/job/17976533031?pr=504#step:7:1608

### Expected behavior

libcudacxx tests should consistently pass.

### Reproduction link

_No response_

### Operating System

_No response_

### nvidia-smi output

_No response_

### NVCC version

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.