[BUG]: cuda.parallel cluster of silent failures in scan and unique_by_key algorithms
- Dominant language
- C++
- Stars
- 2.5k
- Forks
- 486
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 295
Description
### Is this a duplicate?
- [x] I confirmed there appear to be no [duplicate issues](https://github.com/NVIDIA/cccl/issues) for this bug and that I agree to the [Code of Conduct](CODE_OF_CONDUCT.md)
### Type of Bug
Silent Failure
### Component
cuda.parallel (Python)
### Describe the bug
Apply change:
```
diff --git a/c/parallel/src/nvrtc/command_list.h b/c/parallel/src/nvrtc/command_list.h
index c1854cea4..d6977a9f6 100644
--- a/c/parallel/src/nvrtc/command_list.h
+++ b/c/parallel/src/nvrtc/command_list.h
@@ -177,6 +177,7 @@ struct nvrtc_command_list_visitor
auto result = nvJitLinkGetLinkedCubinSize(jitlink.handle, &cleanup.link_result_ref.size);
if (result != NVJITLINK_SUCCESS)
{
+ printf("Could not get CUBIN size, status: %d\n", result);
output_ptx = true;
check(nvJitLinkGetLinkedPtxSize(jitlink.handle, &cleanup.link_result_ref.size));
}
```
to inform when CUBIN could not be obtained when linking all cuda.parallel-generated pieces together in c.parallel library.
Running the test suite in the branch #4249:
```
====================================================================================================================== test session starts ======================================================================================================================
platform linux -- Python 3.10.12, pytest-8.3.5, pluggy-1.5.0
benchmark: 5.1.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/coder/cccl/python/cuda_parallel
configfile: pyproject.toml
plugins: benchmark-5.1.0, xdist-3.6.1
collected 965 items
tests/test_bindings.py ....................................................
tests/test_iterators.py ...............................
tests/test_merge_sort.py .......................................................................................................................................................................................................................................................................................................................................................................................................................................
tests/test_merge_sort_api.py .
tests/test_radix_sort.py .............................................................................................................................................
tests/test_radix_sort_api.py ..
tests/test_reduce.py ...........................................................................................................
tests/test_reduce_api.py .......
tests/test_scan.py Could not get CUBIN size, status: 6
.Could not get CUBIN size, status: 6
.Could not get CUBIN size, status: 6
.Could not get CUBIN size, status: 6
.Could not get CUBIN size, status: 6
.Could not get CUBIN size, status: 6
.Could not get CUBIN size, status: 6
.Could not get CUBIN size, status: 6
.Could not get CUBIN size, status: 6
.Could not get CUBIN size, status: 6
.Could not get CUBIN size, status: 6
.Could not get CUBIN size, status: 6
.Could not get CUBIN size, status: 6
.Could not get CUBIN size, status: 6
.Could not get CUBIN size, status: 6
.Could not get CUBIN size, status: 6
.Could not get CUBIN size, status: 6
.Could not get CUBIN size, status: 6
.Could not get CUBIN size, status: 6
.Could not get CUBIN size, status: 6
.Could not get CUBIN size, status: 6
.Could not get CUBIN size, status: 6
.Could not get CUBIN size, status: 6
.Could not get CUBIN size, status: 6
.Could not get CUBIN size, status: 6
.Could not get CUBIN size, status: 6
.Could not get CUBIN size, status: 6
.Could not get CUBIN size, status: 6
.Could not get CUBIN size, status: 6
.Could not get CUBIN size, status: 6
...
tests/test_scan_api.py Could not get CUBIN size, status: 6
..Could not get CUBIN size, status: 6
.Could not get CUBIN size, status: 6
.
tests/test_segmented_reduce.py .................................................
tests/test_segmented_reduce_api.py ..
tests/test_transform.py ..............................
tests/test_unique_by_key.py Could not get CUBIN size, status: 6
....Could not get CUBIN size, status: 6
....Could not get CUBIN size, status: 6
....Could not get CUBIN size, status: 6
....Could not get CUBIN size, status: 6
....Could not get CUBIN size, status: 6
....Could not get CUBIN size, status: 6
....Could not get CUBIN size, status: 6
....Could not get CUBIN size, status: 6
....Could not get CUBIN size, status: 6
....Could not get CUBIN size, status: 6
....Could not get CUBIN size, status: 6
....Could not get CUBIN size, status: 6
....Could not get CUBIN size, status: 6
....Could not get CUBIN size, status: 6
....Could not get CUBIN size, status: 6
....Could not get CUBIN size, status: 6
....Could not get CUBIN size, status: 6
....Could not get CUBIN size, status: 6
....Could not get CUBIN size, status: 6
....Could not get CUBIN size, status: 6
.Could not get CUBIN size, status: 6
..
tests/test_unique_by_key_api.py Could not get CUBIN size, status: 6
.
```
These failures are concentrated in SCAN and UNIQUE_BY_KEY algorithms.
I see that these failures affect **both** inclusive and exclusive scan algorithms:
```
(cuda_parallel_venv) (cccl) coder ➜ ~/cccl/python/cuda_parallel $ pytest -sv tests/test_scan.py::test_scan_array_input[int32-True]
====================================================================================================================== test session starts ======================================================================================================================
platform linux -- Python 3.10.12, pytest-8.3.5, pluggy-1.5.0 -- /tmp/cuda_parallel_venv/bin/python
cachedir: .pytest_cache
benchmark: 5.1.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/coder/cccl/python/cuda_parallel
configfile: pyproject.toml
plugins: benchmark-5.1.0, xdist-3.6.1
collected 1 item
tests/test_scan.py::test_scan_array_input[int32-True] inclusive
Could not get CUBIN size, status: 6
PASSED
======================================================================================================================= 1 passed in 2.32s =======================================================================================================================
(cuda_parallel_venv) (cccl) coder ➜ ~/cccl/python/cuda_parallel $ pytest -sv tests/test_scan.py::test_scan_array_input[int32-False]
====================================================================================================================== test session starts ======================================================================================================================
platform linux -- Python 3.10.12, pytest-8.3.5, pluggy-1.5.0 -- /tmp/cuda_parallel_venv/bin/python
cachedir: .pytest_cache
benchmark: 5.1.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/coder/cccl/python/cuda_parallel
configfile: pyproject.toml
plugins: benchmark-5.1.0, xdist-3.6.1
collected 1 item
tests/test_scan.py::test_scan_array_input[int32-False] exclusive
Could not get CUBIN size, status: 6
PASSED
======================================================================================================================= 1 passed in 2.30s =======================================================================================================================
```
This latter finding is also reproducible in the main branch.
### How to Reproduce
Apply the patch as shows above, and run `pytest -s tests` from `python/cuda_parallel` folder.
### Expected behavior
The nvJitLinkError reported map to `NVJITLINK_ERROR_INTERNAL` and are unexpected.
They are not causing failure due to silent fall-back to produce PTX instead of SASS, but they cause reattempted compilation to SASS and ideally should be fixed
### Reproduction link
_No response_
### Operating System
Ubuntu 22.04
### nvidia-smi output
```
(cuda_parallel_venv) (cccl) coder ➜ ~/cccl/python/cuda_parallel $ nvidia-smi
Fri Apr 25 20:30:05 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.124.06 Driver Version: 570.124.06 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA RTX A6000 Off | 00000000:01:00.0 Off | Off |
| 31% 44C P5 38W / 300W | 1045MiB / 49140MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA RTX A400 Off | 00000000:C1:00.0 On | N/A |
| 30% 37C P5 N/A / 50W | 132MiB / 4094MiB | 17% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
+-----------------------------------------------------------------------------------------+
```
### NVCC version
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.