NVIDIA / NVIDIA/cccl

[BUG]: Benchmark `search.py` does not scan all type axes

Open
#2,267 0 comments 1 reaction 1 assignee Claimed by @gevtushenko View on GitHub
Dominant language
C++
Stars
2.5k
Forks
486
Avg merge
2d 6h
Merged PRs (30d)
295

Description

### Is this a duplicate?

- [X] I confirmed there appear to be no [duplicate issues](https://github.com/NVIDIA/cccl/issues) for this bug and that I agree to the [Code of Conduct](CODE_OF_CONDUCT.md)

### Type of Bug

Silent Failure

### Component

CUB

### Describe the bug

I run the BabelStream transform CUB benchmarks from my PR #2086 as follows:
```
../benchmarks/scripts/search.py -R 'cub.bench.transform.babelstream'
```
but it only benchmarks the type `I8`. Also explicitly providing the type axis with `a 'T{ct}=[I8,I16,F32,F64,I128]` only runs the benchmarks for `I8`.

### How to Reproduce

Clone #2086
```
mkdir build
cd build
cmake -DCUB_ENABLE_BENCHMARKS=YES\
-DCUB_ENABLE_TUNING=YES\
-DCMAKE_CUDA_ARCHITECTURES='86'\
-DCMAKE_BUILD_TYPE=Release\
-DCCCL_ENABLE_CUB=YES\
-DCCCL_ENABLE_THRUST=NO\
-DCCCL_ENABLE_LIBCUDACXX=NO\
-GNinja ..
../benchmarks/scripts/search.py -R 'cub.bench.transform.babelstream'
```
Alternatively:
```
../benchmarks/scripts/search.py -R 'cub.bench.transform.babelstream' -a 'T{ct}=[I8,I16,F32,F64,I128]'
```
After the benchmark completes, the coverage shows only:
```
../benchmarks/scripts/analyze.py --coverage cccl_meta_bench.db
cub.bench.transform.babelstream[T{ct}=I8] coverage: 21 / 20 (100.0000%)
```

### Expected behavior

All types specified in the benchmarks type axis `[I8,I16,F32,F64,I128]` should be run.

### Reproduction link

_No response_

### Operating System

_No response_

### nvidia-smi output

_No response_

### NVCC version

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.