CUDA 13.4: nogpu CPU test configs disabled due to libcufft valgrind failures
- Dominant language
- Python
- Stars
- 103k
- Forks
- 29.5k
- PR merge metrics
- PR metrics pending
Description
Filing this to track CPU test coverage that had to be dropped from the new CUDA 13.4
periodic job in #190639.
### Summary
The `nogpu_AVX512` and `nogpu_NO_AVX2` configs are omitted from the new
`linux-jammy-cuda13.4-py3.10-gcc11` periodic job added in #190639. They run the cpp
tests under valgrind, and CUDA 13.4's `libcufft.so.12.4.0.16` reads uninitialised
values while initialising itself, before `main`. valgrind runs with
`--error-exitcode=1`, so those reports fail the shard.
https://github.com/pytorch/pytorch/actions/runs/31058413887/job/92485507594
These two configs were carrying essentially all of the job's test coverage. They fall
through to the default branch of `.ci/pytorch/test.sh`, which runs `test_python`,
`test_aten`, `test_vec256`, `test_libtorch`, `test_aot_compilation`,
`test_custom_script_ops`, `test_custom_backend`, `test_torch_function_benchmark` and
`test_benchmarks`. The remaining `jit_legacy` config runs only `test_jit_legacy` and `test_jit_fuser_legacy`.
So CUDA 13.4 currently has build validation but almost no test coverage.
### Details
`aten/tools/run_tests.sh` runs the gtest binary directly under valgrind:
```bash
valgrind --suppressions="$VALGRIND_SUP" --error-exitcode=1 "${CPP_TESTS_DIR}/basic" --gtest_filter='-*CUDA'
```
Output on CUDA 13.4:
```
==132146== Conditional jump or move depends on uninitialised value(s)
==132146== at 0x2EED9C8A: ??? (in /usr/local/cuda-13.4/targets/x86_64-linux/lib/libcufft.so.12.4.0.16)
==132146== by 0x2EE7E9E9: ??? (in .../libcufft.so.12.4.0.16)
==132146== by 0x21FA0EA7: __pthread_once_slow (pthread_once.c:116)
==132146== by 0x400647D: call_init.part.0 (dl-init.c:70)
==132146== by 0x4006567: _dl_init (dl-init.c:117)
...
==132146== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 21 from 5)
```
All 7 gtest cases pass; only the valgrind exit code fails the shard.
### Notes
Every frame is inside the closed-source library during dynamic-library init, so there is nothing to fix in PyTorch.
- `libcufft` loads even in `nogpu` shards because it is linked into `libtorch_cuda`.
- The `cuda13.0` and `cuda13.2` periodic jobs run these same configs and pass, so this is specific to 13.4.
- Only `periodic.yml` runs `nogpu` configs; no other workflow is affected.
### Options
1. Add a `Cond_cufft` suppression to `aten/tools/valgrind.sup` next to the existing `libcuda` ones (cf. #34169, #48187).
2. Set `VALGRIND=OFF` for cuda13.4 in `.ci/pytorch/test.sh`, as already done for clang9/xpu/s390x/rocm/aarch64.
3. Wait for a cuFFT release without these reads.
Restore the two configs in `periodic.yml` once resolved.
cc @ptrblck @msaroufim @eqy @nWEIdia @malfet @pytorch/pytorch-dev-infra @ngimel
Contributor guide
Research direction
Start by comparing the CUDA 13.4 entry in .ci/periodic.yml with the CUDA 13.0 and 13.2 jobs, then read .ci/pytorch/test.sh and aten/tools/valgrind.sup. Determine which listed mitigation is appropriate for the libcufft initialization reports. Done means the nogpu_AVX512 and nogpu_NO_AVX2 configs are restored in periodic.yml and the CUDA 13.4 periodic tests pass without masking unrelated Valgrind failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, shell, yaml
- Domain
- ci-cd, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100