pytorch / pytorch/pytorch

CUDA 13.4: nogpu CPU test configs disabled due to libcufft valgrind failures

Open
#192,377 1 comment 0 reactions 0 assignees View on GitHub
bot-triaged module: ci module: cuda module: dependency bug triaged
Dominant language
Python
Stars
103k
Forks
29.5k
PR merge metrics
PR metrics pending

Description

Filing this to track CPU test coverage that had to be dropped from the new CUDA 13.4
periodic job in #190639.

### Summary

The `nogpu_AVX512` and `nogpu_NO_AVX2` configs are omitted from the new
`linux-jammy-cuda13.4-py3.10-gcc11` periodic job added in #190639. They run the cpp
tests under valgrind, and CUDA 13.4's `libcufft.so.12.4.0.16` reads uninitialised
values while initialising itself, before `main`. valgrind runs with
`--error-exitcode=1`, so those reports fail the shard.
https://github.com/pytorch/pytorch/actions/runs/31058413887/job/92485507594

These two configs were carrying essentially all of the job's test coverage. They fall
through to the default branch of `.ci/pytorch/test.sh`, which runs `test_python`,
`test_aten`, `test_vec256`, `test_libtorch`, `test_aot_compilation`,
`test_custom_script_ops`, `test_custom_backend`, `test_torch_function_benchmark` and
`test_benchmarks`. The remaining `jit_legacy` config runs only `test_jit_legacy` and `test_jit_fuser_legacy`.

So CUDA 13.4 currently has build validation but almost no test coverage.

### Details

`aten/tools/run_tests.sh` runs the gtest binary directly under valgrind:

```bash
valgrind --suppressions="$VALGRIND_SUP" --error-exitcode=1 "${CPP_TESTS_DIR}/basic" --gtest_filter='-*CUDA'
```

Output on CUDA 13.4:

```
==132146== Conditional jump or move depends on uninitialised value(s)
==132146== at 0x2EED9C8A: ??? (in /usr/local/cuda-13.4/targets/x86_64-linux/lib/libcufft.so.12.4.0.16)
==132146== by 0x2EE7E9E9: ??? (in .../libcufft.so.12.4.0.16)
==132146== by 0x21FA0EA7: __pthread_once_slow (pthread_once.c:116)
==132146== by 0x400647D: call_init.part.0 (dl-init.c:70)
==132146== by 0x4006567: _dl_init (dl-init.c:117)
...
==132146== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 21 from 5)
```

All 7 gtest cases pass; only the valgrind exit code fails the shard.

### Notes
Every frame is inside the closed-source library during dynamic-library init, so there is nothing to fix in PyTorch.
- `libcufft` loads even in `nogpu` shards because it is linked into `libtorch_cuda`.
- The `cuda13.0` and `cuda13.2` periodic jobs run these same configs and pass, so this is specific to 13.4.
- Only `periodic.yml` runs `nogpu` configs; no other workflow is affected.

### Options

1. Add a `Cond_cufft` suppression to `aten/tools/valgrind.sup` next to the existing `libcuda` ones (cf. #34169, #48187).
2. Set `VALGRIND=OFF` for cuda13.4 in `.ci/pytorch/test.sh`, as already done for clang9/xpu/s390x/rocm/aarch64.
3. Wait for a cuFFT release without these reads.

Restore the two configs in `periodic.yml` once resolved.

cc @ptrblck @msaroufim @eqy @nWEIdia @malfet @pytorch/pytorch-dev-infra @ngimel

Contributor guide

Open the contributing guide

Research direction

Start by comparing the CUDA 13.4 entry in .ci/periodic.yml with the CUDA 13.0 and 13.2 jobs, then read .ci/pytorch/test.sh and aten/tools/valgrind.sup. Determine which listed mitigation is appropriate for the libcufft initialization reports. Done means the nogpu_AVX512 and nogpu_NO_AVX2 configs are restored in periodic.yml and the CUDA 13.4 periodic tests pass without masking unrelated Valgrind failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, shell, yaml
Domain
ci-cd, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.