Lightning-AI / Lightning-AI/lightning-thunder
Disable Tensor Cores for some consistency tests
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 121
- PR merge metrics
- No merged PRs in 30d
Description
## 🚀Disable `TF32` cores for some consistency tests when `cuda` is available
Local test suite may fail if the env variable for `NVIDIA_TF32_OVERRIDE` is not disabled (`NVIDIA_TF32_OVERRIDE = 0`).
Some examples on RTX ADA 6000:
```
FAILED thunder/tests/test_grad.py::test_phantom_grad_vs_torch_consistency_matmul_torch_cuda_thunder.dtypes.float32 - AssertionError: Tensor-likes are not close!
FAILED thunder/tests/test_grad.py::test_phantom_grad_vs_torch_consistency_matmul_nvfuser_cuda_thunder.dtypes.float32 - AssertionError: Tensor-likes are not close!
FAILED thunder/tests/test_grad.py::test_phantom_grad_vs_torch_consistency_einsum_torch_cuda_thunder.dtypes.float32 - AssertionError: Tensor-likes are not close!
FAILED thunder/tests/test_grad.py::test_phantom_grad_vs_torch_consistency_einsum_nvfuser_cuda_thunder.dtypes.float32 - AssertionError: Tensor-likes are not close!
```
cuBLAS may speed up `float32` ops on Tensor Cores.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in thunder/tests/test_grad.py, focusing on the listed matmul and einsum consistency tests and their CUDA setup. Reproduce the failures with CUDA available while checking NVIDIA_TF32_OVERRIDE, then confirm the affected consistency tests pass without Tensor Core TF32 behavior causing mismatches.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100