Project-MONAI / Project-MONAI/MONAI
unit test errors with GPU that enables tf32
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.7k
- Forks
- 1.6k
- Avg merge
- 5d 1h
- Merged PRs (30d)
- 20
Description
When using a GPU that enables tf32, some of the unit tests will fails:
tests.test_dynunet.TestDynUNetWithInstanceNorm3dNVFusertests.test_bundle_download.TestLoadtests.test_multi_scale.TestMultiScaletests.test_integration_determinism
If disable tf32, all these tests are fine:

We can enlarge the tolerance to avoid this kind of issue, like this test does (https://github.com/Project-MONAI/MONAI/blob/f23077dafafa32117d1224807e0cf7d6cbbce995/tests/test_lltm.py#L20). Or could we consider a more generic way (if changing the tolerance, we need to check which number is the most suitable one case by case)? Such as do export NVIDIA_TF32_OVERRIDE=0 before running the test script. Hi @wyli @ericspod @Nic-Ma @mingxin-zheng , do you have any suggestions?
In addition, I also tried to disable tf32 within the python code, but I met an issue that it seems the way pytorch suggests does not work:

Do you have any opinions about this issue? Thanks!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the four listed tests on a TF32-enabled GPU and compare their failures with TF32 disabled. Inspect the tolerance approach in tests/test_lltm.py, then determine a consistent test configuration or tolerance strategy; done means the affected tests pass reliably under the reported GPU conditions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100