Project-MONAI / Project-MONAI/MONAI

unit test errors with GPU that enables tf32

Open
#5,577 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
8.7k
Forks
1.6k
Avg merge
5d 1h
Merged PRs (30d)
20

Description

When using a GPU that enables tf32, some of the unit tests will fails:

  1. tests.test_dynunet.TestDynUNetWithInstanceNorm3dNVFuser
  2. tests.test_bundle_download.TestLoad
  3. tests.test_multi_scale.TestMultiScale
  4. tests.test_integration_determinism

If disable tf32, all these tests are fine:

check_tf32

We can enlarge the tolerance to avoid this kind of issue, like this test does (https://github.com/Project-MONAI/MONAI/blob/f23077dafafa32117d1224807e0cf7d6cbbce995/tests/test_lltm.py#L20). Or could we consider a more generic way (if changing the tolerance, we need to check which number is the most suitable one case by case)? Such as do export NVIDIA_TF32_OVERRIDE=0 before running the test script. Hi @wyli @ericspod @Nic-Ma @mingxin-zheng , do you have any suggestions?

In addition, I also tried to disable tf32 within the python code, but I met an issue that it seems the way pytorch suggests does not work:

close_tf32

Do you have any opinions about this issue? Thanks!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the four listed tests on a TF32-enabled GPU and compare their failures with TF32 disabled. Inspect the tolerance approach in tests/test_lltm.py, then determine a consistent test configuration or tolerance strategy; done means the affected tests pass reliably under the reported GPU conditions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.