Project-MONAI / Project-MONAI/MONAI

MetaTensor and DistributedDataParallel. bug (SyncBatchNormBackward is a view and is being modified inplace)

Open
#5,283 4 comments 0 reactions 2 assignees View on GitHub

@Nic-Ma is already working on this.

Since Oct 7, 2022.

bug
Dominant language
Python
Stars
8.7k
Forks
1.6k
Avg merge
5d 1h
Merged PRs (30d)
20

Description

When upgrading from MONAI 0.9.0 to 1.0.0, my 3D segmentation code fails due to (most likely) new MetaTensor in transforms, when using DistributedDataParallel (multi-gpu)

the error is
RuntimeError: Output 0 of SyncBatchNormBackward is a view and is being modified inplace. This view was created inside a custom Function (or because an input was returned as-is) and the autograd logic to handle view+inplace would override the custom backward associated with the custom Function, leading to incorrect gradients. This behavior is forbidden. You can fix this by cloning the output of the custom Function.

same issue was reported here (but for 2D MIL classification)
https://github.com/Project-MONAI/MONAI/discussions/5081
and https://github.com/Project-MONAI/MONAI/issues/5198

I've traced it down to this commit
https://github.com/Project-MONAI/MONAI/pull/4548/commits/63e36b6cb41e163024729010534cd9363c6356dc
(prior to it, the code is working fine)

It seems the issue is that dataloader returns data as MetaTensor (and not torch.Tensor as before)
e.g. here https://github.com/Project-MONAI/tutorials/blob/main/pathology/multiple_instance_learning/panda_mil_train_evaluate_pytorch_gpu.py#L51
both data and target are MetaTensor types

if converting explicitly (on gpu or cpu):

data = torch.Tensor(data)
target = torch.Tensor(target)

then the code runs fine, but a bit slower. It seems there is something wrong with MetaTensor

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.