Project-MONAI / Project-MONAI/MONAI
MetaTensor and DistributedDataParallel. bug (SyncBatchNormBackward is a view and is being modified inplace)
@Nic-Ma is already working on this.
Since Oct 7, 2022.
- Dominant language
- Python
- Stars
- 8.7k
- Forks
- 1.6k
- Avg merge
- 5d 1h
- Merged PRs (30d)
- 20
Description
When upgrading from MONAI 0.9.0 to 1.0.0, my 3D segmentation code fails due to (most likely) new MetaTensor in transforms, when using DistributedDataParallel (multi-gpu)
the error is
RuntimeError: Output 0 of SyncBatchNormBackward is a view and is being modified inplace. This view was created inside a custom Function (or because an input was returned as-is) and the autograd logic to handle view+inplace would override the custom backward associated with the custom Function, leading to incorrect gradients. This behavior is forbidden. You can fix this by cloning the output of the custom Function.
same issue was reported here (but for 2D MIL classification)
https://github.com/Project-MONAI/MONAI/discussions/5081
and https://github.com/Project-MONAI/MONAI/issues/5198
I've traced it down to this commit
https://github.com/Project-MONAI/MONAI/pull/4548/commits/63e36b6cb41e163024729010534cd9363c6356dc
(prior to it, the code is working fine)
It seems the issue is that dataloader returns data as MetaTensor (and not torch.Tensor as before)
e.g. here https://github.com/Project-MONAI/tutorials/blob/main/pathology/multiple_instance_learning/panda_mil_train_evaluate_pytorch_gpu.py#L51
both data and target are MetaTensor types
if converting explicitly (on gpu or cpu):
data = torch.Tensor(data)
target = torch.Tensor(target)
then the code runs fine, but a bit slower. It seems there is something wrong with MetaTensor
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.