Project-MONAI / Project-MONAI/MONAILabel

torch.cuda.OutOfMemoryError & RuntimeError: applying transform

Open
#1,351 9 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
891
Forks
269
Avg merge
15h 41m
Merged PRs (30d)
1

Description

Describe the bug
Before I got to update my cudatoolkit 11.8 MonaiLabel was not recognizing cudatoolkit 11.3 and cuda was disabling. The training process was ran in CPU mode and was running fine. After I got to update cudatoolkit 11.8 and then cuda is not getting disabled but I am getting - RuntimeError: applying transform <monai.transforms.compose.Compose object at 0x0000014188C0B160>.
When I go through then I found the **_"torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.69 GiB (GPU 0; 8.00 GiB total capacity; 6.14 GiB already allocated; 0 bytes free; 6.22 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Server logs
[2023-03-22 11:57:12,660] [7720] [MainThread] [INFO] (monailabel.endpoints.datastore:68) - Image: 23.01.03.18; File: <starlette.datastructures.UploadFile object at 0x000001C073531D90>; params: {"client_id": "user-xyz"}
[2023-03-22 11:57:12,746] [7720] [MainThread] [INFO] (monailabel.datastore.local:439) - Adding Image: 23.01.03.18 => C:\Users\keyur\AppData\Local\Temp\tmp1n82rfbc.nii.gz
[2023-03-22 11:57:13,325] [7720] [MainThread] [INFO] (monailabel.endpoints.datastore:101) - Saving Label for 23.01.03.18 for tag: final by admin
[2023-03-22 11:57:13,331] [7720] [MainThread] [INFO] (monailabel.endpoints.datastore:112) - Save Label params: {"label_info": [{"name": "liver", "idx": 1}, {"name": "venaporta", "idx": 2}, {"name": "livervein", "idx": 3}, {"name": "venacava", "idx": 4}, {"name": "lesions", "idx": 5}], "client_id": "user-xyz"}
[2023-03-22 11:57:13,332] [7720] [MainThread] [INFO] (monailabel.datastore.local:486) - Saving Label for Image: 23.01.03.18; Tag: final; Info: {'label_info': [{'name': 'liver', 'idx': 1}, {'name': 'venaporta', 'idx': 2}, {'name': 'livervein', 'idx': 3}, {'name': 'venacava', 'idx': 4}, {'name': 'lesions', 'idx': 5}], 'client_id': 'user-xyz'}
[2023-03-22 11:57:13,333] [7720] [MainThread] [INFO] (monailabel.datastore.local:494) - Adding Label: 23.01.03.18 => final => C:\Users\keyur\AppData\Local\Temp\tmpm17x2cn6.nii.gz
[2023-03-22 11:57:13,338] [7720] [MainThread] [INFO] (monailabel.datastore.local:510) - Label Info: {'label_info': [{'name': 'liver', 'idx': 1}, {'name': 'venaporta', 'idx': 2}, {'name': 'livervein', 'idx': 3}, {'name': 'venacava', 'idx': 4}, {'name': 'lesions', 'idx': 5}], 'client_id': 'user-xyz', 'ts': 1679482633, 'name': '23.01.03.18.nii.gz'}
[2023-03-22 11:57:13,344] [7720] [MainThread] [INFO] (monailabel.interfaces.app:492) - New label saved for: 23.01.03.18 => 23.01.03.18
[2023-03-22 11:57:16,062] [7720] [MainThread] [INFO] (monailabel.utils.async_tasks.task:41) - Train request: {'model': 'segmentation', 'name': 'train_01', 'pretrained': True, 'device': 'cuda', 'max_epochs': 50, 'early_stop_patience': -1, 'val_split': 0.2, 'train_batch_size': 1, 'val_batch_size': 1, 'multi_gpu': True, 'gpus': 'all', 'dataset': 'SmartCacheDataset', 'dataloader': 'ThreadDataLoader', 'tracking': 'mlflow', 'tracking_uri': '', 'tracking_experiment_name': '', 'client_id': 'user-xyz'}
[2023-03-22 11:57:16,063] [7720] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:49) - Before:: C:\Users\keyur\MONAILabel;
[2023-03-22 11:57:16,064] [7720] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:53) - After:: C:\Users\keyur\MONAILabel;
[2023-03-22 11:57:16,065] [7720] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:65) - COMMAND:: C:\Users\keyur.conda\envs\monai\python.exe -m monailabel.interfaces.utils.app -m train -r {"model":"segmentation","name":"train_01","pretrained":true,"device":"cuda","max_epochs":50,"early_stop_patience":-1,"val_split":0.2,"train_batch_size":1,"val_batch_size":1,"multi_gpu":true,"gpus":"all","dataset":"SmartCacheDataset","dataloader":"ThreadDataLoader","tracking":"mlflow","tracking_uri":"","tracking_experiment_name":"","client_id":"user-xyz"}
[2023-03-22 11:57:17,250] [32928] [MainThread] [INFO] (main:37) - Initializing App from: C:\Users\keyur\MONAILabel\monailabel\scripts\apps\radiology; studies: C:\Users\keyur\MONAILabel\monailabel\scripts\datasets\training; conf: {'models': 'segmentation'}
[2023-03-22 11:57:22,938] [32928] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for MONAILabelApp Found: <class 'main.MyApp'>
[2023-03-22 11:57:22,947] [32928] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepedit.DeepEdit'>
[2023-03-22 11:57:22,948] [32928] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_2d.Deepgrow2D'>
[2023-03-22 11:57:22,948] [32928] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_3d.Deepgrow3D'>
[2023-03-22 11:57:22,949] [32928] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.localization_spine.LocalizationSpine'>
[2023-03-22 11:57:22,949] [32928] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.localization_vertebra.LocalizationVertebra'>
[2023-03-22 11:57:22,950] [32928] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation.Segmentation'>
[2023-03-22 11:57:22,950] [32928] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_spleen.SegmentationSpleen'>
[2023-03-22 11:57:22,951] [32928] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_vertebra.SegmentationVertebra'>
[2023-03-22 11:57:22,951] [32928] [MainThread] [INFO] (main:93) - +++ Adding Model: segmentation => lib.configs.segmentation.Segmentation
[2023-03-22 11:57:22,974] [32928] [MainThread] [INFO] (main:96) - +++ Using Models: ['segmentation']
[2023-03-22 11:57:22,974] [32928] [MainThread] [INFO] (monailabel.interfaces.app:134) - Init Datastore for: C:\Users\keyur\MONAILabel\monailabel\scripts\datasets\training
[2023-03-22 11:57:22,975] [32928] [MainThread] [INFO] (monailabel.datastore.local:130) - Auto Reload: False; Extensions: ['.nii.gz', '.nii', '.nrrd', '.jpg', '.png', '.tif', '.svs', '.xml']
[2023-03-22 11:57:22,986] [32928] [MainThread] [INFO] (monailabel.datastore.local:577) - Invalidate count: 0
[2023-03-22 11:57:22,987] [32928] [MainThread] [INFO] (main:126) - +++ Adding Inferer:: segmentation => <lib.infers.segmentation.Segmentation object at 0x00000141821A56D0>
[2023-03-22 11:57:22,987] [32928] [MainThread] [INFO] (main:191) - {'segmentation': <lib.infers.segmentation.Segmentation object at 0x00000141821A56D0>, 'Histogram+GraphCut': <monailabel.scribbles.infer.HistogramBasedGraphCut object at 0x000001418AA7F370>, 'GMM+GraphCut': <monailabel.scribbles.infer.GMMBasedGraphCut object at 0x000001418AA7F340>}
[2023-03-22 11:57:22,987] [32928] [MainThread] [INFO] (main:206) - +++ Adding Trainer:: segmentation => <lib.trainers.segmentation.Segmentation object at 0x000001418AA7F3A0>
[2023-03-22 11:57:22,987] [32928] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: C:\Users\keyur.cache\monailabel\sessions
[2023-03-22 11:57:22,987] [32928] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2023-03-22 11:57:22,987] [32928] [MainThread] [INFO] (monailabel.tasks.train.basic_train:432) - Train Request (input): {'model': 'segmentation', 'name': 'train_01', 'pretrained': True, 'device': 'cuda', 'max_epochs': 50, 'early_stop_patience': -1, 'val_split': 0.2, 'train_batch_size': 1, 'val_batch_size': 1, 'multi_gpu': True, 'gpus': 'all', 'dataset': 'SmartCacheDataset', 'dataloader': 'ThreadDataLoader', 'tracking': 'mlflow', 'tracking_uri': '', 'tracking_experiment_name': '', 'client_id': 'user-xyz', 'local_rank': 0}
[2023-03-22 11:57:22,987] [32928] [MainThread] [INFO] (monailabel.tasks.train.basic_train:442) - CUDA_VISIBLE_DEVICES: None
[2023-03-22 11:57:22,989] [32928] [MainThread] [INFO] (monailabel.tasks.train.basic_train:447) - Distributed/Multi GPU is limited
[2023-03-22 11:57:22,989] [32928] [MainThread] [INFO] (monailabel.tasks.train.basic_train:462) - Distributed Training = FALSE
[2023-03-22 11:57:22,989] [32928] [MainThread] [INFO] (monailabel.tasks.train.basic_train:489) - 0 - Train Request (final): {'name': 'train_01', 'pretrained': True, 'device': 'cuda', 'max_epochs': 50, 'early_stop_patience': -1, 'val_split': 0.2, 'train_batch_size': 1, 'val_batch_size': 1, 'multi_gpu': False, 'gpus': 'all', 'dataset': 'SmartCacheDataset', 'dataloader': 'ThreadDataLoader', 'tracking': 'mlflow', 'tracking_uri': '', 'tracking_experiment_name': '', 'model': 'segmentation', 'client_id': 'user-xyz', 'local_rank': 0, 'run_id': '20230322_115722'}
[2023-03-22 11:57:22,989] [32928] [MainThread] [INFO] (monailabel.tasks.train.basic_train:622) - 0 - Using Device: cuda; IDX: None
[2023-03-22 11:57:22,989] [32928] [MainThread] [INFO] (monailabel.tasks.train.basic_train:515) - Run/Output Path: C:\Users\keyur\MONAILabel\monailabel\scripts\apps\radiology\model\segmentation\train_01
[2023-03-22 11:57:22,989] [32928] [MainThread] [INFO] (monailabel.tasks.train.basic_train:531) - Tracking: mlflow
[2023-03-22 11:57:22,989] [32928] [MainThread] [INFO] (monailabel.tasks.train.basic_train:532) - Tracking URI: file:///C:/Users/keyur/MONAILabel/monailabel/scripts/apps/radiology/model/segmentation/train_01/mlruns;
[2023-03-22 11:57:22,989] [32928] [MainThread] [INFO] (monailabel.tasks.train.basic_train:533) - Tracking Experiment Name: segmentation; Run Name: run_20230322_115722
[2023-03-22 11:57:22,989] [32928] [MainThread] [INFO] (monailabel.tasks.train.basic_train:410) - Total Records for Training: 6
[2023-03-22 11:57:22,989] [32928] [MainThread] [INFO] (monailabel.tasks.train.basic_train:411) - Total Records for Validation: 2
Loading dataset: 0%| | 0/2 [00:00<?, ?it/s]
Loading dataset: 50%|##### | 1/2 [00:11<00:11, 11.37s/it]
Loading dataset: 100%|##########| 2/2 [00:21<00:00, 10.42s/it]
Loading dataset: 100%|##########| 2/2 [00:21<00:00, 10.57s/it]
cache_num is greater or equal than dataset length, fall back to regular monai.data.CacheDataset.
[2023-03-22 11:57:44,226] [32928] [MainThread] [INFO] (monailabel.tasks.train.basic_train:328) - 0 - Records for Validation: 2
[2023-03-22 11:57:44,237] [32928] [MainThread] [INFO] (monailabel.tasks.train.basic_train:318) - 0 - Adding Validation to run every '1' interval
[2023-03-22 11:57:44,240] [32928] [MainThread] [INFO] (monailabel.tasks.train.basic_train:710) - 0 - Load Path C:\Users\keyur\MONAILabel\monailabel\scripts\apps\radiology\model\segmentation\train_01\model.pt
Loading dataset: 0%| | 0/6 [00:00<?, ?it/s]
Loading dataset: 17%|#6 | 1/6 [00:10<00:52, 10.54s/it]
Loading dataset: 33%|###3 | 2/6 [00:15<00:28, 7.00s/it]
Loading dataset: 50%|##### | 3/6 [00:26<00:27, 9.16s/it]
Loading dataset: 67%|######6 | 4/6 [00:35<00:17, 8.80s/it]
Loading dataset: 83%|########3 | 5/6 [00:47<00:10, 10.14s/it]
Loading dataset: 100%|##########| 6/6 [01:02<00:00, 11.67s/it]
Loading dataset: 100%|##########| 6/6 [01:02<00:00, 10.37s/it]
[2023-03-22 11:58:46,454] [32928] [MainThread] [INFO] (monailabel.tasks.train.basic_train:264) - 0 - Records for Training: 6
[2023-03-22 11:58:46,458] [32928] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:876) - Engine run resuming from iteration 0, epoch 0 until 50 epochs
[2023-03-22 11:58:46,617] [32928] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:138) - Restored all variables from C:\Users\keyur\MONAILabel\monailabel\scripts\apps\radiology\model\segmentation\train_01\model.pt
[2023-03-22 11:58:51,634] [32928] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:272) - Epoch: 1/50, Iter: 1/6 -- train_loss: 0.9931
[2023-03-22 11:58:52,005] [32928] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:272) - Epoch: 1/50, Iter: 2/6 -- train_loss: 0.9202
[2023-03-22 11:58:52,382] [32928] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:272) - Epoch: 1/50, Iter: 3/6 -- train_loss: 0.8346
[2023-03-22 11:58:52,736] [32928] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:272) - Epoch: 1/50, Iter: 4/6 -- train_loss: 0.8939
[2023-03-22 11:58:53,133] [32928] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:272) - Epoch: 1/50, Iter: 5/6 -- train_loss: 0.9609
[2023-03-22 11:58:53,435] [32928] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:272) - Epoch: 1/50, Iter: 6/6 -- train_loss: 0.8174
[2023-03-22 11:58:53,442] [32928] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:257) - Got new best metric of train_mean_dice: 0.2552022635936737
[2023-03-22 11:58:53,442] [32928] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:201) - Epoch[1] Metrics -- train_lesions_mean_dice: 0.0045 train_liver_mean_dice: 0.6504 train_livervein_mean_dice: 0.3602 train_mean_dice: 0.2552 train_venacava_mean_dice: 0.0001 train_venaporta_mean_dice: 0.2535
[2023-03-22 11:58:53,442] [32928] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:212) - Key metric: train_mean_dice best value: 0.2552022635936737 at epoch: 1
[2023-03-22 11:58:53,448] [32928] [MainThread] [INFO] (ignite.engine.engine.SupervisedEvaluator:876) - Engine run resuming from iteration 0, epoch 0 until 1 epochs

[2023-03-22 11:58:57,944] [32928] [MainThread] [ERROR] (ignite.engine.engine.SupervisedEvaluator:1086) - Current run is terminating due to exception: applying transform <monai.transforms.compose.Compose object at 0x0000014188C0B160>

[2023-03-22 11:58:57,945] [32928] [MainThread] [ERROR] (ignite.engine.engine.SupervisedEvaluator:180) - Exception: applying transform <monai.transforms.compose.Compose object at 0x0000014188C0B160>
Traceback (most recent call last):
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\transform.py", line 102, in apply_transform
return _apply_transform(transform, data, unpack_items)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\transform.py", line 66, in _apply_transform
return transform(parameters)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\post\dictionary.py", line 202, in call
d[key] = self.converter(d[key], argmax, to_onehot, threshold, rounding)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\post\array.py", line 220, in call
img_t = one_hot(
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\networks\utils.py", line 158, in one_hot
o = torch.zeros(size=sh, dtype=dtype, device=labels.device)

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.69 GiB (GPU 0; 8.00 GiB total capacity; 6.14 GiB already allocated; 0 bytes free; 6.22 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\transform.py", line 102, in apply_transform
return _apply_transform(transform, data, unpack_items)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\transform.py", line 66, in apply_transform
return transform(parameters)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\compose.py", line 174, in call
input = apply_transform(transform, input, self.map_items, self.unpack_items, self.log_stats)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\transform.py", line 129, in apply_transform
raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.post.dictionary.AsDiscreted object at 0x0000014188BFE5B0>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 1068, in _run_once_on_dataset_as_gen
self.state.output = self._process_function(self, self.state.batch)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\engines\evaluator.py", line 308, in _iteration
engine.fire_event(IterationEvents.MODEL_COMPLETED)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 449, in fire_event
return self._fire_event(event_name)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 425, in _fire_event
func(*first, *(event_args + others), **kwargs)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\engines\workflow.py", line 224, in _run_postprocessing
engine.state.batch[i], engine.state.output[i] = engine_apply_transform(b, o, posttrans)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\engines\utils.py", line 258, in engine_apply_transform
transformed_data = apply_transform(transform, data)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\transform.py", line 129, in apply_transform
raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.compose.Compose object at 0x0000014188C0B160>
[2023-03-22 11:58:58,024] [32928] [MainThread] [ERROR] (ignite.engine.engine.SupervisedEvaluator:992) - Engine run is terminating due to exception: applying transform <monai.transforms.compose.Compose object at 0x0000014188C0B160>
[2023-03-22 11:58:58,024] [32928] [MainThread] [ERROR] (ignite.engine.engine.SupervisedEvaluator:180) - Exception: applying transform <monai.transforms.compose.Compose object at 0x0000014188C0B160>
Traceback (most recent call last):
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\transform.py", line 102, in apply_transform
return _apply_transform(transform, data, unpack_items)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\transform.py", line 66, in _apply_transform
return transform(parameters)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\post\dictionary.py", line 202, in call
d[key] = self.converter(d[key], argmax, to_onehot, threshold, rounding)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\post\array.py", line 220, in call
img_t = one_hot(
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\networks\utils.py", line 158, in one_hot
o = torch.zeros(size=sh, dtype=dtype, device=labels.device)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.69 GiB (GPU 0; 8.00 GiB total capacity; 6.14 GiB already allocated; 0 bytes free; 6.22 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\transform.py", line 102, in apply_transform
return _apply_transform(transform, data, unpack_items)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\transform.py", line 66, in apply_transform
return transform(parameters)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\compose.py", line 174, in call
input = apply_transform(transform, input, self.map_items, self.unpack_items, self.log_stats)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\transform.py", line 129, in apply_transform
raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.post.dictionary.AsDiscreted object at 0x0000014188BFE5B0>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 959, in _internal_run_as_gen
epoch_time_taken += yield from self._run_once_on_dataset_as_gen()
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 1087, in _run_once_on_dataset_as_gen
self._handle_exception(e)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 636, in _handle_exception
self._fire_event(Events.EXCEPTION_RAISED, e)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 425, in _fire_event
func(*first, *(event_args + others), **kwargs)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\handlers\stats_handler.py", line 181, in exception_raised
raise e
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 1068, in _run_once_on_dataset_as_gen
self.state.output = self._process_function(self, self.state.batch)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\engines\evaluator.py", line 308, in _iteration
engine.fire_event(IterationEvents.MODEL_COMPLETED)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 449, in fire_event
return self._fire_event(event_name)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 425, in _fire_event
func(*first, *(event_args + others), **kwargs)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\engines\workflow.py", line 224, in _run_postprocessing
engine.state.batch[i], engine.state.output[i] = engine_apply_transform(b, o, posttrans)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\engines\utils.py", line 258, in engine_apply_transform
transformed_data = apply_transform(transform, data)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\transform.py", line 129, in apply_transform
raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.compose.Compose object at 0x0000014188C0B160>
[2023-03-22 11:58:58,027] [32928] [MainThread] [ERROR] (ignite.engine.engine.SupervisedTrainer:992) - Engine run is terminating due to exception: applying transform <monai.transforms.compose.Compose object at 0x0000014188C0B160>
[2023-03-22 11:58:58,027] [32928] [MainThread] [ERROR] (ignite.engine.engine.SupervisedTrainer:180) - Exception: applying transform <monai.transforms.compose.Compose object at 0x0000014188C0B160>
Traceback (most recent call last):
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\transform.py", line 102, in apply_transform
return _apply_transform(transform, data, unpack_items)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\transform.py", line 66, in _apply_transform
return transform(parameters)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\post\dictionary.py", line 202, in call
d[key] = self.converter(d[key], argmax, to_onehot, threshold, rounding)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\post\array.py", line 220, in call
img_t = one_hot(
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\networks\utils.py", line 158, in one_hot
o = torch.zeros(size=sh, dtype=dtype, device=labels.device)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.69 GiB (GPU 0; 8.00 GiB total capacity; 6.14 GiB already allocated; 0 bytes free; 6.22 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\transform.py", line 102, in apply_transform
return _apply_transform(transform, data, unpack_items)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\transform.py", line 66, in apply_transform
return transform(parameters)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\compose.py", line 174, in call
input = apply_transform(transform, input, self.map_items, self.unpack_items, self.log_stats)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\transform.py", line 129, in apply_transform
raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.post.dictionary.AsDiscreted object at 0x0000014188BFE5B0>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 965, in _internal_run_as_gen
self._fire_event(Events.EPOCH_COMPLETED)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 425, in _fire_event
func(*first, *(event_args + others), **kwargs)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\handlers\validation_handler.py", line 76, in call
self.validator.run(engine.state.epoch)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\engines\evaluator.py", line 148, in run
super().run()
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\engines\workflow.py", line 281, in run
super().run(data=self.data_loader, max_epochs=self.state.max_epochs)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 892, in run
return self._internal_run()
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 935, in _internal_run
return next(self._internal_run_generator)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 993, in _internal_run_as_gen
self._handle_exception(e)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 636, in _handle_exception
self._fire_event(Events.EXCEPTION_RAISED, e)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 425, in _fire_event
func(*first, *(event_args + others), **kwargs)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\handlers\stats_handler.py", line 181, in exception_raised
raise e
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 959, in _internal_run_as_gen
epoch_time_taken += yield from self._run_once_on_dataset_as_gen()
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 1087, in _run_once_on_dataset_as_gen
self._handle_exception(e)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 636, in _handle_exception
self._fire_event(Events.EXCEPTION_RAISED, e)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 425, in _fire_event
func(*first, *(event_args + others), **kwargs)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\handlers\stats_handler.py", line 181, in exception_raised
raise e
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 1068, in _run_once_on_dataset_as_gen
self.state.output = self._process_function(self, self.state.batch)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\engines\evaluator.py", line 308, in _iteration
engine.fire_event(IterationEvents.MODEL_COMPLETED)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 449, in fire_event
return self._fire_event(event_name)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 425, in _fire_event
func(*first, *(event_args + others), **kwargs)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\engines\workflow.py", line 224, in _run_postprocessing
engine.state.batch[i], engine.state.output[i] = engine_apply_transform(b, o, posttrans)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\engines\utils.py", line 258, in engine_apply_transform
transformed_data = apply_transform(transform, data)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\transform.py", line 129, in apply_transform
raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.compose.Compose object at 0x0000014188C0B160>
Traceback (most recent call last):
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\transform.py", line 102, in apply_transform
return _apply_transform(transform, data, unpack_items)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\transform.py", line 66, in _apply_transform
return transform(parameters)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\post\dictionary.py", line 202, in call
d[key] = self.converter(d[key], argmax, to_onehot, threshold, rounding)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\post\array.py", line 220, in call
img_t = one_hot(
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\networks\utils.py", line 158, in one_hot
o = torch.zeros(size=sh, dtype=dtype, device=labels.device)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.69 GiB (GPU 0; 8.00 GiB total capacity; 6.14 GiB already allocated; 0 bytes free; 6.22 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\transform.py", line 102, in apply_transform
return _apply_transform(transform, data, unpack_items)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\transform.py", line 66, in apply_transform
return transform(parameters)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\compose.py", line 174, in call
input = apply_transform(transform, input, self.map_items, self.unpack_items, self.log_stats)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\transform.py", line 129, in apply_transform
raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.post.dictionary.AsDiscreted object at 0x0000014188BFE5B0>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\keyur.conda\envs\monai\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\keyur.conda\envs\monai\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Users\keyur\MONAILabel\monailabel\interfaces\utils\app.py", line 128, in
run_main()
File "C:\Users\keyur\MONAILabel\monailabel\interfaces\utils\app.py", line 113, in run_main
result = a.train(request)
File "C:\Users\keyur\MONAILabel\monailabel\interfaces\app.py", line 422, in train
result = task(request, self.datastore())
File "C:\Users\keyur\MONAILabel\monailabel\tasks\train\basic_train.py", line 463, in call
res = self.train(0, world_size, req, datalist)
File "C:\Users\keyur\MONAILabel\monailabel\tasks\train\basic_train.py", line 552, in train
context.trainer.run()
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\engines\trainer.py", line 53, in run
super().run()
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\engines\workflow.py", line 281, in run
super().run(data=self.data_loader, max_epochs=self.state.max_epochs)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 892, in run
return self._internal_run()
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 935, in _internal_run
return next(self._internal_run_generator)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 993, in _internal_run_as_gen
self._handle_exception(e)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 636, in _handle_exception
self._fire_event(Events.EXCEPTION_RAISED, e)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 425, in _fire_event
func(*first, *(event_args + others), **kwargs)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\handlers\stats_handler.py", line 181, in exception_raised
raise e
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 965, in _internal_run_as_gen
self._fire_event(Events.EPOCH_COMPLETED)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 425, in _fire_event
func(*first, *(event_args + others), **kwargs)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\handlers\validation_handler.py", line 76, in call
self.validator.run(engine.state.epoch)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\engines\evaluator.py", line 148, in run
super().run()
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\engines\workflow.py", line 281, in run
super().run(data=self.data_loader, max_epochs=self.state.max_epochs)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 892, in run
return self._internal_run()
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 935, in _internal_run
return next(self._internal_run_generator)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 993, in _internal_run_as_gen
self._handle_exception(e)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 636, in _handle_exception
self._fire_event(Events.EXCEPTION_RAISED, e)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 425, in _fire_event
func(*first, *(event_args + others), **kwargs)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\handlers\stats_handler.py", line 181, in exception_raised
raise e
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 959, in _internal_run_as_gen
epoch_time_taken += yield from self._run_once_on_dataset_as_gen()
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 1087, in _run_once_on_dataset_as_gen
self._handle_exception(e)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 636, in _handle_exception
self._fire_event(Events.EXCEPTION_RAISED, e)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 425, in _fire_event
func(*first, *(event_args + others), **kwargs)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\handlers\stats_handler.py", line 181, in exception_raised
raise e
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 1068, in _run_once_on_dataset_as_gen
self.state.output = self._process_function(self, self.state.batch)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\engines\evaluator.py", line 308, in _iteration
engine.fire_event(IterationEvents.MODEL_COMPLETED)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 449, in fire_event
return self._fire_event(event_name)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\ignite\engine\engine.py", line 425, in _fire_event
func(*first, *(event_args + others), **kwargs)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\engines\workflow.py", line 224, in _run_postprocessing
engine.state.batch[i], engine.state.output[i] = engine_apply_transform(b, o, posttrans)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\engines\utils.py", line 258, in engine_apply_transform
transformed_data = apply_transform(transform, data)
File "C:\Users\keyur.conda\envs\monai\lib\site-packages\monai\transforms\transform.py", line 129, in apply_transform
raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.compose.Compose object at 0x0000014188C0B160>
[2023-03-22 11:58:59,641] [7720] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:83) - Return code: 1

To Reproduce
Steps to reproduce the behavior:

  1. Activate Monailabel server
  2. Start training from 3D Slicer
  3. Run commands '....' (how you have started monailabel server) : monailabel start_server --app apps/radiology --studies datasets/training --conf models segmentation

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Environment

Ensuring you use the relevant python executable, please paste the output of:

python -c 'import monai; monai.config.print_debug_info()'

================================
Printing MONAI config...
================================
MONAI version: 1.1.0
Numpy version: 1.23.5
Pytorch version: 2.0.0+cu118
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: a2ec3752f54bfc3b40e7952234fbeb5452ed63e3
MONAI __file__: C:\Users\keyur\.conda\envs\monai\lib\site-packages\monai\__init__.py

Optional dependencies:
Pytorch Ignite version: 0.4.10
Nibabel version: 5.0.1
scikit-image version: 0.20.0
Pillow version: 9.3.0
Tensorboard version: 2.12.0
gdown version: 4.6.4
TorchVision version: 0.15.1+cpu
tqdm version: 4.65.0
lmdb version: 1.4.0
psutil version: 5.9.0
pandas version: 1.5.3
einops version: 0.6.0
transformers version: NOT INSTALLED or UNKNOWN VERSION.
mlflow version: 2.2.2
pynrrd version: 0.4.3

For details about installing the optional dependencies, please visit:
    https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies


================================
Printing system config...
================================
System: Windows
Win32 version: ('10', '10.0.22621', 'SP0', 'Multiprocessor Free')
Win32 edition: Core
Platform: Windows-10-10.0.22621-SP0
Processor: Intel64 Family 6 Model 186 Stepping 2, GenuineIntel
Machine: AMD64
Python version: 3.9.16
Process name: python.exe
Command: ['C:/Users/keyur/.conda/envs/monai\\python.exe', '-m', 'ipykernel_launcher', '-f', 'C:\\Users\\keyur\\AppData\\Roaming\\jupyter\\runtime\\kernel-16ea1610-8b90-4286-bcdb-5d8bc9d19305.json']
Open files: [popenfile(path='C:\\Users\\keyur\\.ipython\\profile_default\\history.sqlite', fd=-1), popenfile(path='C:\\Program Files\\WindowsApps\\Microsoft.LanguageExperiencePacken-GB_22621.13.87.0_neutral__8wekyb3d8bbwe\\Windows\\System32\\en-GB\\2d99171d54bafb1068cad8303bddb437\\tzres.dll.mui', fd=-1), popenfile(path='C:\\Windows\\System32\\en-US\\kernel32.dll.mui', fd=-1), popenfile(path='C:\\Windows\\System32\\DriverStore\\FileRepository\\nvsmui.inf_amd64_1e558733305022a1\\nvcubins.bin', fd=-1), popenfile(path='C:\\Windows\\System32\\en-US\\KernelBase.dll.mui', fd=-1)]
Num physical CPUs: 14
Num logical CPUs: 20
Num usable CPUs: 20
CPU usage (%): [0.7, 0.4, 1.0, 0.1, 5.1, 1.1, 3.7, 1.2, 1.8, 0.4, 0.9, 0.4, 11.4, 14.7, 9.1, 10.1, 10.1, 10.6, 11.7, 12.5]
CPU freq. (MHz): 2600
Load avg. in last 1, 5, 15 mins (%): [13.8, 13.8, 10.0]
Disk usage (%): 44.1
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 31.6
Available memory (GB): 18.1
Used memory (GB): 13.5

================================
Printing GPU config...
================================
Num GPUs: 1
Has CUDA: True
CUDA version: 11.8
cuDNN enabled: True
cuDNN version: 8700
Current device: 0
Library compiled for CUDA architectures: ['sm_37', 'sm_50', 'sm_60', 'sm_61', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'sm_90', 'compute_37']
GPU 0 Name: NVIDIA GeForce RTX 4070 Laptop GPU
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 36
GPU 0 Total memory (GB): 8.0
GPU 0 CUDA capability (maj.min): 8.9

**Additional context**
Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the training entry point in monailabel.tasks.train.basic_train and the radiology app paths shown in the logs. Reproduce the segmentation training request with the reported CUDA and PyTorch settings, then trace the Compose transform failure and out-of-memory report; done requires an identified, reproducible cause and a confirmed resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.