Auto3DSeg in Azure fails - new to Auto3DSeg

Dear all, i am starting to use Auto3DSeg to develop a segmentation model for vertebrae from CT images. I am using a A100 GPU in an Azure environment.

I installed pytorch via conda and installed MONAI via the git repository git clone https://github.com/Project-MONAI/MONAI.git. I created the yaml file and the files have been properly assigned. Yet i get the following (long) error when i try to run the pipeline.

(monai) azureuser@rs-a100b:~/cloudfiles/private-info$ python -m monai.apps.auto3dseg AutoRunner run --input="/home/azureuser/cloudfiles/private-info/AutoSeg3D/Spine1/Spine1.yaml"
 missing cuda symbols while dynamic loading
 cuFile initialization failed
2023-10-26 06:50:05,063 - INFO - AutoRunner using work directory ./work_dir
2023-10-26 06:50:05,117 - INFO - Loading input config /home/azureuser/cloudfiles/private-info/AutoSeg3D/Spine1/Spine1.yaml
2023-10-26 06:50:05,361 - INFO - Datalist was copied to work_dir: /mnt/batch/tasks/shared/LS_root/mounts/clusters/rs-a100b/private-info/work_dir/spine1_folds.json
2023-10-26 06:50:05,379 - INFO - Setting num_fold 1 based on the input datalist /mnt/batch/tasks/shared/LS_root/mounts/clusters/rs-a100b/private-info/work_dir/spine1_folds.json.
2023-10-26 06:50:05,793 - INFO - Using user defined command running prefix , will override other settings
2023-10-26 06:50:05,803 - INFO - Running data analysis...
2023-10-26 06:50:05,806 - INFO - Found 1 GPUs for data analyzing!
  0%|                                                                                                                                                                                       | 0/307 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/anaconda/envs/monai/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/anaconda/envs/monai/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/rs-a100b/private-info/MONAI-git/monai/apps/auto3dseg/__main__.py", line 24, in <module>
    fire.Fire(
  File "/anaconda/envs/monai/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/anaconda/envs/monai/lib/python3.9/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/anaconda/envs/monai/lib/python3.9/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/rs-a100b/private-info/MONAI-git/monai/apps/auto3dseg/auto_runner.py", line 743, in run
    da.get_all_case_stats()
  File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/rs-a100b/private-info/MONAI-git/monai/apps/auto3dseg/data_analyzer.py", line 230, in get_all_case_stats
    result_bycase = self._get_all_case_stats(0, 1, None, key, transform_list)
  File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/rs-a100b/private-info/MONAI-git/monai/apps/auto3dseg/data_analyzer.py", line 333, in _get_all_case_stats
    for batch_data in tqdm(dataloader) if (has_tqdm and rank == 0) else dataloader:
  File "/anaconda/envs/monai/lib/python3.9/site-packages/tqdm/std.py", line 1182, in __iter__
    for obj in iterable:
  File "/anaconda/envs/monai/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 630, in __next__
    data = self._next_data()
  File "/anaconda/envs/monai/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1345, in _next_data
    return self._process_data(data)
  File "/anaconda/envs/monai/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1371, in _process_data
    data.reraise()
  File "/anaconda/envs/monai/lib/python3.9/site-packages/torch/_utils.py", line 694, in reraise
    raise exception
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/rs-a100b/private-info/MONAI-git/monai/transforms/transform.py", line 141, in apply_transform
    return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats)
  File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/rs-a100b/private-info/MONAI-git/monai/transforms/transform.py", line 98, in _apply_transform
    return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data)
  File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/rs-a100b/private-info/MONAI-git/monai/transforms/io/dictionary.py", line 161, in __call__
    for key, meta_key, meta_key_postfix in self.key_iterator(d, self.meta_keys, self.meta_key_postfix):
  File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/rs-a100b/private-info/MONAI-git/monai/transforms/transform.py", line 475, in key_iterator
    raise KeyError(
KeyError: 'Key `image` of transform `LoadImaged` was missing in the data and allow_missing_keys==False.'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/rs-a100b/private-info/MONAI-git/monai/transforms/transform.py", line 141, in apply_transform
    return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats)
  File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/rs-a100b/private-info/MONAI-git/monai/transforms/transform.py", line 98, in _apply_transform
    return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data)
  File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/rs-a100b/private-info/MONAI-git/monai/transforms/compose.py", line 335, in __call__
    result = execute_compose(
  File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/rs-a100b/private-info/MONAI-git/monai/transforms/compose.py", line 111, in execute_compose
    data = apply_transform(
  File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/rs-a100b/private-info/MONAI-git/monai/transforms/transform.py", line 171, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.io.dictionary.LoadImaged object at 0x7f36a37dd6d0>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/anaconda/envs/monai/lib/python3.9/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
    data = fetcher.fetch(index)
  File "/anaconda/envs/monai/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/anaconda/envs/monai/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 51, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/rs-a100b/private-info/MONAI-git/monai/data/dataset.py", line 112, in __getitem__
    return self._transform(index)
  File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/rs-a100b/private-info/MONAI-git/monai/data/dataset.py", line 98, in _transform
    return apply_transform(self.transform, data_i) if self.transform is not None else data_i
  File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/rs-a100b/private-info/MONAI-git/monai/transforms/transform.py", line 171, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.compose.Compose object at 0x7f36a37dda30>

Your help in troubleshooting this is very much appreciated, especially since i am new to MONAI/Auto3DSeg.

Thanks for all.

Best
Rui

Start with the referenced Spine1.yaml and datalist, then trace the AutoRunner.run and data_analyzer.py path to transforms/io/dictionary.py, where LoadImaged reports the missing image key. Compare the datalist entries with the keys expected by that transform and rerun the Auto3DSeg command; done means data analysis proceeds without the DataLoader KeyError.