No files generated by monai auto 3d seg during infrence

Describe the bug
I had run the auto 3d seg on private dataset and i have no files in results folder of the test set.

To Reproduce
run code

runner = AutoRunner(
    work_dir=work_dir,
    input={
        "modality": "MRI",
        "datalist": datalist_path,
        "dataroot": work_dir    },
)
runner.run()

in datalist path the path to the file below is passed

{"training": [{"fold": 0, "image": "/workspaces/prostate_opi_year_3/data/dataset/post_norm_min_max_bias_on/1188/1188_adc.nii.gz", "label": 
...
 "/workspaces/prostate_opi_year_3/data/dataset/post_norm_min_max_bias_on/1218/1218_lesion_union_adc.nii.gz"}], "testing": [{"image": "/workspaces/prostate_opi_year_3/data/dataset/post_norm_min_max_bias_on/3322/3322_adc.nii.gz", "label": "/workspaces/prostate_opi_year_3/data/dataset/post_norm_min_max_bias_on/3322/3322_lesion_union_adc.nii.gz"}, {"image": "/workspaces/prostate_opi_year_3/data/dataset/post_norm_min_max_bias_on/066/66_adc.nii.gz", "label": 
...
, "label": "/workspaces/prostate_opi_year_3/data/dataset/post_norm_min_max_bias_on/523/523_lesion_union_adc.nii.gz"}], "modality": "MRI"}

Then I get the information that auto3d seg completed succesfully and files from test set will be in ensemble_output - but no files are present there (although folder gets created)

2024/08/22 19:40:47 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
dints_3 - training ...: 100%|███████████████████████████████████| 40/40 [9:24:19<00:00, 846.48s/round]
dints_3 - validation at original spacing/resolution
2024-08-23 05:16:03,679 - WARNING - dints_3 - training: finished
2024-08-23 05:16:06,920 - INFO - ['python', '/workspaces/prostate_opi_year_3/data/work_dir_b/dints_4/scripts/train.py', 'run', "--config_file='/workspaces/prostate_opi_year_3/data/work_dir_b/dints_4/configs/hyper_parameters.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/dints_4/configs/hyper_parameters_search.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/dints_4/configs/network.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/dints_4/configs/network_search.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/dints_4/configs/transforms_infer.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/dints_4/configs/transforms_train.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/dints_4/configs/transforms_validate.yaml'"]
2024/08/23 05:16:16 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
dints_4 - training ...: 100%|███████████████████████████████████| 40/40 [9:25:59<00:00, 849.00s/round]
dints_4 - validation at original spacing/resolution
2024-08-23 14:53:05,172 - WARNING - dints_4 - training: finished
2024-08-23 14:53:08,393 - INFO - ['python', '/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet2d_0/scripts/train.py', 'run', "--config_file='/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet2d_0/configs/hyper_parameters.yaml'"]
2024/08/23 14:53:16 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
segresnet2d_0 - training: 100%|████████████████████████████████| 300/300 [3:29:16<00:00, 41.86s/epoch]
2024-08-23 18:22:35,724 - INFO - ['python', '/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet2d_1/scripts/train.py', 'run', "--config_file='/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet2d_1/configs/hyper_parameters.yaml'"]
2024/08/23 18:22:43 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
segresnet2d_1 - training: 100%|████████████████████████████████| 300/300 [3:30:26<00:00, 42.09s/epoch]
2024-08-23 21:53:12,197 - INFO - ['python', '/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet2d_2/scripts/train.py', 'run', "--config_file='/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet2d_2/configs/hyper_parameters.yaml'"]
2024/08/23 21:53:19 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
segresnet2d_2 - training: 100%|████████████████████████████████| 300/300 [3:30:25<00:00, 42.08s/epoch]
2024-08-24 01:23:47,636 - INFO - ['python', '/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet2d_3/scripts/train.py', 'run', "--config_file='/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet2d_3/configs/hyper_parameters.yaml'"]
2024/08/24 01:23:54 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
segresnet2d_3 - training: 100%|████████████████████████████████| 300/300 [3:31:10<00:00, 42.23s/epoch]
2024-08-24 04:55:07,927 - INFO - ['python', '/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet2d_4/scripts/train.py', 'run', "--config_file='/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet2d_4/configs/hyper_parameters.yaml'"]
2024/08/24 04:55:15 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
segresnet2d_4 - training: 100%|████████████████████████████████| 300/300 [3:31:01<00:00, 42.21s/epoch]
2024-08-24 08:26:19,707 - INFO - ['python', '/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet_0/scripts/train.py', 'run', "--config_file='/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet_0/configs/hyper_parameters.yaml'"]
`apex.normalization.InstanceNorm3dNVFuser` is not installed properly, use nn.InstanceNorm3d instead.
2024/08/24 08:26:27 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
segresnet_0 - training:  89%|███████████████████████████▋   | 268/300 [9:30:51<1   segresnet_0 - training:  90%|███████████████████████████▊   | 269/300 [9:32:53<1   segresnet_0 - training:  90%|███████████████████████████▉   | 270/300 [9:34:55<1   segresnet_0 - training:  90%|████████████████████████████   | 271/300 [9:36:58<1   segresnet_0 - training:  91%|████████████████████████████   | 272/300 [9:39:43<1   segresnet_0 - training:  91%|████████████████████████████▏  | 273/300 [9:41:45<1   segresnet_0 - training:  91%|██████████████████████████████▏  | 274/300 [9:43:47   segresnet_0 - training:  92%|██████████████████████████████▎  | 275/300 [9:45:49   segresnet_0 - training:  92%|██████████████████████████████▎  | 276/300 [9:48:34   segresnet_0 - training:  92%|██████████████████████████████▍  | 277/300 [9:50:36   segresnet_0 - training:  93%|██████████████████████████████▌  | 278/300 [9:52:38   segresnet_0 - training:  93%|██████████████████████████████▋  | 279/300 [9:54:41<44segresnet_0 - training:  93%|██████████████████████████████▊  | 280/300 [9:57:26<46segresnet_0 - training:  94%|██████████████████████████████▉  | 281/300 [9:59:29<42segresnet_0 - training:  94%|██████████████████████████████  | 282/300 [10:01:32<39segresnet_0 - training:  94%|██████████████████████████████▏ | 283/300 [10:04:18<40segresnet_0 - training:  95%|██████████████████████████████▎ | 284/300 [10:06:21<36segresnet_0 - training:  95%|██████████████████████████████▍ | 285/300 [10:08:23<32segresnet_0 - training:  95%|██████████████████████████████▌ | 286/300 [10:11:08<33segresnet_0 - training:  96%|██████████████████████████████▌ | 287/300 [10:13:11<29segresnet_0 - training:  96%|██████████████████████████████▋ | 288/300 [10:15:13<26segresnet_0 - training:  96%|██████████████████████████████▊ | 289/300 [10:17:58<26segresnet_0 - training:  97%|██████████████████████████████▉ | 290/300 [10:20:01<22segresnet_0 - training:  97%|███████████████████████████████ | 291/300 [10:22:46<21segresnet_0 - training:  97%|███████████████████████████████▏| 292/300 [10:24:49<18segresnet_0 - training:  98%|███████████████████████████████▎| 293/300 [10:27:34<17segresnet_0 - training:  98%|███████████████████████████████▎| 294/300 [10:29:37<13segresnet_0 - training:  98%|███████████████████████████████▍| 295/300 [10:32:22<12segresnet_0 - training:  99%|███████████████████████████████▌| 296/300 [10:34:25<09segresnet_0 - training:  99%|███████████████████████████████▋| 297/300 [10:37:10<07segresnet_0 - training:  99%|███████████████████████████████▊| 298/300 [10:39:56<05segresnet_0 - training: 100%|███████████████████████████████▉| 299/300 [10:42:41<02segresnet_0 - training: 100%|████████████████████████████████| 300/300 [10:45:26<00segresnet_0 - training: 100%|████████████████████████████████| 300/300 [10:45:26<00:00, 129.09s/epoch]
`apex.normalization.InstanceNorm3dNVFuser` is not installed properly, use nn.InstanceNorm3d instead.
2024-08-24 19:12:42,797 - INFO - ['python', '/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet_1/scripts/train.py', 'run', "--config_file='/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet_1/configs/hyper_parameters.yaml'"]
`apex.normalization.InstanceNorm3dNVFuser` is not installed properly, use nn.InstanceNorm3d instead.
2024/08/24 19:12:50 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
segresnet_1 - training: 100%|█████████████| 300/300 [10:48:35<00:00, 129.72s/epoch]
`apex.normalization.InstanceNorm3dNVFuser` is not installed properly, use nn.InstanceNorm3d instead.
2024-08-25 06:02:15,986 - INFO - ['python', '/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet_2/scripts/train.py', 'run', "--config_file='/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet_2/configs/hyper_parameters.yaml'"]
`apex.normalization.InstanceNorm3dNVFuser` is not installed properly, use nn.InstanceNorm3d instead.
2024/08/25 06:02:24 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
segresnet_2 - training: 100%|█████████████| 300/300 [10:51:15<00:00, 130.25s/epoch]
`apex.normalization.InstanceNorm3dNVFuser` is not installed properly, use nn.InstanceNorm3d instead.
2024-08-25 16:54:29,743 - INFO - ['python', '/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet_3/scripts/train.py', 'run', "--config_file='/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet_3/configs/hyper_parameters.yaml'"]
`apex.normalization.InstanceNorm3dNVFuser` is not installed properly, use nn.InstanceNorm3d instead.
2024/08/25 16:54:37 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
segresnet_3 - training: 100%|████████████████████████████████████████████████████████████████████████████████████| 300/300 [10:51:07<00:00, 130.23s/epoch]
`apex.normalization.InstanceNorm3dNVFuser` is not installed properly, use nn.InstanceNorm3d instead.
2024-08-26 03:46:35,712 - INFO - ['python', '/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet_4/scripts/train.py', 'run', "--config_file='/workspaces/prostate_opi_year_3/data/work_dir_b/segresnet_4/configs/hyper_parameters.yaml'"]
`apex.normalization.InstanceNorm3dNVFuser` is not installed properly, use nn.InstanceNorm3d instead.
2024/08/26 03:46:43 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
segresnet_4 - training: 100%|████████████████████████████████████████████████████████████████████████████████████| 300/300 [10:50:27<00:00, 130.09s/epoch]
`apex.normalization.InstanceNorm3dNVFuser` is not installed properly, use nn.InstanceNorm3d instead.
2024-08-26 14:38:00,553 - INFO - ['python', '/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_0/scripts/train.py', 'run', "--config_file='/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_0/configs/hyper_parameters.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_0/configs/network.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_0/configs/transforms_infer.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_0/configs/transforms_train.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_0/configs/transforms_validate.yaml'"]
monai.networks.nets.swin_unetr SwinUNETR.__init__:img_size: Argument `img_size` has been deprecated since version 1.3. It will be removed in version 1.5. The img_size argument is not required anymore and checks on the input size are run during forward().
2024-08-26 14:38:10,744 - INFO - Downloaded: /workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_0/pretrained_model/swin_unetr.base_5000ep_f48_lr2e-4_pretrained.pt
2024-08-26 14:38:10,744 - INFO - Expected md5 is None, skip md5 check for file /workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_0/pretrained_model/swin_unetr.base_5000ep_f48_lr2e-4_pretrained.pt.
2024/08/26 14:38:11 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
swinunetr_0 - training ...:   6%|████▌                                                                           | 7/123 [1:38:31<27:12:43, 844.51s/round]
2024-08-26 16:23:58,012 - WARNING - swinunetr_0 - training: finished with early stop
2024-08-26 16:23:59,986 - INFO - ['python', '/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_1/scripts/train.py', 'run', "--config_file='/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_1/configs/hyper_parameters.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_1/configs/network.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_1/configs/transforms_infer.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_1/configs/transforms_train.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_1/configs/transforms_validate.yaml'"]
monai.networks.nets.swin_unetr SwinUNETR.__init__:img_size: Argument `img_size` has been deprecated since version 1.3. It will be removed in version 1.5. The img_size argument is not required anymore and checks on the input size are run during forward().
2024-08-26 16:24:25,515 - INFO - Downloaded: /workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_1/pretrained_model/swin_unetr.base_5000ep_f48_lr2e-4_pretrained.pt
2024-08-26 16:24:25,516 - INFO - Expected md5 is None, skip md5 check for file /workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_1/pretrained_model/swin_unetr.base_5000ep_f48_lr2e-4_pretrained.pt.
2024/08/26 16:24:25 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
swinunetr_1 - training ...:  12%|█████████▋                                                                     | 15/123 [3:16:50<23:37:13, 787.35s/round]
2024-08-26 19:48:49,743 - WARNING - swinunetr_1 - training: finished with early stop
2024-08-26 19:48:51,829 - INFO - ['python', '/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_2/scripts/train.py', 'run', "--config_file='/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_2/configs/hyper_parameters.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_2/configs/network.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_2/configs/transforms_infer.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_2/configs/transforms_train.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_2/configs/transforms_validate.yaml'"]
monai.networks.nets.swin_unetr SwinUNETR.__init__:img_size: Argument `img_size` has been deprecated since version 1.3. It will be removed in version 1.5. The img_size argument is not required anymore and checks on the input size are run during forward().
2024-08-26 19:49:01,723 - INFO - Downloaded: /workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_2/pretrained_model/swin_unetr.base_5000ep_f48_lr2e-4_pretrained.pt
2024-08-26 19:49:01,724 - INFO - Expected md5 is None, skip md5 check for file /workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_2/pretrained_model/swin_unetr.base_5000ep_f48_lr2e-4_pretrained.pt.
2024/08/26 19:49:01 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
swinunetr_2 - training ...:  11%|████████▉                                                                      | 14/123 [3:05:00<24:00:27, 792.91s/round]
2024-08-26 23:01:44,110 - WARNING - swinunetr_2 - training: finished with early stop
2024-08-26 23:01:46,219 - INFO - ['python', '/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_3/scripts/train.py', 'run', "--config_file='/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_3/configs/hyper_parameters.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_3/configs/network.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_3/configs/transforms_infer.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_3/configs/transforms_train.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_3/configs/transforms_validate.yaml'"]
monai.networks.nets.swin_unetr SwinUNETR.__init__:img_size: Argument `img_size` has been deprecated since version 1.3. It will be removed in version 1.5. The img_size argument is not required anymore and checks on the input size are run during forward().
2024-08-26 23:01:55,547 - INFO - Downloaded: /workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_3/pretrained_model/swin_unetr.base_5000ep_f48_lr2e-4_pretrained.pt
2024-08-26 23:01:55,547 - INFO - Expected md5 is None, skip md5 check for file /workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_3/pretrained_model/swin_unetr.base_5000ep_f48_lr2e-4_pretrained.pt.
2024/08/26 23:01:55 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
swinunetr_3 - training ...:  11%|████████▉                                                                      | 14/123 [3:04:04<23:53:06, 788.87s/round]
2024-08-27 02:13:10,402 - WARNING - swinunetr_3 - training: finished with early stop
2024-08-27 02:13:12,505 - INFO - ['python', '/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_4/scripts/train.py', 'run', "--config_file='/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_4/configs/hyper_parameters.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_4/configs/network.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_4/configs/transforms_infer.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_4/configs/transforms_train.yaml,/workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_4/configs/transforms_validate.yaml'"]
monai.networks.nets.swin_unetr SwinUNETR.__init__:img_size: Argument `img_size` has been deprecated since version 1.3. It will be removed in version 1.5. The img_size argument is not required anymore and checks on the input size are run during forward().
2024-08-27 02:13:21,974 - INFO - Downloaded: /workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_4/pretrained_model/swin_unetr.base_5000ep_f48_lr2e-4_pretrained.pt
2024-08-27 02:13:21,974 - INFO - Expected md5 is None, skip md5 check for file /workspaces/prostate_opi_year_3/data/work_dir_b/swinunetr_4/pretrained_model/swin_unetr.base_5000ep_f48_lr2e-4_pretrained.pt.
2024/08/27 02:13:22 INFO mlflow.tracking.fluent: Experiment with name 'Auto3DSeg' does not exist. Creating a new experiment.
swinunetr_4 - training ...:  15%|████████████▏                                                                  | 19/123 [4:05:10<22:21:59, 774.23s/round]
2024-08-27 06:25:33,346 - WARNING - swinunetr_4 - training: finished with early stop
2024-08-27 06:25:35,516 - INFO - Ensembling using single GPU!
2024-08-27 06:25:35,516 - INFO - The output_dir is not specified. /workspaces/prostate_opi_year_3/data/work_dir_b/ensemble_output will be used to save ensemble predictions.
2024-08-27 06:25:35,516 - INFO - Directory /workspaces/prostate_opi_year_3/data/work_dir_b/ensemble_output is created to save ensemble predictions
2024-08-27 06:25:35,822 - INFO - Auto3Dseg picked the following networks to ensemble:
2024-08-27 06:25:35,822 - INFO - segresnet_0
2024-08-27 06:25:35,822 - INFO - segresnet_1
2024-08-27 06:25:35,822 - INFO - segresnet_2
2024-08-27 06:25:35,823 - INFO - segresnet_3
2024-08-27 06:25:35,823 - INFO - segresnet_4
2024-08-27 06:25:35,823 - INFO - Auto3Dseg ensemble prediction outputs will be saved in /workspaces/prostate_opi_year_3/data/work_dir_b/ensemble_output.
Ensembling (rank 0)...:   0%|                                                                                                      | 0/55 [00:00<?, ?it/s]`apex.normalization.InstanceNorm3dNVFuser` is not installed properly, use nn.InstanceNorm3d instead.
Ensembling (rank 0)...: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 55/55 [11:00<00:00, 12.01s/it]
Auto3Dseg pipeline is completed successfully.

Expected behavior
files (outputs of infrence of the test set) should be present in /workspaces/prostate_opi_year_3/data/work_dir_b/ensemble_output

Environment

Ensuring you use the relevant python executable, please paste the output of:

python -c "import monai; monai.config.print_debug_info()"

root@jm-l4-1-vm:/workspaces/prostate_opi_year_3# python -c "import monai; monai.config.print_debug_info()"
================================
Printing MONAI config...
================================
MONAI version: 1.4.0rc1+1.g4877767c
Numpy version: 1.24.4
Pytorch version: 2.3.0a0+40ec155e58.nv24.03
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: 4877767cf92649a38ffda0fc590f2b92ba59f019
MONAI __file__: /usr/local/lib/python3.10/dist-packages/monai/__init__.py

Optional dependencies:
Pytorch Ignite version: 0.4.11
ITK version: 5.4.0
Nibabel version: 5.2.1
scikit-image version: 0.23.2
scipy version: 1.12.0
Pillow version: 10.2.0
Tensorboard version: 2.17.0
gdown version: 5.2.0
TorchVision version: 0.18.0a0
tqdm version: 4.66.2
lmdb version: 1.5.1
psutil version: 5.9.4
pandas version: 1.5.3
einops version: 0.7.0
transformers version: 4.40.2
mlflow version: 2.15.1
pynrrd version: 1.0.0
clearml version: 1.16.3

For details about installing the optional dependencies, please visit:
    https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies


================================
Printing system config...
================================
System: Linux
Linux version: Ubuntu 22.04.4 LTS
Platform: Linux-6.5.0-1018-gcp-x86_64-with-glibc2.35
Processor: x86_64
Machine: x86_64
Python version: 3.10.12
Process name: python
Command: ['python', '-c', 'import monai; monai.config.print_debug_info()']
Open files: [popenfile(path='/root/.vscode-server/data/logs/20240815T180848/network.log', fd=20, position=0, mode='a', flags=33793), popenfile(path='/root/.vscode-server/data/logs/20240815T180848/ptyhost.log', fd=21, position=9870, mode='a', flags=33793), popenfile(path='/root/.vscode-server/data/logs/20240815T180848/remoteagent.log', fd=22, position=1283022, mode='a', flags=33793)]
Num physical CPUs: 8
Num logical CPUs: 16
Num usable CPUs: 16
CPU usage (%): [2.9, 2.7, 4.8, 2.6, 2.7, 2.4, 2.8, 15.8, 3.4, 3.6, 2.6, 2.6, 2.8, 3.6, 2.9, 84.7]
CPU freq. (MHz): 2000
Load avg. in last 1, 5, 15 mins (%): [0.9, 0.5, 0.2]
Disk usage (%): 14.7
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 102.2
Available memory (GB): 96.4
Used memory (GB): 4.8

================================
Printing GPU config...
================================
Num GPUs: 2
Has CUDA: True
CUDA version: 12.4
cuDNN enabled: True
NVIDIA_TF32_OVERRIDE: None
TORCH_ALLOW_TF32_CUBLAS_OVERRIDE: 1
cuDNN version: 90000
Current device: 0
Library compiled for CUDA architectures: ['sm_52', 'sm_60', 'sm_61', 'sm_70', 'sm_72', 'sm_75', 'sm_80', 'sm_86', 'sm_87', 'sm_90', 'compute_90']
GPU 0 Name: Tesla V100-SXM2-16GB
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 80
GPU 0 Total memory (GB): 15.8
GPU 0 CUDA capability (maj.min): 7.0
GPU 1 Name: Tesla V100-SXM2-16GB
GPU 1 Is integrated: False
GPU 1 Is multi GPU board: False
GPU 1 Multi processor count: 80
GPU 1 Total memory (GB): 15.8
GPU 1 CUDA capability (maj.min): 7.0

Additional context
Dockerfile based on monai dockerfile

ARG PYTORCH_IMAGE=nvcr.io/nvidia/pytorch:24.03-py3
FROM ${PYTORCH_IMAGE}

LABEL maintainer="monai.contact@gmail.com"

# TODO: remark for issue [revise the dockerfile](https://github.com/zarr-developers/numcodecs/issues/431)
RUN if [[ $(uname -m) =~ "aarch64" ]]; then \
      export CFLAGS="-O3" && \
      export DISABLE_NUMCODECS_SSE2=true && \
      export DISABLE_NUMCODECS_AVX2=true && \
      pip install numcodecs; \
    fi

WORKDIR /opt/monai

# install full deps
COPY requirements.txt requirements-min.txt requirements-dev.txt /tmp/
RUN cp /tmp/requirements.txt /tmp/req.bak \
  && awk '!/torch/' /tmp/requirements.txt > /tmp/tmp && mv /tmp/tmp /tmp/requirements.txt \
  && python -m pip install --upgrade --no-cache-dir pip \
  && python -m pip install --no-cache-dir -r /tmp/requirements-dev.txt

# compile ext and remove temp files
# TODO: remark for issue [revise the dockerfile #1276](https://github.com/Project-MONAI/MONAI/issues/1276)
# please specify exact files and folders to be copied -- else, basically always, the Docker build process cannot cache
# this or anything below it and always will build from at most here; one file change leads to no caching from here on...

COPY LICENSE CHANGELOG.md CODE_OF_CONDUCT.md CONTRIBUTING.md README.md versioneer.py setup.py setup.cfg runtests.sh MANIFEST.in ./
COPY tests ./tests
COPY monai ./monai
# RUN BUILD_MONAI=1 FORCE_CUDA=1 python setup.py develop \
#   && rm -rf build __pycache__
RUN python -m pip install git+https://github.com/Project-MONAI/MONAI#egg=monai

# NGC Client
WORKDIR /opt/tools
ARG NGC_CLI_URI="https://ngc.nvidia.com/downloads/ngccli_linux.zip"
RUN wget -q ${NGC_CLI_URI} && unzip ngccli_linux.zip && chmod u+x ngc-cli/ngc && \
    find ngc-cli/ -type f -exec md5sum {} + | LC_ALL=C sort | md5sum -c ngc-cli.md5 && \
    rm -rf ngccli_linux.zip ngc-cli.md5
ENV PATH=${PATH}:/opt/tools:/opt/tools/ngc-cli
RUN apt-get update \
  && DEBIAN_FRONTEND="noninteractive" apt-get install -y libopenslide0  \
  && rm -rf /var/lib/apt/lists/*
# append /opt/tools to runtime path for NGC CLI to be accessible from all file system locations
ENV PATH=${PATH}:/opt/tools
WORKDIR /opt/monai

Start by tracing AutoRunner.run from the provided datalist through the testing and ensemble_output stages, using the listed training and inference logs to identify where output generation stops. Verify how the testing entries are handled and where result files are expected to be written. Done means inference completes with prediction files present in ensemble_output for the test set.