CUDA is not getting find by Monailabel

Describe the bug
I am using pc with gpu nvidia rtx 4070 with 12gb memory and is cuda enabeled. I have installed pytorch with cuda option. the Monailabel is still not able to find or work with cuda on the pc. can you please help with that.

Server logs
2023-03-13 23:19:59,209 - USING:: version = False
2023-03-13 23:19:59,209 - USING:: app = C:\Users\keyur\apps\radiology
2023-03-13 23:19:59,209 - USING:: studies = C:\Users\keyur\datasets\training
2023-03-13 23:19:59,209 - USING:: verbose = INFO
2023-03-13 23:19:59,209 - USING:: conf = [['models', 'segmentation']]
2023-03-13 23:19:59,210 - USING:: host = 0.0.0.0
2023-03-13 23:19:59,210 - USING:: port = 8000
2023-03-13 23:19:59,210 - USING:: uvicorn_app = monailabel.app:app
2023-03-13 23:19:59,210 - USING:: ssl_keyfile = None
2023-03-13 23:19:59,210 - USING:: ssl_certfile = None
2023-03-13 23:19:59,210 - USING:: ssl_keyfile_password = None
2023-03-13 23:19:59,211 - USING:: ssl_ca_certs = None
2023-03-13 23:19:59,211 - USING:: workers = None
2023-03-13 23:19:59,211 - USING:: limit_concurrency = None
2023-03-13 23:19:59,211 - USING:: access_log = False
2023-03-13 23:19:59,211 - USING:: log_config = None
2023-03-13 23:19:59,211 - USING:: dryrun = False
2023-03-13 23:19:59,212 - USING:: action = start_server
2023-03-13 23:19:59,212 - ENV SETTINGS:: MONAI_LABEL_API_STR =
2023-03-13 23:19:59,212 - ENV SETTINGS:: MONAI_LABEL_PROJECT_NAME = MONAILabel
2023-03-13 23:19:59,212 - ENV SETTINGS:: MONAI_LABEL_APP_DIR =
2023-03-13 23:19:59,212 - ENV SETTINGS:: MONAI_LABEL_STUDIES =
2023-03-13 23:19:59,212 - ENV SETTINGS:: MONAI_LABEL_AUTH_ENABLE = False
2023-03-13 23:19:59,212 - ENV SETTINGS:: MONAI_LABEL_AUTH_DB =
2023-03-13 23:19:59,213 - ENV SETTINGS:: MONAI_LABEL_APP_CONF = '{}'
2023-03-13 23:19:59,213 - ENV SETTINGS:: MONAI_LABEL_TASKS_TRAIN = True
2023-03-13 23:19:59,213 - ENV SETTINGS:: MONAI_LABEL_TASKS_STRATEGY = True
2023-03-13 23:19:59,213 - ENV SETTINGS:: MONAI_LABEL_TASKS_SCORING = True
2023-03-13 23:19:59,213 - ENV SETTINGS:: MONAI_LABEL_TASKS_BATCH_INFER = True
2023-03-13 23:19:59,213 - ENV SETTINGS:: MONAI_LABEL_DATASTORE =
2023-03-13 23:19:59,213 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_URL =
2023-03-13 23:19:59,213 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_USERNAME =
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_PASSWORD =
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_API_KEY =
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_CACHE_PATH =
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_PROJECT =
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_ASSET_PATH =
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_DSA_ANNOTATION_GROUPS =
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_USERNAME =
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_PASSWORD =
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_CACHE_PATH =
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_QIDO_PREFIX = None
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_WADO_PREFIX = None
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_STOW_PREFIX = None
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_FETCH_BY_FRAME = False
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_CONVERT_TO_NIFTI = True
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_SEARCH_FILTER = '{"Modality": "CT"}'
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_CACHE_EXPIRY = 180
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_PROXY_TIMEOUT = 30.0
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_READ_TIMEOUT = 5.0
2023-03-13 23:19:59,214 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_AUTO_RELOAD = True
2023-03-13 23:19:59,215 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_READ_ONLY = False
2023-03-13 23:19:59,215 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_FILE_EXT = '[".nii.gz", ".nii", ".nrrd", ".jpg", ".png", ".tif", ".svs", ".xml"]'
2023-03-13 23:19:59,215 - ENV SETTINGS:: MONAI_LABEL_SERVER_PORT = 8000
2023-03-13 23:19:59,215 - ENV SETTINGS:: MONAI_LABEL_CORS_ORIGINS = '[]'
2023-03-13 23:19:59,215 - ENV SETTINGS:: MONAI_LABEL_SESSIONS = True
2023-03-13 23:19:59,215 - ENV SETTINGS:: MONAI_LABEL_SESSION_PATH =
2023-03-13 23:19:59,215 - ENV SETTINGS:: MONAI_LABEL_SESSION_EXPIRY = 3600
2023-03-13 23:19:59,215 - ENV SETTINGS:: MONAI_LABEL_INFER_CONCURRENCY = -1
2023-03-13 23:19:59,215 - ENV SETTINGS:: MONAI_LABEL_INFER_TIMEOUT = 600
2023-03-13 23:19:59,216 - ENV SETTINGS:: MONAI_LABEL_TRACKING_ENABLED = True
2023-03-13 23:19:59,216 - ENV SETTINGS:: MONAI_LABEL_TRACKING_URI =
2023-03-13 23:19:59,216 - ENV SETTINGS:: MONAI_ZOO_SOURCE = github
2023-03-13 23:19:59,216 - ENV SETTINGS:: MONAI_ZOO_REPO = Project-MONAI/model-zoo/hosting_storage_v1
2023-03-13 23:19:59,219 - ENV SETTINGS:: MONAI_ZOO_AUTH_TOKEN =
2023-03-13 23:19:59,219 - ENV SETTINGS:: MONAI_LABEL_AUTO_UPDATE_SCORING = True
2023-03-13 23:19:59,219 -
Allow Origins: ['']
[2023-03-13 23:20:14,054] [3404] [MainThread] [INFO] (uvicorn.error:75) - Started server process [3404]
[2023-03-13 23:20:14,056] [3404] [MainThread] [INFO] (uvicorn.error:45) - Waiting for application startup.
[2023-03-13 23:20:14,058] [3404] [MainThread] [INFO] (monailabel.interfaces.utils.app:37) - Initializing App from: C:\Users\keyur\apps\radiology; studies: C:\Users\keyur\datasets\training; conf: {'models': 'segmentation'}
[2023-03-13 23:20:14,274] [3404] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for MONAILabelApp Found: <class 'main.MyApp'>
[2023-03-13 23:20:14,290] [3404] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepedit.DeepEdit'>
[2023-03-13 23:20:14,292] [3404] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_2d.Deepgrow2D'>
[2023-03-13 23:20:14,293] [3404] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_3d.Deepgrow3D'>
[2023-03-13 23:20:14,296] [3404] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.localization_spine.LocalizationSpine'>
[2023-03-13 23:20:14,300] [3404] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.localization_vertebra.LocalizationVertebra'>
[2023-03-13 23:20:14,324] [3404] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation.Segmentation'>
[2023-03-13 23:20:14,333] [3404] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_spleen.SegmentationSpleen'>
[2023-03-13 23:20:14,348] [3404] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_vertebra.SegmentationVertebra'>
[2023-03-13 23:20:14,362] [3404] [MainThread] [INFO] (main:93) - +++ Adding Model: segmentation => lib.configs.segmentation.Segmentation
[2023-03-13 23:20:14,451] [3404] [MainThread] [INFO] (main:96) - +++ Using Models: ['segmentation']
[2023-03-13 23:20:14,459] [3404] [MainThread] [INFO] (monailabel.interfaces.app:134) - Init Datastore for: C:\Users\keyur\datasets\training
[2023-03-13 23:20:14,472] [3404] [MainThread] [INFO] (monailabel.datastore.local:130) - Auto Reload: True; Extensions: ['.nii.gz', '.nii', '.nrrd', '.jpg', '.png', '.tif', '.svs', '.xml']
[2023-03-13 23:20:14,507] [3404] [MainThread] [INFO] (monailabel.datastore.local:577) - Invalidate count: 0
[2023-03-13 23:20:14,519] [3404] [MainThread] [INFO] (monailabel.datastore.local:151) - Start observing external modifications on datastore (AUTO RELOAD)
[2023-03-13 23:20:14,538] [3404] [MainThread] [INFO] (main:126) - +++ Adding Inferer:: segmentation => <lib.infers.segmentation.Segmentation object at 0x000001F12698C2E0>
[2023-03-13 23:20:14,539] [3404] [MainThread] [INFO] (main:191) - {'segmentation': <lib.infers.segmentation.Segmentation object at 0x000001F12698C2E0>, 'Histogram+GraphCut': <monailabel.scribbles.infer.HistogramBasedGraphCut object at 0x000001F12AB54580>, 'GMM+GraphCut': <monailabel.scribbles.infer.GMMBasedGraphCut object at 0x000001F12AB54550>}
[2023-03-13 23:20:14,540] [3404] [MainThread] [INFO] (main:206) - +++ Adding Trainer:: segmentation => <lib.trainers.segmentation.Segmentation object at 0x000001F12AB7E6A0>
[2023-03-13 23:20:14,542] [3404] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: C:\Users\keyur.cache\monailabel\sessions
[2023-03-13 23:20:14,543] [3404] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2023-03-13 23:20:14,543] [3404] [MainThread] [INFO] (monailabel.interfaces.app:468) - App Init - completed
[2023-03-13 23:20:14,544] [timeloop] [INFO] Starting Timeloop..
[2023-03-13 23:20:14,544] [3404] [MainThread] [INFO] (timeloop:60) - Starting Timeloop..
[2023-03-13 23:20:14,549] [timeloop] [INFO] Registered job <function MONAILabelApp.on_init_complete..run_scheduler at 0x000001F1269F0700>
[2023-03-13 23:20:14,549] [3404] [MainThread] [INFO] (timeloop:42) - Registered job <function MONAILabelApp.on_init_complete..run_scheduler at 0x000001F1269F0700>
[2023-03-13 23:20:14,549] [timeloop] [INFO] Timeloop now started. Jobs will run based on the interval set
[2023-03-13 23:20:14,549] [3404] [MainThread] [INFO] (timeloop:63) - Timeloop now started. Jobs will run based on the interval set
[2023-03-13 23:20:14,550] [3404] [MainThread] [INFO] (uvicorn.error:59) - Application startup complete.
[2023-03-13 23:20:14,552] [3404] [MainThread] [INFO] (uvicorn.error:206) - Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
[2023-03-13 23:20:36,423] [3404] [MainThread] [INFO] (monailabel.endpoints.activelearning:43) - Active Learning Request: {'strategy': 'random', 'client_id': 'user-xyz'}
[2023-03-13 23:21:50,060] [3404] [MainThread] [INFO] (monailabel.endpoints.datastore:67) - Image: 23.02.14.17; File: <starlette.datastructures.UploadFile object at 0x000001F12AF3F7F0>; params: {"client_id": "user-xyz"}
[2023-03-13 23:21:50,116] [3404] [MainThread] [INFO] (monailabel.datastore.local:439) - Adding Image: 23.02.14.17 => C:\Users\keyur\AppData\Local\Temp\tmpv0s5oyc5.nii.gz
[2023-03-13 23:21:50,698] [3404] [MainThread] [INFO] (monailabel.endpoints.datastore:100) - Saving Label for 23.02.14.17 for tag: final by admin
[2023-03-13 23:21:50,702] [3404] [MainThread] [INFO] (monailabel.endpoints.datastore:111) - Save Label params: {"label_info": [{"name": "liver", "idx": 1}, {"name": "venaporta", "idx": 2}, {"name": "livervein", "idx": 3}, {"name": "venacava", "idx": 4}, {"name": "tumor", "idx": 5}], "client_id": "user-xyz"}
[2023-03-13 23:21:50,703] [3404] [MainThread] [INFO] (monailabel.datastore.local:486) - Saving Label for Image: 23.02.14.17; Tag: final; Info: {'label_info': [{'name': 'liver', 'idx': 1}, {'name': 'venaporta', 'idx': 2}, {'name': 'livervein', 'idx': 3}, {'name': 'venacava', 'idx': 4}, {'name': 'tumor', 'idx': 5}], 'client_id': 'user-xyz'}
[2023-03-13 23:21:50,704] [3404] [MainThread] [INFO] (monailabel.datastore.local:494) - Adding Label: 23.02.14.17 => final => C:\Users\keyur\AppData\Local\Temp\tmp2qgif4r_.nii.gz
[2023-03-13 23:21:50,710] [3404] [MainThread] [INFO] (monailabel.datastore.local:510) - Label Info: {'label_info': [{'name': 'liver', 'idx': 1}, {'name': 'venaporta', 'idx': 2}, {'name': 'livervein', 'idx': 3}, {'name': 'venacava', 'idx': 4}, {'name': 'tumor', 'idx': 5}], 'client_id': 'user-xyz', 'ts': 1678746110, 'name': '23.02.14.17.nii.gz'}
[2023-03-13 23:21:50,715] [3404] [MainThread] [INFO] (monailabel.interfaces.app:492) - New label saved for: 23.02.14.17 => 23.02.14.17
[2023-03-13 23:21:50,775] [3404] [Thread-1] [INFO] (monailabel.datastore.local:624) - Adding New Label: final5 => 23.02.14.17 => 23.02.14.17.nii
[2023-03-13 23:21:50,780] [3404] [Thread-1] [INFO] (monailabel.datastore.local:577) - Invalidate count: 1
[2023-03-13 23:21:55,028] [3404] [MainThread] [INFO] (monailabel.utils.async_tasks.task:41) - Train request: {'model': 'segmentation', 'name': 'train_01', 'pretrained': True, 'device': 'cuda', 'max_epochs': 50, 'early_stop_patience': -1, 'val_split': 0.2, 'train_batch_size': 1, 'val_batch_size': 1, 'multi_gpu': True, 'gpus': 'all', 'dataset': 'SmartCacheDataset', 'dataloader': 'ThreadDataLoader', 'tracking': 'mlflow', 'tracking_uri': '', 'tracking_experiment_name': '', 'client_id': 'user-xyz'}
[2023-03-13 23:21:55,039] [3404] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:49) - Before:: C:\Users\keyur.conda\envs;
[2023-03-13 23:21:55,041] [3404] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:53) - After:: C:\Users\keyur.conda\envs;
[2023-03-13 23:21:55,042] [3404] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:65) - COMMAND:: C:\Users\keyur.conda\envs\monailabel-env\python.exe -m monailabel.interfaces.utils.app -m train -r {"model":"segmentation","name":"train_01","pretrained":true,"device":"cuda","max_epochs":50,"early_stop_patience":-1,"val_split":0.2,"train_batch_size":1,"val_batch_size":1,"multi_gpu":true,"gpus":"all","dataset":"SmartCacheDataset","dataloader":"ThreadDataLoader","tracking":"mlflow","tracking_uri":"","tracking_experiment_name":"","client_id":"user-xyz"}
[2023-03-13 23:21:55,668] [22344] [MainThread] [INFO] (main:37) - Initializing App from: C:\Users\keyur\apps\radiology; studies: C:\Users\keyur\datasets\training; conf: {'models': 'segmentation'}
[2023-03-13 23:21:59,895] [22344] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for MONAILabelApp Found: <class 'main.MyApp'>
[2023-03-13 23:21:59,903] [22344] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepedit.DeepEdit'>
[2023-03-13 23:21:59,903] [22344] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_2d.Deepgrow2D'>
[2023-03-13 23:21:59,904] [22344] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_3d.Deepgrow3D'>
[2023-03-13 23:21:59,905] [22344] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.localization_spine.LocalizationSpine'>
[2023-03-13 23:21:59,905] [22344] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.localization_vertebra.LocalizationVertebra'>
[2023-03-13 23:21:59,906] [22344] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation.Segmentation'>
[2023-03-13 23:21:59,907] [22344] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_spleen.SegmentationSpleen'>
[2023-03-13 23:21:59,907] [22344] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_vertebra.SegmentationVertebra'>
[2023-03-13 23:21:59,907] [22344] [MainThread] [INFO] (main:93) - +++ Adding Model: segmentation => lib.configs.segmentation.Segmentation
[2023-03-13 23:21:59,974] [22344] [MainThread] [INFO] (main:96) - +++ Using Models: ['segmentation']
[2023-03-13 23:21:59,975] [22344] [MainThread] [INFO] (monailabel.interfaces.app:134) - Init Datastore for: C:\Users\keyur\datasets\training
[2023-03-13 23:21:59,975] [22344] [MainThread] [INFO] (monailabel.datastore.local:130) - Auto Reload: False; Extensions: ['.nii.gz', '.nii', '.nrrd', '.jpg', '.png', '.tif', '.svs', '*.xml']
[2023-03-13 23:21:59,986] [22344] [MainThread] [INFO] (monailabel.datastore.local:577) - Invalidate count: 0
[2023-03-13 23:21:59,986] [22344] [MainThread] [INFO] (main:126) - +++ Adding Inferer:: segmentation => <lib.infers.segmentation.Segmentation object at 0x0000013C3E77D9D0>
[2023-03-13 23:21:59,986] [22344] [MainThread] [INFO] (main:191) - {'segmentation': <lib.infers.segmentation.Segmentation object at 0x0000013C3E77D9D0>, 'Histogram+GraphCut': <monailabel.scribbles.infer.HistogramBasedGraphCut object at 0x0000013C3FC5FF40>, 'GMM+GraphCut': <monailabel.scribbles.infer.GMMBasedGraphCut object at 0x0000013C3FC5FF10>}
[2023-03-13 23:21:59,986] [22344] [MainThread] [INFO] (main:206) - +++ Adding Trainer:: segmentation => <lib.trainers.segmentation.Segmentation object at 0x0000013C3FC5FFA0>
[2023-03-13 23:21:59,986] [22344] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: C:\Users\keyur.cache\monailabel\sessions
[2023-03-13 23:21:59,986] [22344] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2023-03-13 23:21:59,986] [22344] [MainThread] [INFO] (monailabel.tasks.train.basic_train:432) - Train Request (input): {'model': 'segmentation', 'name': 'train_01', 'pretrained': True, 'device': 'cuda', 'max_epochs': 50, 'early_stop_patience': -1, 'val_split': 0.2, 'train_batch_size': 1, 'val_batch_size': 1, 'multi_gpu': True, 'gpus': 'all', 'dataset': 'SmartCacheDataset', 'dataloader': 'ThreadDataLoader', 'tracking': 'mlflow', 'tracking_uri': '', 'tracking_experiment_name': '', 'client_id': 'user-xyz', 'local_rank': 0}
[2023-03-13 23:21:59,986] [22344] [MainThread] [INFO] (monailabel.tasks.train.basic_train:442) - CUDA_VISIBLE_DEVICES: None
[2023-03-13 23:21:59,988] [22344] [MainThread] [INFO] (monailabel.tasks.train.basic_train:447) - Distributed/Multi GPU is limited
[2023-03-13 23:21:59,988] [22344] [MainThread] [INFO] (monailabel.tasks.train.basic_train:462) - Distributed Training = FALSE
[2023-03-13 23:21:59,988] [22344] [MainThread] [INFO] (monailabel.tasks.train.basic_train:489) - 0 - Train Request (final): {'name': 'train_01', 'pretrained': True, 'device': 'cuda', 'max_epochs': 50, 'early_stop_patience': -1, 'val_split': 0.2, 'train_batch_size': 1, 'val_batch_size': 1, 'multi_gpu': False, 'gpus': 'all', 'dataset': 'SmartCacheDataset', 'dataloader': 'ThreadDataLoader', 'tracking': 'mlflow', 'tracking_uri': '', 'tracking_experiment_name': '', 'model': 'segmentation', 'client_id': 'user-xyz', 'local_rank': 0, 'run_id': '20230313_232159'}
[2023-03-13 23:21:59,988] [22344] [MainThread] [INFO] (monailabel.tasks.train.basic_train:622) - 0 - Using Device: cpu; IDX: None
[2023-03-13 23:21:59,988] [22344] [MainThread] [INFO] (monailabel.tasks.train.basic_train:515) - Run/Output Path: C:\Users\keyur\apps\radiology\model\segmentation\train_01
[2023-03-13 23:21:59,988] [22344] [MainThread] [INFO] (monailabel.tasks.train.basic_train:531) - Tracking: mlflow
[2023-03-13 23:21:59,988] [22344] [MainThread] [INFO] (monailabel.tasks.train.basic_train:532) - Tracking URI: file:///C:/Users/keyur/apps/radiology/model/segmentation/train_01/mlruns;
[2023-03-13 23:21:59,988] [22344] [MainThread] [INFO] (monailabel.tasks.train.basic_train:533) - Tracking Experiment Name: segmentation; Run Name: run_20230313_232159
[2023-03-13 23:21:59,989] [22344] [MainThread] [INFO] (monailabel.tasks.train.basic_train:410) - Total Records for Training: 7
[2023-03-13 23:21:59,989] [22344] [MainThread] [INFO] (monailabel.tasks.train.basic_train:411) - Total Records for Validation: 2
Loading dataset: 0%| | 0/2 [00:00<?, ?it/s]
Loading dataset: 50%|##### | 1/2 [00:04<00:04, 4.95s/it]
Loading dataset: 100%|##########| 2/2 [00:08<00:00, 4.10s/it]
Loading dataset: 100%|##########| 2/2 [00:08<00:00, 4.23s/it]
cache_num is greater or equal than dataset length, fall back to regular monai.data.CacheDataset.
[2023-03-13 23:22:08,462] [22344] [MainThread] [INFO] (monailabel.tasks.train.basic_train:328) - 0 - Records for Validation: 2
[2023-03-13 23:22:08,468] [22344] [MainThread] [INFO] (monailabel.tasks.train.basic_train:318) - 0 - Adding Validation to run every '1' interval
[2023-03-13 23:22:08,471] [22344] [MainThread] [INFO] (monailabel.tasks.train.basic_train:710) - 0 - Load Path C:\Users\keyur\apps\radiology\model\pretrained_segmentation.pt
Loading dataset: 0%| | 0/7 [00:00<?, ?it/s]
Loading dataset: 14%|#4 | 1/7 [00:07<00:47, 7.93s/it]
Loading dataset: 29%|##8 | 2/7 [00:13<00:33, 6.69s/it]
Loading dataset: 43%|####2 | 3/7 [00:19<00:25, 6.47s/it]
Loading dataset: 57%|#####7 | 4/7 [00:23<00:15, 5.14s/it]
Loading dataset: 71%|#######1 | 5/7 [00:28<00:10, 5.27s/it]
Loading dataset: 86%|########5 | 6/7 [00:35<00:05, 5.82s/it]
Loading dataset: 100%|##########| 7/7 [00:42<00:00, 6.14s/it]
Loading dataset: 100%|##########| 7/7 [00:42<00:00, 6.04s/it]
[2023-03-13 23:22:50,735] [22344] [MainThread] [INFO] (monailabel.tasks.train.basic_train:264) - 0 - Records for Training: 7
torch.cuda.amp.GradScaler is enabled, but CUDA is not available. Disabling.
[2023-03-13 23:22:50,750] [22344] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:876) - Engine run resuming from iteration 0, epoch 0 until 50 epochs
[2023-03-13 23:22:50,766] [22344] [MainThread] [ERROR] (ignite.engine.engine.SupervisedTrainer:992) - Engine run is terminating due to exception: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.
[2023-03-13 23:22:50,766] [22344] [MainThread] [ERROR] (ignite.engine.engine.SupervisedTrainer:180) - Exception: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.
Traceback (most recent call last):
File "C:\Users\keyur.conda\envs\monailabel-env\lib\site-packages\ignite\engine\engine.py", line 946, in _internal_run_as_gen
self._fire_event(Events.STARTED)
File "C:\Users\keyur.conda\envs\monailabel-env\lib\site-packages\ignite\engine\engine.py", line 425, in _fire_event
func(*first, *(event_args + others), **kwargs)
File "C:\Users\keyur.conda\envs\monailabel-env\lib\site-packages\monai\handlers\checkpoint_loader.py", line 107, in call
checkpoint = torch.load(self.load_path, map_location=self.map_location)
File "C:\Users\keyur.conda\envs\monailabel-env\lib\site-packages\torch\serialization.py", line 789, in load
return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
File "C:\Users\keyur.conda\envs\monailabel-env\lib\site-packages\torch\serialization.py", line 1131, in _load
result = unpickler.load()
File "C:\Users\keyur.conda\envs\monailabel-env\lib\site-packages\torch\serialization.py", line 1101, in persistent_load
load_tensor(dtype, nbytes, key, _maybe_decode_ascii(location))
File "C:\Users\keyur.conda\envs\monailabel-env\lib\site-packages\torch\serialization.py", line 1083, in load_tensor
wrap_storage=restore_location(storage, location),
File "C:\Users\keyur.conda\envs\monailabel-env\lib\site-packages\torch\serialization.py", line 215, in default_restore_location
result = fn(storage, location)
File "C:\Users\keyur.conda\envs\monailabel-env\lib\site-packages\torch\serialization.py", line 182, in _cuda_deserialize
device = validate_cuda_device(location)
File "C:\Users\keyur.conda\envs\monailabel-env\lib\site-packages\torch\serialization.py", line 166, in validate_cuda_device
raise RuntimeError('Attempting to deserialize object on a CUDA '
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.
Traceback (most recent call last):
File "C:\Users\keyur.conda\envs\monailabel-env\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\keyur.conda\envs\monailabel-env\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Users\keyur.conda\envs\monailabel-env\lib\site-packages\monailabel\interfaces\utils\app.py", line 128, in
run_main()
File "C:\Users\keyur.conda\envs\monailabel-env\lib\site-packages\monailabel\interfaces\utils\app.py", line 113, in run_main
result = a.train(request)
File "C:\Users\keyur.conda\envs\monailabel-env\lib\site-packages\monailabel\interfaces\app.py", line 422, in train
result = task(request, self.datastore())
File "C:\Users\keyur.conda\envs\monailabel-env\lib\site-packages\monailabel\tasks\train\basic_train.py", line 463, in call
res = self.train(0, world_size, req, datalist)
File "C:\Users\keyur.conda\envs\monailabel-env\lib\site-packages\monailabel\tasks\train\basic_train.py", line 552, in train
context.trainer.run()
File "C:\Users\keyur.conda\envs\monailabel-env\lib\site-packages\monai\engines\trainer.py", line 53, in run
super().run()
File "C:\Users\keyur.conda\envs\monailabel-env\lib\site-packages\monai\engines\workflow.py", line 281, in run
super().run(data=self.data_loader, max_epochs=self.state.max_epochs)
File "C:\Users\keyur.conda\envs\monailabel-env\lib\site-packages\ignite\engine\engine.py", line 892, in run
return self._internal_run()
File "C:\Users\keyur.conda\envs\monailabel-env\lib\site-packages\ignite\engine\engine.py", line 935, in _internal_run
return next(self._internal_run_generator)
File "C:\Users\keyur.conda\envs\monailabel-env\lib\site-packages\ignite\engine\engine.py", line 993, in _internal_run_as_gen
self._handle_exception(e)
File "C:\Users\keyur.conda\envs\monailabel-env\lib\site-packages\ignite\engine\engine.py", line 636, in _handle_exception
self._fire_event(Events.EXCEPTION_RAISED, e)
File "C:\Users\keyur.conda\envs\monailabel-env\lib\site-packages\ignite\engine\engine.py", line 425, in _fire_event
func(*first, *(event_args + others), **kwargs)
File "C:\Users\keyur.conda\envs\monailabel-env\lib\site-packages\monai\handlers\stats_handler.py", line 181, in exception_raised
raise e
File "C:\Users\keyur.conda\envs\monailabel-env\lib\site-packages\ignite\engine\engine.py", line 946, in _internal_run_as_gen
self._fire_event(Events.STARTED)
File "C:\Users\keyur.conda\envs\monailabel-env\lib\site-packages\ignite\engine\engine.py", line 425, in _fire_event
func(*first, *(event_args + others), **kwargs)
File "C:\Users\keyur.conda\envs\monailabel-env\lib\site-packages\monai\handlers\checkpoint_loader.py", line 107, in call
checkpoint = torch.load(self.load_path, map_location=self.map_location)
File "C:\Users\keyur.conda\envs\monailabel-env\lib\site-packages\torch\serialization.py", line 789, in load
return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
File "C:\Users\keyur.conda\envs\monailabel-env\lib\site-packages\torch\serialization.py", line 1131, in _load
result = unpickler.load()
File "C:\Users\keyur.conda\envs\monailabel-env\lib\site-packages\torch\serialization.py", line 1101, in persistent_load
load_tensor(dtype, nbytes, key, _maybe_decode_ascii(location))
File "C:\Users\keyur.conda\envs\monailabel-env\lib\site-packages\torch\serialization.py", line 1083, in load_tensor
wrap_storage=restore_location(storage, location),
File "C:\Users\keyur.conda\envs\monailabel-env\lib\site-packages\torch\serialization.py", line 215, in default_restore_location
result = fn(storage, location)
File "C:\Users\keyur.conda\envs\monailabel-env\lib\site-packages\torch\serialization.py", line 182, in _cuda_deserialize
device = validate_cuda_device(location)
File "C:\Users\keyur.conda\envs\monailabel-env\lib\site-packages\torch\serialization.py", line 166, in validate_cuda_device
raise RuntimeError('Attempting to deserialize object on a CUDA '
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.
[2023-03-13 23:22:51,452] [3404] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:83) - Return code: 1

Start by reproducing the report with the logged monailabel.app:app entry point and the radiology app configuration, then inspect the model initialization shown for lib.configs.segmentation.Segmentation and lib.infers.segmentation.Segmentation. Done means establishing whether the installed PyTorch build recognizes the NVIDIA GPU and whether MONAI Label uses it during inference.