AutoRunner GPU usage

Describe the bug
AutoRunner spikes CPU usage and ends with SIGKILL9. Pytorch recognizes GPU and other scripts utilize GPU.

To Reproduce
Steps to reproduce the behavior:

1. run autorunner on HPC GPU instance
2. run data analysis
3. run algorithm generation
4. fail occurs on Step 3: Model training, validation, and inference

Expected behavior
GPU utilized and no fail on run
Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• OS --> ubuntu 20.04 (also tried 22.04)

• Python version --> 3.8 (also tried 3.10.6)

• MONAI version [e.g. git commit hash] --> 1.1.0+21.g4b464e7b

• GPU models and configuration -->

• exouser@auto3dseg:~/Documents$ nvidia-smi
  Fri Jan 13 19:19:07 2023
  +-----------------------------------------------------------------------------+
  | NVIDIA-SMI 525.60.13 Driver Version: 525.60.13 CUDA Version: 12.0 |
  |-------------------------------+----------------------+----------------------+
  | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
  | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
  | | | MIG M. |
  |===============================+======================+======================|
  | 0 GRID A100X-40C On | 00000000:04:00.0 Off | 0 |
  | N/A N/A P0 N/A / N/A | 0MiB / 40960MiB | 0% Default |
  | | | Disabled |
  +-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+

Error:

exouser@auto3dseg:~/Documents$ python -m monai.apps.auto3dseg AutoRunner run --input='./task.yaml'
2023-01-13 15:12:06,572 - INFO - AutoRunner using work directory ./work_dir
2023-01-13 15:12:06,574 - INFO - Loading input config ./task.yaml
2023-01-13 15:12:06,604 - INFO - The output_dir is not specified. /home/exouser/Documents/work_dir/ensemble_output will be used to save ensemble predictions
2023-01-13 15:12:06,604 - INFO - Skipping data analysis...
2023-01-13 15:12:06,604 - INFO - Skipping algorithm generation...
2023-01-13 15:12:06,616 - INFO - Launching: python /home/exouser/Documents/work_dir/dints_0/scripts/search.py run --config_file='/home/exouser/Documents/work_dir/dints_0/configs/hyper_parameters_search.yaml','/home/exouser/Documents/work_dir/dints_0/configs/network.yaml','/home/exouser/Documents/work_dir/dints_0/configs/transforms_infer.yaml','/home/exouser/Documents/work_dir/dints_0/configs/transforms_validate.yaml','/home/exouser/Documents/work_dir/dints_0/configs/network_search.yaml','/home/exouser/Documents/work_dir/dints_0/configs/transforms_train.yaml','/home/exouser/Documents/work_dir/dints_0/configs/hyper_parameters.yaml'
[info] number of GPUs: 1
[info] world_size: 1
train_files_w: 64
train_files_a: 64
val_files: 33
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/exouser/Documents/MONAI/monai/apps/auto3dseg/main.py", line 24, in
fire.Fire(
File "/home/exouser/.local/lib/python3.8/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/home/exouser/.local/lib/python3.8/site-packages/fire/core.py", line 475, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/home/exouser/.local/lib/python3.8/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/home/exouser/Documents/MONAI/monai/apps/auto3dseg/auto_runner.py", line 685, in run
self._train_algo_in_sequence(history)
File "/home/exouser/Documents/MONAI/monai/apps/auto3dseg/auto_runner.py", line 557, in _train_algo_in_sequence
algo.train(self.train_params)
File "/home/exouser/Documents/work_dir/algorithm_templates/dints/scripts/algo.py", line 398, in train
self._run_cmd(cmd_search, devices_info)
File "/home/exouser/Documents/MONAI/monai/apps/auto3dseg/bundle_gen.py", line 191, in _run_cmd
normal_out = subprocess.run(cmd.split(), env=ps_environ, check=True)
File "/usr/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['python', '/home/exouser/Documents/work_dir/dints_0/scripts/search.py', 'run', "--config_file='/home/exouser/Documents/work_dir/dints_0/configs/hyper_parameters_search.yaml','/home/exouser/Documents/work_dir/dints_0/configs/network.yaml','/home/exouser/Documents/work_dir/dints_0/configs/transforms_infer.yaml','/home/exouser/Documents/work_dir/dints_0/configs/transforms_validate.yaml','/home/exouser/Documents/work_dir/dints_0/configs/network_search.yaml','/home/exouser/Documents/work_dir/dints_0/configs/transforms_train.yaml','/home/exouser/Documents/work_dir/dints_0/configs/hyper_parameters.yaml'"]' died with <Signals.SIGKILL: 9>.

CPU/GPU traces from start till crash:

まず monai/apps/auto3dseg/auto_runner.py の _train_algo_in_sequence と bundle_gen.py の _run_cmd から確認し、次に traceback にある生成済みの work_dir/dints_0/scripts/search.py コマンドを調べます。CPU と GPU の使用状況を監視しながら、Step 3 のトレーニング実行を再現します。AutoRunner が SIGKILL なしでトレーニング、検証、推論を完了し、利用可能な GPU を使用すれば完了です。