Running with multiple GPUs and have this problem, anyone know what's the reason?

Nobody has claimed this yet.

## Running with multiple GPUs and have this problem, anyone know what's the reason

```
[2023-07-28 04:28:39,671] [INFO] [real_accelerator.py:133:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-07-28 04:28:39,671] [INFO] [real_accelerator.py:133:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-07-28 04:28:39,671] [INFO] [real_accelerator.py:133:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-07-28 04:28:39,672] [INFO] [real_accelerator.py:133:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-07-28 04:28:39,677] [INFO] [real_accelerator.py:133:get_accelerator] Setting ds_accelerator to cuda (auto detect)
WARNING:root:Flash attention is only supported on A100 or H100 GPU during training due to head dim > 64 backward.ref: https://github.com/HazyResearch/flash-attention/issues/190#issuecomment-1523359593
WARNING:root:Flash attention is only supported on A100 or H100 GPU during training due to head dim > 64 backward.ref: https://github.com/HazyResearch/flash-attention/issues/190#issuecomment-1523359593
WARNING:root:Flash attention is only supported on A100 or H100 GPU during training due to head dim > 64 backward.ref: https://github.com/HazyResearch/flash-attention/issues/190#issuecomment-1523359593
WARNING:root:Flash attention is only supported on A100 or H100 GPU during training due to head dim > 64 backward.ref: https://github.com/HazyResearch/flash-attention/issues/190#issuecomment-1523359593
WARNING:root:Flash attention is only supported on A100 or H100 GPU during training due to head dim > 64 backward.ref: https://github.com/HazyResearch/flash-attention/issues/190#issuecomment-1523359593
/root/miniconda3/envs/myenv/lib/python3.11/site-packages/transformers/training_args.py:1516: FutureWarning: using `--fsdp_transformer_layer_cls_to_wrap` is deprecated. Use fsdp_config instead
warnings.warn(
/root/miniconda3/envs/myenv/lib/python3.11/site-packages/transformers/training_args.py:1516: FutureWarning: using `--fsdp_transformer_layer_cls_to_wrap` is deprecated. Use fsdp_config instead
warnings.warn(
/root/miniconda3/envs/myenv/lib/python3.11/site-packages/transformers/training_args.py:1516: FutureWarning: using `--fsdp_transformer_layer_cls_to_wrap` is deprecated. Use fsdp_config instead
warnings.warn(
/root/miniconda3/envs/myenv/lib/python3.11/site-packages/transformers/training_args.py:1516: FutureWarning: using `--fsdp_transformer_layer_cls_to_wrap` is deprecated. Use fsdp_config instead
warnings.warn(
/root/miniconda3/envs/myenv/lib/python3.11/site-packages/transformers/training_args.py:1516: FutureWarning: using `--fsdp_transformer_layer_cls_to_wrap` is deprecated. Use fsdp_config instead
warnings.warn(
Loading checkpoint shards: 0%| | 0/2 [00:00
sys.exit(load_entry_point('torch==2.0.1', 'console_scripts', 'torchrun')())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/myenv/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/myenv/lib/python3.11/site-packages/torch/distributed/run.py", line 794, in main
run(args)
File "/root/miniconda3/envs/myenv/lib/python3.11/site-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/root/miniconda3/envs/myenv/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/myenv/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
======================================================
fastchat/train/train_mem.py FAILED
------------------------------------------------------
Failures:

------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2023-07-28_04:30:01
host : autodl-container-fe7949915e-90c6881a
rank : 3 (local_rank: 3)
exitcode : -9 (pid: 42206)
error_file:
traceback : Signal 9 (SIGKILL) received by PID 42206
======================================================
```

The report runs fastchat/train/train_mem.py through torchrun; begin by reproducing the multi-GPU command and checking the rank-3 SIGKILL alongside the logged warnings. The issue does not name a test or requested code change, so done would require confirming the cause and defining an actionable fix.

Written by the indexing model from the issue text.