How to Train models under AMD Graphic Card

Nobody has claimed this yet.

Hi Teams, the work you've done is fantastic! I am genuinely impressed with the deployment on my local setup; everything looks great. However, I'm currently facing some challenges while trying to fine-tune it on my own.

Could anyone kindly provide me with some hints or guidance on how to modify the command for fine-tuning?

**System Info**

MacBook Pro (16-inch, 2019)
AMD Radeon Pro 5500M 4 GB

**Running Command**
```
torchrun --nproc_per_node=1 --master_port=9778 fastchat/train/train_flant5.py \
--model_name_or_path lmsys/fastchat-t5-3b-v1.0 \
--data_path /data/dummy.json \
--bf16 True \
--output_dir ./checkpoints_flant5_3b \
--num_train_epochs 3 \
--per_device_train_batch_size 1 \
--per_device_eval_batch_size 1 \
--gradient_accumulation_steps 4 \
--evaluation_strategy "no" \
--save_strategy "steps" \
--save_steps 300 \
--save_total_limit 1 \
--learning_rate 2e-5 \
--weight_decay 0. \
--warmup_ratio 0.03 \
--lr_scheduler_type "cosine" \
--logging_steps 1 \
--fsdp "full_shard auto_wrap" \
--fsdp_transformer_layer_cls_to_wrap T5Block \
--tf32 True \
--model_max_length 2048 \
--preprocessed_path ./preprocessed_data/processed.json \
--gradient_checkpointing True
```

**Error Message**
```
NOTE: Redirects are currently not supported in Windows or MacOs.
WARNING:torch.distributed.run:
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
*****************************************
Traceback (most recent call last):
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 790, in urlopen
response = self._make_request(
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 491, in _make_request
raise new_e
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 467, in _make_request
self._validate_conn(conn)
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1092, in _validate_conn
conn.connect()
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/urllib3/connection.py", line 635, in connect
sock_and_verified = _ssl_wrap_socket_and_match_hostname(
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/urllib3/connection.py", line 774, in _ssl_wrap_socket_and_match_hostname
ssl_sock = ssl_wrap_socket(
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 459, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 503, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/Users/nelson/anaconda3/lib/python3.10/ssl.py", line 513, in wrap_socket
return self.sslsocket_class._create(
File "/Users/nelson/anaconda3/lib/python3.10/ssl.py", line 1071, in _create
self.do_handshake()
File "/Users/nelson/anaconda3/lib/python3.10/ssl.py", line 1342, in do_handshake
self._sslobj.do_handshake()
ConnectionResetError: [Errno 54] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 844, in urlopen
retries = retries.increment(
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/urllib3/util/retry.py", line 470, in increment
raise reraise(type(error), error, _stacktrace)
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/urllib3/util/util.py", line 38, in reraise
raise value.with_traceback(tb)
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 790, in urlopen
response = self._make_request(
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 491, in _make_request
raise new_e
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 467, in _make_request
self._validate_conn(conn)
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1092, in _validate_conn
conn.connect()
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/urllib3/connection.py", line 635, in connect
sock_and_verified = _ssl_wrap_socket_and_match_hostname(
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/urllib3/connection.py", line 774, in _ssl_wrap_socket_and_match_hostname
ssl_sock = ssl_wrap_socket(
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 459, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 503, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/Users/nelson/anaconda3/lib/python3.10/ssl.py", line 513, in wrap_socket
return self.sslsocket_class._create(
File "/Users/nelson/anaconda3/lib/python3.10/ssl.py", line 1071, in _create
self.do_handshake()
File "/Users/nelson/anaconda3/lib/python3.10/ssl.py", line 1342, in do_handshake
self._sslobj.do_handshake()
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/nelson/Pictures/FastChat/fastchat/train/train_flant5.py", line 35, in
default_conversation = get_conversation_template("t5")
File "/Users/nelson/Pictures/FastChat/fastchat/model/model_adapter.py", line 256, in get_conversation_template
adapter = get_model_adapter(model_path)
File "/Users/nelson/Pictures/FastChat/fastchat/model/model_adapter.py", line 106, in get_model_adapter
if adapter.match(model_path_basename) and type(adapter) != BaseModelAdapter:
File "/Users/nelson/Pictures/FastChat/fastchat/model/model_adapter.py", line 393, in match
list_remote_files = list_repo_files(model_path)
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py", line 103, in inner_f
return f(*args, **kwargs)
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 120, in _inner_fn
return fn(*args, **kwargs)
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 1966, in list_repo_files
return [
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 1966, in
return [
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 1932, in list_files_info
for subpath_info in paginate(path=tree_url, headers=headers, params={"recursive": True}):
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/huggingface_hub/utils/_pagination.py", line 35, in paginate
r = session.get(path, params=params, headers=headers)
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/requests/sessions.py", line 600, in get
return self.request("GET", url, **kwargs)
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/requests/adapters.py", line 501, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))
Traceback (most recent call last):
File "/Users/nelson/Pictures/FastChat/fastchat/train/train_flant5.py", line 436, in
train()
File "/Users/nelson/Pictures/FastChat/fastchat/train/train_flant5.py", line 399, in train
model_args, data_args, training_args = parser.parse_args_into_dataclasses()
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/transformers/hf_argparser.py", line 332, in parse_args_into_dataclasses
obj = dtype(**inputs)
File "", line 112, in __init__
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/transformers/training_args.py", line 1259, in __post_init__
and (self.device.type != "cuda")
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/transformers/training_args.py", line 1694, in device
return self._setup_devices
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/transformers/utils/generic.py", line 54, in __get__
cached = self.fget(obj)
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/transformers/training_args.py", line 1679, in _setup_devices
Traceback (most recent call last):
File "/Users/nelson/Pictures/FastChat/fastchat/train/train_flant5.py", line 436, in
train()
File "/Users/nelson/Pictures/FastChat/fastchat/train/train_flant5.py", line 399, in train
model_args, data_args, training_args = parser.parse_args_into_dataclasses()
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/transformers/hf_argparser.py", line 332, in parse_args_into_dataclasses
torch.distributed.init_process_group(backend="nccl", timeout=self.ddp_timeout_delta)
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 907, in init_process_group
obj = dtype(**inputs)
File "", line 112, in __init__
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/transformers/training_args.py", line 1259, in __post_init__
and (self.device.type != "cuda")
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/transformers/training_args.py", line 1694, in device
return self._setup_devices
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/transformers/utils/generic.py", line 54, in __get__
cached = self.fget(obj)
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/transformers/training_args.py", line 1679, in _setup_devices
default_pg = _new_process_group_helper(
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1013, in _new_process_group_helper
raise RuntimeError("Distributed package doesn't have NCCL " "built in")
RuntimeError: Distributed package doesn't have NCCL built in
torch.distributed.init_process_group(backend="nccl", timeout=self.ddp_timeout_delta)
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 907, in init_process_group
default_pg = _new_process_group_helper(
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1013, in _new_process_group_helper
raise RuntimeError("Distributed package doesn't have NCCL " "built in")
RuntimeError: Distributed package doesn't have NCCL built in
Traceback (most recent call last):
File "/Users/nelson/Pictures/FastChat/fastchat/train/train_flant5.py", line 436, in
train()
File "/Users/nelson/Pictures/FastChat/fastchat/train/train_flant5.py", line 399, in train
model_args, data_args, training_args = parser.parse_args_into_dataclasses()
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/transformers/hf_argparser.py", line 332, in parse_args_into_dataclasses
obj = dtype(**inputs)
File "", line 112, in __init__
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/transformers/training_args.py", line 1259, in __post_init__
and (self.device.type != "cuda")
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/transformers/training_args.py", line 1694, in device
return self._setup_devices
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/transformers/utils/generic.py", line 54, in __get__
cached = self.fget(obj)
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/transformers/training_args.py", line 1679, in _setup_devices
torch.distributed.init_process_group(backend="nccl", timeout=self.ddp_timeout_delta)
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 907, in init_process_group
default_pg = _new_process_group_helper(
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 1013, in _new_process_group_helper
raise RuntimeError("Distributed package doesn't have NCCL " "built in")
RuntimeError: Distributed package doesn't have NCCL built in
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 78892) of binary: /Users/nelson/anaconda3/bin/python
Traceback (most recent call last):
File "/Users/nelson/anaconda3/bin/torchrun", line 8, in
sys.exit(main())
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
return f(*args, **kwargs)
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/torch/distributed/run.py", line 794, in main
run(args)
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/Users/nelson/anaconda3/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
fastchat/train/train_flant5.py FAILED
------------------------------------------------------------

```

Read fastchat/train/train_flant5.py and fastchat/model/model_adapter.py, then compare the supplied torchrun output with the training entry point and the reported NCCL initialization failure. There is no concrete repository change or acceptance condition in the issue, so a contributor cannot define "done" from the payload alone.

Written by the indexing model from the issue text.