请教：微调报错怎么解决？

硬件：单机，8张3090
配置：
command_file: null
commands: null
compute_environment: LOCAL_MACHINE
deepspeed_config:
gradient_accumulation_steps: 1
gradient_clipping: 1.0
offload_optimizer_device: none
offload_param_device: none
zero3_init_flag: true
zero3_save_16bit_model: true
zero_stage: 3
distributed_type: DEEPSPEED
downcast_bf16: 'no'
dynamo_backend: 'yes'
fsdp_config: {}
gpu_ids: null
machine_rank: 0
main_process_ip: 192.168.33.201
main_process_port: 21889
main_training_function: main
megatron_lm_config: {}
mixed_precision: fp16
num_machines: 1
num_processes: 8
rdzv_backend: static
same_network: true
tpu_name: null
tpu_zone: null
use_cpu: false
和
num_machines=1
num_processes=$((num_machines * 8))
machine_rank=0

accelerate launch
--config_file ./configs/sft.yaml
--num_processes $num_processes
--num_machines $num_machines
--machine_rank $machine_rank
--deepspeed_multinode_launcher standard finetune_moss.py
--model_name_or_path fnlp/moss-moon-003-sft-int4
--data_dir ./sft_data
--output_dir ./ckpts/moss-moon-003-sft-int4
--log_dir ./train_logs/moss-moon-003-sft-int4
--n_epochs 2
--train_bsz_per_gpu 4
--eval_bsz_per_gpu 4
--learning_rate 0.000015
--eval_step 15
--save_step 35 \

报错：
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.

---

INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7
INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6
INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
Explicitly passing a revision is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a revision is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a revision is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a revision is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a revision is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a revision is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Traceback (most recent call last):
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 449, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 444, in _make_request
httplib_response = conn.getresponse()
File "/opt/anaconda3/lib/python3.10/http/client.py", line 1374, in getresponse
response.begin()
File "/opt/anaconda3/lib/python3.10/http/client.py", line 318, in begin
version, status, reason = self._read_status()
File "/opt/anaconda3/lib/python3.10/http/client.py", line 287, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/util/retry.py", line 550, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/packages/six.py", line 769, in reraise
raise value.with_traceback(tb)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 449, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 444, in _make_request
httplib_response = conn.getresponse()
File "/opt/anaconda3/lib/python3.10/http/client.py", line 1374, in getresponse
response.begin()
File "/opt/anaconda3/lib/python3.10/http/client.py", line 318, in begin
version, status, reason = self._read_status()
File "/opt/anaconda3/lib/python3.10/http/client.py", line 287, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/zhangzhong/MOSS/finetune_moss.py", line 305, in
train(args)
File "/home/zhangzhong/MOSS/finetune_moss.py", line 177, in train
tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path, trust_remote_code=True)
File "/opt/anaconda3/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 626, in from_pretrained
tokenizer_class = get_class_from_dynamic_module(
File "/opt/anaconda3/lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 363, in get_class_from_dynamic_module
final_module = get_cached_module_file(
File "/opt/anaconda3/lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 261, in get_cached_module_file
commit_hash = model_info(pretrained_model_name_or_path, revision=revision, token=token).sha
File "/opt/anaconda3/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 94, in _inner_fn
return fn(*args, **kwargs)
File "/opt/anaconda3/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py", line 98, in inner_f
return f(*args, **kwargs)
File "/opt/anaconda3/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 1283, in model_info
r = requests.get(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/adapters.py", line 501, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
Traceback (most recent call last):
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 449, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 444, in _make_request
httplib_response = conn.getresponse()
File "/opt/anaconda3/lib/python3.10/http/client.py", line 1374, in getresponse
response.begin()
File "/opt/anaconda3/lib/python3.10/http/client.py", line 318, in begin
version, status, reason = self._read_status()
File "/opt/anaconda3/lib/python3.10/http/client.py", line 287, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/util/retry.py", line 550, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/packages/six.py", line 769, in reraise
raise value.with_traceback(tb)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 449, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 444, in _make_request
httplib_response = conn.getresponse()
File "/opt/anaconda3/lib/python3.10/http/client.py", line 1374, in getresponse
response.begin()
File "/opt/anaconda3/lib/python3.10/http/client.py", line 318, in begin
version, status, reason = self._read_status()
File "/opt/anaconda3/lib/python3.10/http/client.py", line 287, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/zhangzhong/MOSS/finetune_moss.py", line 305, in
train(args)
File "/home/zhangzhong/MOSS/finetune_moss.py", line 177, in train
tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path, trust_remote_code=True)
File "/opt/anaconda3/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 626, in from_pretrained
tokenizer_class = get_class_from_dynamic_module(
File "/opt/anaconda3/lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 363, in get_class_from_dynamic_module
final_module = get_cached_module_file(
File "/opt/anaconda3/lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 261, in get_cached_module_file
commit_hash = model_info(pretrained_model_name_or_path, revision=revision, token=token).sha
File "/opt/anaconda3/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 94, in _inner_fn
return fn(*args, **kwargs)
File "/opt/anaconda3/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py", line 98, in inner_f
return f(*args, **kwargs)
File "/opt/anaconda3/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 1283, in model_info
r = requests.get(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/adapters.py", line 501, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
Explicitly passing a revision is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a revision is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Traceback (most recent call last):
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1042, in validate_conn
conn.connect()
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connection.py", line 419, in connect
self.sock = ssl_wrap_socket(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/util/ssl.py", line 449, in ssl_wrap_socket
ssl_sock = ssl_wrap_socket_impl(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/util/ssl.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/opt/anaconda3/lib/python3.10/ssl.py", line 513, in wrap_socket
return self.sslsocket_class._create(
File "/opt/anaconda3/lib/python3.10/ssl.py", line 1071, in _create
self.do_handshake()
File "/opt/anaconda3/lib/python3.10/ssl.py", line 1342, in do_handshake
self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/util/retry.py", line 550, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/packages/six.py", line 769, in reraise
raise value.with_traceback(tb)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1042, in validate_conn
conn.connect()
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connection.py", line 419, in connect
self.sock = ssl_wrap_socket(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/util/ssl.py", line 449, in ssl_wrap_socket
ssl_sock = ssl_wrap_socket_impl(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/util/ssl.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/opt/anaconda3/lib/python3.10/ssl.py", line 513, in wrap_socket
return self.sslsocket_class._create(
File "/opt/anaconda3/lib/python3.10/ssl.py", line 1071, in _create
self.do_handshake()
File "/opt/anaconda3/lib/python3.10/ssl.py", line 1342, in do_handshake
self._sslobj.do_handshake()
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/zhangzhong/MOSS/finetune_moss.py", line 305, in
train(args)
File "/home/zhangzhong/MOSS/finetune_moss.py", line 177, in train
tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path, trust_remote_code=True)
File "/opt/anaconda3/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 626, in from_pretrained
tokenizer_class = get_class_from_dynamic_module(
File "/opt/anaconda3/lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 363, in get_class_from_dynamic_module
final_module = get_cached_module_file(
File "/opt/anaconda3/lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 261, in get_cached_module_file
commit_hash = model_info(pretrained_model_name_or_path, revision=revision, token=token).sha
File "/opt/anaconda3/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 94, in _inner_fn
return fn(*args, **kwargs)
File "/opt/anaconda3/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py", line 98, in inner_f
return f(*args, **kwargs)
File "/opt/anaconda3/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 1283, in model_info
r = requests.get(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/adapters.py", line 501, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
Traceback (most recent call last):
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1042, in validate_conn
conn.connect()
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connection.py", line 419, in connect
self.sock = ssl_wrap_socket(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/util/ssl.py", line 449, in ssl_wrap_socket
ssl_sock = ssl_wrap_socket_impl(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/util/ssl.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/opt/anaconda3/lib/python3.10/ssl.py", line 513, in wrap_socket
return self.sslsocket_class._create(
File "/opt/anaconda3/lib/python3.10/ssl.py", line 1071, in _create
self.do_handshake()
File "/opt/anaconda3/lib/python3.10/ssl.py", line 1342, in do_handshake
self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/util/retry.py", line 550, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/packages/six.py", line 769, in reraise
raise value.with_traceback(tb)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1042, in validate_conn
conn.connect()
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connection.py", line 419, in connect
self.sock = ssl_wrap_socket(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/util/ssl.py", line 449, in ssl_wrap_socket
ssl_sock = ssl_wrap_socket_impl(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/util/ssl.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/opt/anaconda3/lib/python3.10/ssl.py", line 513, in wrap_socket
return self.sslsocket_class._create(
File "/opt/anaconda3/lib/python3.10/ssl.py", line 1071, in _create
self.do_handshake()
File "/opt/anaconda3/lib/python3.10/ssl.py", line 1342, in do_handshake
self._sslobj.do_handshake()
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/zhangzhong/MOSS/finetune_moss.py", line 305, in
train(args)
File "/home/zhangzhong/MOSS/finetune_moss.py", line 177, in train
tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path, trust_remote_code=True)
File "/opt/anaconda3/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 626, in from_pretrained
tokenizer_class = get_class_from_dynamic_module(
File "/opt/anaconda3/lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 363, in get_class_from_dynamic_module
final_module = get_cached_module_file(
File "/opt/anaconda3/lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 261, in get_cached_module_file
commit_hash = model_info(pretrained_model_name_or_path, revision=revision, token=token).sha
File "/opt/anaconda3/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 94, in _inner_fn
return fn(*args, **kwargs)
File "/opt/anaconda3/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py", line 98, in inner_f
return f(*args, **kwargs)
File "/opt/anaconda3/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 1283, in model_info
r = requests.get(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/adapters.py", line 501, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
Traceback (most recent call last):
File "/home/zhangzhong/MOSS/finetune_moss.py", line 305, in
train(args)
File "/home/zhangzhong/MOSS/finetune_moss.py", line 177, in train
tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path, trust_remote_code=True)
File "/opt/anaconda3/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 641, in from_pretrained
return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
File "/opt/anaconda3/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1801, in from_pretrained
return cls._from_pretrained(
File "/opt/anaconda3/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1956, in _from_pretrained
tokenizer = cls(*init_inputs, **init_kwargs)
File "/home/zhangzhong/.cache/huggingface/modules/transformers_modules/fnlp/moss-moon-003-sft-int4/e3f0d7e7fba3944d5932ca2608b816678220ed25/tokenization_moss.py", line 173, in init
with open(vocab_file, encoding="utf-8") as vocab_handle:
TypeError: expected str, bytes or os.PathLike object, not NoneType
Traceback (most recent call last):
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1042, in validate_conn
conn.connect()
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connection.py", line 419, in connect
self.sock = ssl_wrap_socket(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/util/ssl.py", line 449, in ssl_wrap_socket
ssl_sock = ssl_wrap_socket_impl(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/util/ssl.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/opt/anaconda3/lib/python3.10/ssl.py", line 513, in wrap_socket
return self.sslsocket_class._create(
File "/opt/anaconda3/lib/python3.10/ssl.py", line 1071, in _create
self.do_handshake()
File "/opt/anaconda3/lib/python3.10/ssl.py", line 1342, in do_handshake
self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/util/retry.py", line 550, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/packages/six.py", line 769, in reraise
raise value.with_traceback(tb)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1042, in validate_conn
conn.connect()
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connection.py", line 419, in connect
self.sock = ssl_wrap_socket(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/util/ssl.py", line 449, in ssl_wrap_socket
ssl_sock = ssl_wrap_socket_impl(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/util/ssl.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/opt/anaconda3/lib/python3.10/ssl.py", line 513, in wrap_socket
return self.sslsocket_class._create(
File "/opt/anaconda3/lib/python3.10/ssl.py", line 1071, in _create
self.do_handshake()
File "/opt/anaconda3/lib/python3.10/ssl.py", line 1342, in do_handshake
self._sslobj.do_handshake()
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/zhangzhong/MOSS/finetune_moss.py", line 305, in
train(args)
File "/home/zhangzhong/MOSS/finetune_moss.py", line 177, in train
tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path, trust_remote_code=True)
File "/opt/anaconda3/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 641, in from_pretrained
return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
File "/opt/anaconda3/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1760, in from_pretrained
resolved_vocab_files[file_id] = cached_file(
File "/opt/anaconda3/lib/python3.10/site-packages/transformers/utils/hub.py", line 409, in cached_file
resolved_file = hf_hub_download(
File "/opt/anaconda3/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1226, in hf_hub_download
http_get(
File "/opt/anaconda3/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 470, in http_get
r = _request_wrapper(
File "/opt/anaconda3/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 433, in _request_wrapper
return http_backoff(
File "/opt/anaconda3/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 105, in http_backoff
response = requests.request(method=method, url=url, **kwargs)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/adapters.py", line 501, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
Traceback (most recent call last):
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
conn.connect()
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connection.py", line 461, in connect
cert = self.sock.getpeercert()
File "/opt/anaconda3/lib/python3.10/ssl.py", line 1154, in getpeercert
self._check_connected()
File "/opt/anaconda3/lib/python3.10/ssl.py", line 1119, in _check_connected
self.getpeername()
OSError: [Errno 107] Transport endpoint is not connected

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/util/retry.py", line 550, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/packages/six.py", line 769, in reraise
raise value.with_traceback(tb)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
conn.connect()
File "/home/zhangzhong/.local/lib/python3.10/site-packages/urllib3/connection.py", line 461, in connect
cert = self.sock.getpeercert()
File "/opt/anaconda3/lib/python3.10/ssl.py", line 1154, in getpeercert
self._check_connected()
File "/opt/anaconda3/lib/python3.10/ssl.py", line 1119, in _check_connected
self.getpeername()
urllib3.exceptions.ProtocolError: ('Connection aborted.', OSError(107, 'Transport endpoint is not connected'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/zhangzhong/MOSS/finetune_moss.py", line 305, in
train(args)
File "/home/zhangzhong/MOSS/finetune_moss.py", line 177, in train
tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path, trust_remote_code=True)
File "/opt/anaconda3/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 641, in from_pretrained
return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
File "/opt/anaconda3/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1760, in from_pretrained
resolved_vocab_files[file_id] = cached_file(
File "/opt/anaconda3/lib/python3.10/site-packages/transformers/utils/hub.py", line 409, in cached_file
resolved_file = hf_hub_download(
File "/opt/anaconda3/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1226, in hf_hub_download
http_get(
File "/opt/anaconda3/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 470, in http_get
r = _request_wrapper(
File "/opt/anaconda3/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 433, in _request_wrapper
return http_backoff(
File "/opt/anaconda3/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 105, in http_backoff
response = requests.request(method=method, url=url, **kwargs)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/requests/adapters.py", line 501, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', OSError(107, 'Transport endpoint is not connected'))
Downloading: 44%|██████████████████████████▊ | 1.10M/2.50M [00:01<00:00, 1.50MB/s]WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 2556277 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 1 (pid: 2556278) of binary: /opt/anaconda3/bin/python
Traceback (most recent call last):
File "/opt/anaconda3/bin/accelerate", line 8, in
sys.exit(main())
File "/opt/anaconda3/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 45, in main
args.func(args)
File "/opt/anaconda3/lib/python3.10/site-packages/accelerate/commands/launch.py", line 900, in launch_command
deepspeed_launcher(args)
File "/opt/anaconda3/lib/python3.10/site-packages/accelerate/commands/launch.py", line 643, in deepspeed_launcher
distrib_run.run(args)
File "/home/zhangzhong/.local/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/home/zhangzhong/.local/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/zhangzhong/.local/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

finetune_moss.py FAILED

Failures:
[1]:
time : 2023-05-26_09:56:29
host : s012.ai.ldap
rank : 2 (local_rank: 2)
exitcode : 1 (pid: 2556279)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
time : 2023-05-26_09:56:29
host : s012.ai.ldap
rank : 3 (local_rank: 3)
exitcode : 1 (pid: 2556280)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
time : 2023-05-26_09:56:29
host : s012.ai.ldap
rank : 4 (local_rank: 4)
exitcode : 1 (pid: 2556281)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[4]:
time : 2023-05-26_09:56:29
host : s012.ai.ldap
rank : 5 (local_rank: 5)
exitcode : 1 (pid: 2556282)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[5]:
time : 2023-05-26_09:56:29
host : s012.ai.ldap
rank : 6 (local_rank: 6)
exitcode : 1 (pid: 2556283)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[6]:
time : 2023-05-26_09:56:29
host : s012.ai.ldap
rank : 7 (local_rank: 7)
exitcode : 1 (pid: 2556284)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

Root Cause (first observed failure):
[0]:
time : 2023-05-26_09:56:29
host : s012.ai.ldap
rank : 1 (local_rank: 1)
exitcode : 1 (pid: 2556278)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

Read finetune_moss.py around train() line 177 and reproduce the AutoTokenizer.from_pretrained call for fnlp/moss-moon-003-sft-int4. Compare the distributed launch and model-loading logs; done means documenting a reproducible fix or clear diagnosis for the connection failure.