Metadata downloading fails and no way to resume the download

I'm using Sagemaker to download the xlarge data, but for some reason, the download fails half way and there is no way to resume the download where it was left. Any idea how to overcome this?

File "/home/ec2-user/SageMaker/datacomp/download_upstream.py", line 181, in █████████| 131M/131M [00:03<00:00, 35.1MB/s]
snapshot_download(**hf_snapshot_args)
File "/home/ec2-user/anaconda3/envs/datacomp/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 120, in _inner_fn
return fn(*args, **kwargs)
File "/home/ec2-user/anaconda3/envs/datacomp/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 239, in snapshot_download
thread_map(
File "/home/ec2-user/anaconda3/envs/datacomp/lib/python3.10/site-packages/tqdm/contrib/concurrent.py", line 94, in thread_map
return _executor_map(ThreadPoolExecutor, fn, *iterables, **tqdm_kwargs)
File "/home/ec2-user/anaconda3/envs/datacomp/lib/python3.10/site-packages/tqdm/contrib/concurrent.py", line 76, in _executor_map
return list(tqdm_class(ex.map(fn, *iterables, **map_args), **kwargs))
File "/home/ec2-user/anaconda3/envs/datacomp/lib/python3.10/site-packages/tqdm/std.py", line 1195, in iter
for obj in iterable:
File "/home/ec2-user/anaconda3/envs/datacomp/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
yield _result_or_cancel(fs.pop())
File "/home/ec2-user/anaconda3/envs/datacomp/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
return fut.result(timeout)
File "/home/ec2-user/anaconda3/envs/datacomp/lib/python3.10/concurrent/futures/_base.py", line 458, in result
return self.__get_result()
File "/home/ec2-user/anaconda3/envs/datacomp/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/home/ec2-user/anaconda3/envs/datacomp/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/ec2-user/anaconda3/envs/datacomp/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 215, in _inner_hf_hub_download
return hf_hub_download(
File "/home/ec2-user/anaconda3/envs/datacomp/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 120, in _inner_fn
return fn(*args, **kwargs)
File "/home/ec2-user/anaconda3/envs/datacomp/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1291, in hf_hub_download
raise LocalEntryNotFoundError(
huggingface_hub.utils._errors.LocalEntryNotFoundError: Connection error, and we cannot find the requested files in the disk cache. Please try again or make sure your Internet connection is on.

Start at download_upstream.py line 181 and inspect the snapshot_download call and its Hugging Face arguments. Reproduce the interrupted xlarge-data download, then verify that rerunning it preserves completed files and resumes after a connection failure. Done means the metadata download can continue without restarting from the beginning.