GoogleCloudPlatform / GoogleCloudPlatform/gsutil
Broken pipe / 503 None
- Dominant language
- Python
- Stars
- 918
- Forks
- 335
- PR merge metrics
- No merged PRs in 30d
Description
I am syncing from gcloud to AWS and I am getting these:
```
Starting synchronization...
Copying gs://xxx/130580.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130239.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130259_original.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130579_original.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130579.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130259.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130291_original.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130357_original.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130356_original.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130270_original.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130580_original.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130271_original.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130290_original.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130272.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130290.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130260.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130581.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130578.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130357.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130271.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130358.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130578_original.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130581_original.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/4844.png [Content-Type=image/png]...
Copying gs://xxx/5019.png [Content-Type=image/png]...
Copying gs://xxx/5020.png [Content-Type=image/png]...
[Errno 32] Broken pipe
[Errno 32] Broken pipe
ServiceException: 503 None
[Errno 32] Broken pipe
[Errno 32] Broken pipe
Exception in thread Thread-105:
Traceback (most recent call last):
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/gcs_json_api.py", line 1227, in GetObjectMedia
decryption_tuple=decryption_tuple)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/gcs_json_api.py", line 1365, in _PerformDownload
use_chunks=False)
File "/usr/lib/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/transfer.py", line 541, in StreamMedia
additional_headers=additional_headers)
File "/usr/lib/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/transfer.py", line 420, in __GetChunk
retries=self.num_retries)
File "/usr/lib/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/http_wrapper.py", line 360, in MakeRequest
max_retry_wait, total_wait_sec))
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/utils/retry_util.py", line 84, in RetriesInDataTransferHandler
http_wrapper.RethrowExceptionHandler(retry_args)
File "/usr/lib/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/http_wrapper.py", line 350, in MakeRequest
check_response_func=check_response_func)
File "/usr/lib/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/http_wrapper.py", line 406, in _MakeRequestNoRetry
check_response_func(response)
File "/usr/lib/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/http_wrapper.py", line 223, in CheckResponse
raise exceptions.BadStatusCodeError.FromResponse(response)
apitools.base.py.exceptions.BadStatusCodeError: HttpError accessing : response: <{'x-guploader-uploadid': 'ADPycdvW9M8-GklsrkWyJZcXshwsui_LNL3SoVQvmPEGJr7x6p2sg2GmZEUa5-yc4gcTzsfae0JyDqD1av1z4SWQWgaw_pXnTQ', 'content-type': 'text/html; charset=UTF-8', 'date': 'Sat, 25 Sep 2021 20:57:14 GMT', 'vary': 'Origin, X-Origin', 'expires': 'Sat, 25 Sep 2021 20:57:14 GMT', 'cache-control': 'private, max-age=0', 'content-length': '24', 'server': 'UploadServer', 'status': '503'}>, content
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/daisy_chain_wrapper.py", line 218, in PerformDownload
decryption_tuple=self.decryption_tuple)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/cloud_api_delegator.py", line 366, in GetObjectMedia
decryption_tuple=decryption_tuple)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/gcs_json_api.py", line 1238, in GetObjectMedia
generation=generation)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/gcs_json_api.py", line 2255, in _TranslateExceptionAndRaise
raise translated_exception
gslib.cloud_api.ServiceException: ServiceException: 503 None
[Errno 32] Broken pipe
[Errno 32] Broken pipe
[Errno 32] Broken pipe
[Errno 32] Broken pipe
[Errno 32] Broken pipe
[Errno 32] Broken pipe
[Errno 32] Broken pipe
[Errno 32] Broken pipe
[Errno 32] Broken pipe
[Errno 32] Broken pipe
CommandException: 15 files/objects could not be copied/removed.
```
This happens randomly, but often. I gues there should be some kind of retries?
```
gsutil/4.68
```
Contributor guide
Research direction
Reproduce the intermittent sync failure with gsutil 4.68, then trace the reported path through gslib/daisy_chain_wrapper.py, gslib/gcs_json_api.py, and the retry handling in gslib/utils/retry_util.py and apitools/base/py/http_wrapper.py. Done means transient 503 and broken-pipe errors are handled during synchronization without leaving the reported files uncopied.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, google-cloud, python
- Domain
- cli, cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100