GoogleCloudPlatform / GoogleCloudPlatform/gsutil

Broken pipe / 503 None

Open
#1,348 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

I am syncing from gcloud to AWS and I am getting these:

```
Starting synchronization...
Copying gs://xxx/130580.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130239.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130259_original.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130579_original.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130579.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130259.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130291_original.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130357_original.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130356_original.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130270_original.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130580_original.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130271_original.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130290_original.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130272.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130290.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130260.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130581.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130578.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130357.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130271.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130358.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130578_original.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/130581_original.jpg [Content-Type=image/jpeg]...
Copying gs://xxx/4844.png [Content-Type=image/png]...
Copying gs://xxx/5019.png [Content-Type=image/png]...
Copying gs://xxx/5020.png [Content-Type=image/png]...
[Errno 32] Broken pipe
[Errno 32] Broken pipe
ServiceException: 503 None
[Errno 32] Broken pipe
[Errno 32] Broken pipe
Exception in thread Thread-105:
Traceback (most recent call last):
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/gcs_json_api.py", line 1227, in GetObjectMedia
decryption_tuple=decryption_tuple)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/gcs_json_api.py", line 1365, in _PerformDownload
use_chunks=False)
File "/usr/lib/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/transfer.py", line 541, in StreamMedia
additional_headers=additional_headers)
File "/usr/lib/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/transfer.py", line 420, in __GetChunk
retries=self.num_retries)
File "/usr/lib/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/http_wrapper.py", line 360, in MakeRequest
max_retry_wait, total_wait_sec))
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/utils/retry_util.py", line 84, in RetriesInDataTransferHandler
http_wrapper.RethrowExceptionHandler(retry_args)
File "/usr/lib/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/http_wrapper.py", line 350, in MakeRequest
check_response_func=check_response_func)
File "/usr/lib/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/http_wrapper.py", line 406, in _MakeRequestNoRetry
check_response_func(response)
File "/usr/lib/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/http_wrapper.py", line 223, in CheckResponse
raise exceptions.BadStatusCodeError.FromResponse(response)
apitools.base.py.exceptions.BadStatusCodeError: HttpError accessing : response: <{'x-guploader-uploadid': 'ADPycdvW9M8-GklsrkWyJZcXshwsui_LNL3SoVQvmPEGJr7x6p2sg2GmZEUa5-yc4gcTzsfae0JyDqD1av1z4SWQWgaw_pXnTQ', 'content-type': 'text/html; charset=UTF-8', 'date': 'Sat, 25 Sep 2021 20:57:14 GMT', 'vary': 'Origin, X-Origin', 'expires': 'Sat, 25 Sep 2021 20:57:14 GMT', 'cache-control': 'private, max-age=0', 'content-length': '24', 'server': 'UploadServer', 'status': '503'}>, content

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/usr/lib/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/daisy_chain_wrapper.py", line 218, in PerformDownload
decryption_tuple=self.decryption_tuple)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/cloud_api_delegator.py", line 366, in GetObjectMedia
decryption_tuple=decryption_tuple)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/gcs_json_api.py", line 1238, in GetObjectMedia
generation=generation)
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/gcs_json_api.py", line 2255, in _TranslateExceptionAndRaise
raise translated_exception
gslib.cloud_api.ServiceException: ServiceException: 503 None

[Errno 32] Broken pipe
[Errno 32] Broken pipe
[Errno 32] Broken pipe
[Errno 32] Broken pipe
[Errno 32] Broken pipe
[Errno 32] Broken pipe
[Errno 32] Broken pipe
[Errno 32] Broken pipe
[Errno 32] Broken pipe
[Errno 32] Broken pipe
CommandException: 15 files/objects could not be copied/removed.
```

This happens randomly, but often. I gues there should be some kind of retries?

```
gsutil/4.68
```

Contributor guide

Open the contributing guide

Research direction

Reproduce the intermittent sync failure with gsutil 4.68, then trace the reported path through gslib/daisy_chain_wrapper.py, gslib/gcs_json_api.py, and the retry handling in gslib/utils/retry_util.py and apitools/base/py/http_wrapper.py. Done means transient 503 and broken-pipe errors are handled during synchronization without leaving the reported files uncopied.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, google-cloud, python
Domain
cli, cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.