GoogleCloudPlatform / GoogleCloudPlatform/gsutil
gsutil rsync throws encoding error while syncing .gz files from S3 with Python3
- Dominant language
- Python
- Stars
- 918
- Forks
- 335
- PR merge metrics
- No merged PRs in 30d
Description
**Problem Description**
I am running following command to sync AWS S3 bucket with GCS bucket
`gsutil rsync -r s3://source_bucket gs://target_bucket`
I have also tried with -J option because there are .gz files.
Here is the error -
```
Exception in thread Thread-4: B]
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/gslib/boto_translation.py", line 636, in _PerformSimpleDownload
hash_algs=hash_algs)
TypeError: get_contents_to_file() got an unexpected keyword argument 'hash_algs'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.7/site-packages/gslib/daisy_chain_wrapper.py", line 213, in PerformDownload
decryption_tuple=self.decryption_tuple)
File "/usr/local/lib/python3.7/site-packages/gslib/cloud_api_delegator.py", line 353, in GetObjectMedia
decryption_tuple=decryption_tuple)
File "/usr/local/lib/python3.7/site-packages/gslib/boto_translation.py", line 582, in GetObjectMedia
hash_algs=hash_algs)
File "/usr/local/lib/python3.7/site-packages/gslib/boto_translation.py", line 641, in _PerformSimpleDownload
headers=headers)
File "/usr/local/lib/python3.7/site-packages/gslib/vendored/boto/boto/s3/key.py", line 1670, in get_contents_to_file
response_headers=response_headers)
File "/usr/local/lib/python3.7/site-packages/gslib/vendored/boto/boto/s3/key.py", line 1502, in get_file
query_args=None)
File "/usr/local/lib/python3.7/site-packages/gslib/vendored/boto/boto/s3/key.py", line 1556, in _get_file_internal
print_to_fd(six.ensure_binary(key_bytes), file=fp, end=b'')
File "/usr/local/lib/python3.7/site-packages/gslib/vendored/boto/boto/utils.py", line 1206, in print_to_fd
write_to_fd(file, data)
File "/usr/local/lib/python3.7/site-packages/gslib/vendored/boto/boto/utils.py", line 1222, in write_to_fd
fd.write(six.ensure_text(data))
File "/usr/local/lib/python3.7/site-packages/gslib/vendored/boto/boto/vendored/six.py", line 901, in ensure_text
return s.decode(encoding, errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
```
System Specs -
```
gsutil version: 4.47
checksum: PACKAGED_GSUTIL_INSTALLS_DO_NOT_HAVE_CHECKSUMS (!= da2648451f3edb644090ab6c8d57d5f5)
boto version: 2.49.0
python version: 3.7.3 (default, Mar 27 2019, 09:23:15) [Clang 10.0.1 (clang-1001.0.46.3)]
OS: Darwin 18.6.0
multiprocessing available: True
using cloud sdk: False
pass cloud sdk credentials to gsutil: False
config path(s): /Users/username/.boto, /Users/username/.aws/credentials
gsutil path: /usr/local/bin/gsutil
compiled crcmod: True
installed via package manager: True
editable install: False
```
Contributor guide
Assessment
This issue has not been assessed yet.