GoogleCloudPlatform / GoogleCloudPlatform/gsutil
rsync fails when copying to KMS-enabled S3 buckets
- Dominant language
- Python
- Stars
- 918
- Forks
- 335
- PR merge metrics
- No merged PRs in 30d
Description
If I run the following command, where the S3 bucket has KMS encrpytion enabled:
```
gsutil rsync -r gs://my_bucket/my_data s3://some_bucket/some_data
```
I get the following error:
```
Traceback (most recent call last):
File "./gsutil", line 21, in
gsutil.RunMain()
File "/Users/desbo/code/gsutil/gsutil/gsutil.py", line 122, in RunMain
sys.exit(gslib.__main__.main())
File "/Users/desbo/code/gsutil/gsutil/gslib/__main__.py", line 444, in main
user_project=user_project)
File "/Users/desbo/code/gsutil/gsutil/gslib/__main__.py", line 780, in _RunNamedCommandAndHandleExceptions
_HandleUnknownFailure(e)
File "/Users/desbo/code/gsutil/gsutil/gslib/__main__.py", line 639, in _RunNamedCommandAndHandleExceptions
user_project=user_project)
File "/Users/desbo/code/gsutil/gsutil/gslib/command_runner.py", line 410, in RunNamedCommand
return_code = command_inst.RunCommand()
File "/Users/desbo/code/gsutil/gsutil/gslib/commands/rsync.py", line 1691, in RunCommand
seek_ahead_iterator=seek_ahead_iterator)
File "/Users/desbo/code/gsutil/gsutil/gslib/command.py", line 1529, in Apply
arg_checker, should_return_results, fail_on_error)
File "/Users/desbo/code/gsutil/gsutil/gslib/command.py", line 1602, in _SequentialApply
worker_thread.PerformTask(task, self)
File "/Users/desbo/code/gsutil/gsutil/gslib/command.py", line 2324, in PerformTask
results = task.func(cls, task.args, thread_state=self.thread_gsutil_api)
File "/Users/desbo/code/gsutil/gsutil/gslib/commands/rsync.py", line 1466, in _RsyncFunc
preserve_posix=cls.preserve_posix_attrs)
File "/Users/desbo/code/gsutil/gsutil/gslib/utils/copy_helper.py", line 3864, in PerformCopy
decryption_key=decryption_key)
File "/Users/desbo/code/gsutil/gsutil/gslib/utils/copy_helper.py", line 3385, in _CopyObjToObjDaisyChainMode
uploaded_object)
File "/Users/desbo/code/gsutil/gsutil/gslib/utils/copy_helper.py", line 824, in _CheckCloudHashes
(alg, download_b64_digest, upload_b64_digest, dst_url))
gslib.exception.HashMismatchException: md5 signature for source object (rbCzhcFNJPKCRGjPqZoChg==) doesn't match destination object digest (333vz/L4pkDe8iO7bYT26g==). Object (s3://sd-tmp-kms-2/tmpr5/data-000000000000.snappy.parquet) will be deleted.
```
I noticed https://github.com/GoogleCloudPlatform/gsutil/pull/1258 which seems to introduce a fix for the (intented) mismatch between ETags and MD5 hashes when KMS is enabled, but it seems the code in `copy_helper.py` doesn't include a similar bypass.
Following [the comment in `copy_helper.py`](https://github.com/GoogleCloudPlatform/gsutil/blob/master/gslib/utils/copy_helper.py#L787-L791), if I set my boto config to `check_hashes = never`, the above command works. But this suggest the fix applied in #1258 is redundant and ideally I wouldn't have to set this globally.
gsutil version: 4.67
Contributor guide
Research direction
Read gslib/utils/copy_helper.py around _CheckCloudHashes and the check_hashes comment near lines 787-791, then trace the call from _CopyObjToObjDaisyChainMode and rsync.py. Compare the behavior with pull request #1258 and reproduce the reported gsutil rsync command against a KMS-enabled S3 bucket. Done means the command handles the digest mismatch without requiring a global check_hashes setting.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cli, cloud, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100