GoogleCloudPlatform / GoogleCloudPlatform/gsutil

rsync fails when copying to KMS-enabled S3 buckets

Open
#1,323 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

If I run the following command, where the S3 bucket has KMS encrpytion enabled:

```
gsutil rsync -r gs://my_bucket/my_data s3://some_bucket/some_data
```

I get the following error:

```
Traceback (most recent call last):
File "./gsutil", line 21, in
gsutil.RunMain()
File "/Users/desbo/code/gsutil/gsutil/gsutil.py", line 122, in RunMain
sys.exit(gslib.__main__.main())
File "/Users/desbo/code/gsutil/gsutil/gslib/__main__.py", line 444, in main
user_project=user_project)
File "/Users/desbo/code/gsutil/gsutil/gslib/__main__.py", line 780, in _RunNamedCommandAndHandleExceptions
_HandleUnknownFailure(e)
File "/Users/desbo/code/gsutil/gsutil/gslib/__main__.py", line 639, in _RunNamedCommandAndHandleExceptions
user_project=user_project)
File "/Users/desbo/code/gsutil/gsutil/gslib/command_runner.py", line 410, in RunNamedCommand
return_code = command_inst.RunCommand()
File "/Users/desbo/code/gsutil/gsutil/gslib/commands/rsync.py", line 1691, in RunCommand
seek_ahead_iterator=seek_ahead_iterator)
File "/Users/desbo/code/gsutil/gsutil/gslib/command.py", line 1529, in Apply
arg_checker, should_return_results, fail_on_error)
File "/Users/desbo/code/gsutil/gsutil/gslib/command.py", line 1602, in _SequentialApply
worker_thread.PerformTask(task, self)
File "/Users/desbo/code/gsutil/gsutil/gslib/command.py", line 2324, in PerformTask
results = task.func(cls, task.args, thread_state=self.thread_gsutil_api)
File "/Users/desbo/code/gsutil/gsutil/gslib/commands/rsync.py", line 1466, in _RsyncFunc
preserve_posix=cls.preserve_posix_attrs)
File "/Users/desbo/code/gsutil/gsutil/gslib/utils/copy_helper.py", line 3864, in PerformCopy
decryption_key=decryption_key)
File "/Users/desbo/code/gsutil/gsutil/gslib/utils/copy_helper.py", line 3385, in _CopyObjToObjDaisyChainMode
uploaded_object)
File "/Users/desbo/code/gsutil/gsutil/gslib/utils/copy_helper.py", line 824, in _CheckCloudHashes
(alg, download_b64_digest, upload_b64_digest, dst_url))
gslib.exception.HashMismatchException: md5 signature for source object (rbCzhcFNJPKCRGjPqZoChg==) doesn't match destination object digest (333vz/L4pkDe8iO7bYT26g==). Object (s3://sd-tmp-kms-2/tmpr5/data-000000000000.snappy.parquet) will be deleted.
```

I noticed https://github.com/GoogleCloudPlatform/gsutil/pull/1258 which seems to introduce a fix for the (intented) mismatch between ETags and MD5 hashes when KMS is enabled, but it seems the code in `copy_helper.py` doesn't include a similar bypass.

Following [the comment in `copy_helper.py`](https://github.com/GoogleCloudPlatform/gsutil/blob/master/gslib/utils/copy_helper.py#L787-L791), if I set my boto config to `check_hashes = never`, the above command works. But this suggest the fix applied in #1258 is redundant and ideally I wouldn't have to set this globally.

gsutil version: 4.67

Contributor guide

Open the contributing guide

Research direction

Read gslib/utils/copy_helper.py around _CheckCloudHashes and the check_hashes comment near lines 787-791, then trace the call from _CopyObjToObjDaisyChainMode and rsync.py. Compare the behavior with pull request #1258 and reproduce the reported gsutil rsync command against a KMS-enabled S3 bucket. Done means the command handles the digest mismatch without requiring a global check_hashes setting.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cli, cloud, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.