GoogleCloudPlatform / GoogleCloudPlatform/gsutil
Invalid checksum when copying from an encrypted Amazon S3 bucket
- Dominant language
- Python
- Stars
- 918
- Forks
- 335
- PR merge metrics
- No merged PRs in 30d
Description
Creating on behalf of @animalillo who is unable to access his account currently:
> When attempting to copy objects that are encrypted on Amazon S3 the gsutil command returns invalid checksum. It doesn't matter if the destination is local or another remote.
>
> ```
> gsutil -o s3:host=s3-eu-west-1.amazonaws.com cp s3://bucket/test/export_info_test.json /tmp/t/
> Copying s3://bucket/test/export_info_test.json...
> Traceback (most recent call last):
> File "/usr/lib/google-cloud-sdk/platform/gsutil/gsutil", line 21, in
> gsutil.RunMain()
> File "/usr/lib/google-cloud-sdk/platform/gsutil/gsutil.py", line 151, in RunMain
> sys.exit(gslib.__main__.main())
> File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 435, in main
> return _RunNamedCommandAndHandleExceptions(
> File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 783, in _RunNamedCommandAndHandleExceptions
> _HandleUnknownFailure(e)
> File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 632, in _RunNamedCommandAndHandleExceptions
> return command_runner.RunNamedCommand(command_name,
> File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/command_runner.py", line 421, in RunNamedCommand
> return_code = command_inst.RunCommand()
> File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/commands/cp.py", line 1141, in RunCommand
> self.Apply(_CopyFuncWrapper,
> File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/command.py", line 1570, in Apply
> self._SequentialApply(func, args_iterator, exception_handler, caller_id,
> File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/command.py", line 1649, in _SequentialApply
> worker_thread.PerformTask(task, self)
> File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/command.py", line 2399, in PerformTask
> results = task.func(cls, task.args, thread_state=self.thread_gsutil_api)
> File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/commands/cp.py", line 697, in _CopyFuncWrapper
> cls.CopyFunc(args,
> File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/commands/cp.py", line 931, in CopyFunc
> _, bytes_transferred, result_url, md5 = copy_helper.PerformCopy(
> File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/utils/copy_helper.py", line 3889, in PerformCopy
> return _DownloadObjectToFile(src_url,
> File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/utils/copy_helper.py", line 3098, in _DownloadObjectToFile
> local_md5 = _ValidateAndCompleteDownload(logger,
> File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/utils/copy_helper.py", line 3271, in _ValidateAndCompleteDownload
> _CheckHashes(logger, src_url, src_obj_metadata, final_file_name,
> File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/utils/copy_helper.py", line 925, in _CheckHashes
> raise HashMismatchException(
> gslib.exception.HashMismatchException: md5 signature computed for local file (b'zKHbWL9hhD0dAsFOzySgKA==') doesn't match cloud-supplied digest (b'd8P9lv1YZSf0jIbWVcYnvQ=='). Local file (/tmp/t/export_info_test.json) will be deleted.
> ```
Contributor guide
Research direction
Reproduce the encrypted S3 copy with the command in the report, then trace checksum handling from gslib/commands/cp.py through gslib/utils/copy_helper.py, especially _DownloadObjectToFile, _ValidateAndCompleteDownload, and _CheckHashes. Done means copying the encrypted object to a local file or another remote no longer raises HashMismatchException; inspect existing checksum-related tests before adding coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cli, cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 38/100