GoogleCloudPlatform / GoogleCloudPlatform/gsutil

Invalid checksum when copying from an encrypted Amazon S3 bucket

Open
#1,647 1 comment 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

Creating on behalf of @animalillo who is unable to access his account currently:

> When attempting to copy objects that are encrypted on Amazon S3 the gsutil command returns invalid checksum. It doesn't matter if the destination is local or another remote.
>
> ```
> gsutil -o s3:host=s3-eu-west-1.amazonaws.com cp s3://bucket/test/export_info_test.json /tmp/t/
> Copying s3://bucket/test/export_info_test.json...
> Traceback (most recent call last):
> File "/usr/lib/google-cloud-sdk/platform/gsutil/gsutil", line 21, in
> gsutil.RunMain()
> File "/usr/lib/google-cloud-sdk/platform/gsutil/gsutil.py", line 151, in RunMain
> sys.exit(gslib.__main__.main())
> File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 435, in main
> return _RunNamedCommandAndHandleExceptions(
> File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 783, in _RunNamedCommandAndHandleExceptions
> _HandleUnknownFailure(e)
> File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 632, in _RunNamedCommandAndHandleExceptions
> return command_runner.RunNamedCommand(command_name,
> File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/command_runner.py", line 421, in RunNamedCommand
> return_code = command_inst.RunCommand()
> File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/commands/cp.py", line 1141, in RunCommand
> self.Apply(_CopyFuncWrapper,
> File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/command.py", line 1570, in Apply
> self._SequentialApply(func, args_iterator, exception_handler, caller_id,
> File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/command.py", line 1649, in _SequentialApply
> worker_thread.PerformTask(task, self)
> File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/command.py", line 2399, in PerformTask
> results = task.func(cls, task.args, thread_state=self.thread_gsutil_api)
> File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/commands/cp.py", line 697, in _CopyFuncWrapper
> cls.CopyFunc(args,
> File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/commands/cp.py", line 931, in CopyFunc
> _, bytes_transferred, result_url, md5 = copy_helper.PerformCopy(
> File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/utils/copy_helper.py", line 3889, in PerformCopy
> return _DownloadObjectToFile(src_url,
> File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/utils/copy_helper.py", line 3098, in _DownloadObjectToFile
> local_md5 = _ValidateAndCompleteDownload(logger,
> File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/utils/copy_helper.py", line 3271, in _ValidateAndCompleteDownload
> _CheckHashes(logger, src_url, src_obj_metadata, final_file_name,
> File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/utils/copy_helper.py", line 925, in _CheckHashes
> raise HashMismatchException(
> gslib.exception.HashMismatchException: md5 signature computed for local file (b'zKHbWL9hhD0dAsFOzySgKA==') doesn't match cloud-supplied digest (b'd8P9lv1YZSf0jIbWVcYnvQ=='). Local file (/tmp/t/export_info_test.json) will be deleted.
> ```

Contributor guide

Open the contributing guide

Research direction

Reproduce the encrypted S3 copy with the command in the report, then trace checksum handling from gslib/commands/cp.py through gslib/utils/copy_helper.py, especially _DownloadObjectToFile, _ValidateAndCompleteDownload, and _CheckHashes. Done means copying the encrypted object to a local file or another remote no longer raises HashMismatchException; inspect existing checksum-related tests before adding coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cli, cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.