GoogleCloudPlatform / GoogleCloudPlatform/gsutil
MD5 mismatch issue with both `rsync` and `cp` object files encrypted with `SSE-KMS` from S3
- Dominant language
- Python
- Stars
- 918
- Forks
- 335
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
I encountered an MD5 mismatch issue with both `rsync` and `cp` object files encrypted with `SSE-KMS` from S3. I believe that `rsync` and `cp` were using cloud supplied MD5, which stored in `ETag` instead of generating its MD5 from the content itself. According to the [doc](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html), "objects encrypted by SSE-C or SSE-KMS have ETags that are not an MD5 digest of their object data."
I also verified by `gsutil hash` on the same files living on my local machine. Their MD5 checksums were different on my local machine and S3 (etag).
```bash
$ gsutil version
gsutil version: 4.61
```
Contributor guide
Research direction
The issue identifies gsutil 4.61 and a checksum mismatch for S3 objects encrypted with SSE-KMS, but names no repository file or test. Start by reproducing the case and comparing the local content MD5 with the S3 ETag using gsutil hash. Done should establish and verify the correct checksum behavior for these objects.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cli, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100