GoogleCloudPlatform / GoogleCloudPlatform/gsutil

MD5 mismatch issue with both `rsync` and `cp` object files encrypted with `SSE-KMS` from S3

Open
#1,249 7 comments 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

Hi,

I encountered an MD5 mismatch issue with both `rsync` and `cp` object files encrypted with `SSE-KMS` from S3. I believe that `rsync` and `cp` were using cloud supplied MD5, which stored in `ETag` instead of generating its MD5 from the content itself. According to the [doc](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html), "objects encrypted by SSE-C or SSE-KMS have ETags that are not an MD5 digest of their object data."

I also verified by `gsutil hash` on the same files living on my local machine. Their MD5 checksums were different on my local machine and S3 (etag).

```bash
$ gsutil version
gsutil version: 4.61
```

Contributor guide

Open the contributing guide

Research direction

The issue identifies gsutil 4.61 and a checksum mismatch for S3 objects encrypted with SSE-KMS, but names no repository file or test. Start by reproducing the case and comparing the local content MD5 with the S3 ETag using gsutil hash. Done should establish and verify the correct checksum behavior for these objects.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cli, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.