GoogleCloudPlatform / GoogleCloudPlatform/gsutil

gsutil rsync -c option doesn't work as expected

Open
#1,093 3 comments 3 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

The documentation on https://cloud.google.com/storage/docs/gsutil/commands/rsync#copying-in-the-cloud-and-metadata-preservation states:

`-c`
>Causes the rsync command to compute and compare checksums (instead of comparing mtime) for files if the size of source and destination match. This option increases local disk I/O and run time if either src_url or dst_url are on the local file system.

---

However, when I run the rsync command with the -c option, rsync copies each and every file, where I expect it to copy 0 files as I am recopying the exact same files and none of the checksums have changed.

**rsync command with -c option**

```
gsutil -m -h "Cache-Control:public, max-age=7776000" rsync -r -c /Users/username/project/static-assets gs://static-assets-bucket
```

When running above command, every single file is copied.

I checked the hash and size of both the local file and the file in cloud storage via `gsutil hash` and `gsutil ls -L`/`ls -al` respectively. Unless I understand the `-c` option incorrectly, I thing this is not working as expected.

Here's sample command and output:

**Same hash on source and destination (crc32c and md5)**

```
$ gsutil hash dist/static/themes/weights/heavy.b02374c0d6c03b4475d8a0f1de359723.scss
Hashes [base64] for dist/static/themes/weights/heavy.b02374c0d6c03b4475d8a0f1de359723.scss:
Hash (crc32c): 74FOIA==
Hash (md5): sCN0wNbAO0R12KDx3jWXIw==
```

```
$ gsutil hash gs://static-assets-bucket/themes/weights/heavy.b02374c0d6c03b4475d8a0f1de359723.scss
Hashes [base64] for themes/weights/heavy.b02374c0d6c03b4475d8a0f1de359723.scss:
Hash (crc32c): 74FOIA==
Hash (md5): sCN0wNbAO0R12KDx3jWXIw==
```

**Same file size on source and destination (399 bytes)**

```
$ gsutil ls -L gs://static-assets-bucket/themes/weights/heavy.b02374c0d6c03b4475d8a0f1de359723.scss
gs://static-assets-bucket/themes/weights/heavy.b02374c0d6c03b4475d8a0f1de359723.scss:
Creation time: Wed, 26 Feb 2020 17:23:09 GMT
Update time: Wed, 26 Feb 2020 17:23:09 GMT
Storage class: REGIONAL
Cache-Control: public, max-age=7776000
Content-Length: 399
Content-Type: application/octet-stream
Metadata:
goog-reserved-file-mtime:1582737717
Hash (crc32c): 74FOIA==
Hash (md5): sCN0wNbAO0R12KDx3jWXIw==
ETag: CLLBwd3d7+cCEAE=
Generation: 1582737789051058
Metageneration: 1
```

```
$ ls -al static-assets/themes/weights/heavy.b02374c0d6c03b4475d8a0f1de359723.scss
-rw-r--r-- 1 govindrai staff 399 Aug 19 08:28 static-assets/themes/weights/heavy.b02374c0d6c03b4475d8a0f1de359723.scss
```

I am using the latest gsutil version as of today,
```
gsutil version: 4.52
```

Contributor guide

Open the contributing guide

Research direction

Start with the documented gsutil rsync -c behavior and reproduce the supplied command against the matching local and gs:// files. Trace the rsync entry point and compare the reported sizes and hashes; done means determining why identical files are copied instead of skipped.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
cli, cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.