GoogleCloudPlatform / GoogleCloudPlatform/gsutil
gsutil rsync -c option doesn't work as expected
- Dominant language
- Python
- Stars
- 918
- Forks
- 335
- PR merge metrics
- No merged PRs in 30d
Description
The documentation on https://cloud.google.com/storage/docs/gsutil/commands/rsync#copying-in-the-cloud-and-metadata-preservation states:
`-c`
>Causes the rsync command to compute and compare checksums (instead of comparing mtime) for files if the size of source and destination match. This option increases local disk I/O and run time if either src_url or dst_url are on the local file system.
---
However, when I run the rsync command with the -c option, rsync copies each and every file, where I expect it to copy 0 files as I am recopying the exact same files and none of the checksums have changed.
**rsync command with -c option**
```
gsutil -m -h "Cache-Control:public, max-age=7776000" rsync -r -c /Users/username/project/static-assets gs://static-assets-bucket
```
When running above command, every single file is copied.
I checked the hash and size of both the local file and the file in cloud storage via `gsutil hash` and `gsutil ls -L`/`ls -al` respectively. Unless I understand the `-c` option incorrectly, I thing this is not working as expected.
Here's sample command and output:
**Same hash on source and destination (crc32c and md5)**
```
$ gsutil hash dist/static/themes/weights/heavy.b02374c0d6c03b4475d8a0f1de359723.scss
Hashes [base64] for dist/static/themes/weights/heavy.b02374c0d6c03b4475d8a0f1de359723.scss:
Hash (crc32c): 74FOIA==
Hash (md5): sCN0wNbAO0R12KDx3jWXIw==
```
```
$ gsutil hash gs://static-assets-bucket/themes/weights/heavy.b02374c0d6c03b4475d8a0f1de359723.scss
Hashes [base64] for themes/weights/heavy.b02374c0d6c03b4475d8a0f1de359723.scss:
Hash (crc32c): 74FOIA==
Hash (md5): sCN0wNbAO0R12KDx3jWXIw==
```
**Same file size on source and destination (399 bytes)**
```
$ gsutil ls -L gs://static-assets-bucket/themes/weights/heavy.b02374c0d6c03b4475d8a0f1de359723.scss
gs://static-assets-bucket/themes/weights/heavy.b02374c0d6c03b4475d8a0f1de359723.scss:
Creation time: Wed, 26 Feb 2020 17:23:09 GMT
Update time: Wed, 26 Feb 2020 17:23:09 GMT
Storage class: REGIONAL
Cache-Control: public, max-age=7776000
Content-Length: 399
Content-Type: application/octet-stream
Metadata:
goog-reserved-file-mtime:1582737717
Hash (crc32c): 74FOIA==
Hash (md5): sCN0wNbAO0R12KDx3jWXIw==
ETag: CLLBwd3d7+cCEAE=
Generation: 1582737789051058
Metageneration: 1
```
```
$ ls -al static-assets/themes/weights/heavy.b02374c0d6c03b4475d8a0f1de359723.scss
-rw-r--r-- 1 govindrai staff 399 Aug 19 08:28 static-assets/themes/weights/heavy.b02374c0d6c03b4475d8a0f1de359723.scss
```
I am using the latest gsutil version as of today,
```
gsutil version: 4.52
```
Contributor guide
Research direction
Start with the documented gsutil rsync -c behavior and reproduce the supplied command against the matching local and gs:// files. Trace the rsync entry point and compare the reported sizes and hashes; done means determining why identical files are copied instead of skipped.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python
- Domain
- cli, cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100