GoogleCloudPlatform / GoogleCloudPlatform/gsutil
gsutil rsync -c alternative for if mtime is changed, then check hash (repeated file upload issue.)
- Dominant language
- Python
- Stars
- 918
- Forks
- 335
- PR merge metrics
- No merged PRs in 30d
Description
Similar to the `-c` option, a parameter should be added that calculates CRC/MD5 hash in case of differences in `mtime`, and the file is only moved if there is a difference.
The situation is not clear, but identical files are being synchronized from a NAS server - I assume because there has been a change in the `mtime` value.
I currently use this:
```bash
gsutil -m rsync -r -d "/sourcedir" gs://destination-bucket/
```
And i see this in google cloud console web interface:

This situation is not constant, but there are files that have been uploaded multiple times, even though it is the same file being synchronized upwards.
#### Expected
Adding a new parameter that performs a content hash/crc check when the size is the same, but the mtime value is different.
Contributor guide
Research direction
Start by reviewing the gsutil rsync behavior and the existing -c option, then reproduce the issue with the provided gsutil -m rsync command against a NAS source. Done means a new parameter checks content hashes when file size matches but mtime differs, and avoids re-uploading identical files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- gcp, python
- Domain
- cli, cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100