GoogleCloudPlatform / GoogleCloudPlatform/gsutil

gsutil rsync -c alternative for if mtime is changed, then check hash (repeated file upload issue.)

Open
#1,675 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

Similar to the `-c` option, a parameter should be added that calculates CRC/MD5 hash in case of differences in `mtime`, and the file is only moved if there is a difference.

The situation is not clear, but identical files are being synchronized from a NAS server - I assume because there has been a change in the `mtime` value.

I currently use this:
```bash
gsutil -m rsync -r -d "/sourcedir" gs://destination-bucket/
```

And i see this in google cloud console web interface:

![same-file-sync-gs-console](https://user-images.githubusercontent.com/9432032/220575958-1e764aa8-6ff0-4844-876c-76afc46485c3.png)

This situation is not constant, but there are files that have been uploaded multiple times, even though it is the same file being synchronized upwards.

#### Expected

Adding a new parameter that performs a content hash/crc check when the size is the same, but the mtime value is different.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the gsutil rsync behavior and the existing -c option, then reproduce the issue with the provided gsutil -m rsync command against a NAS source. Done means a new parameter checks content hashes when file size matches but mtime differs, and avoids re-uploading identical files.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, python
Domain
cli, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.