GoogleCloudPlatform / GoogleCloudPlatform/gsutil
rsync: Is there any way to speed up the "Building synchronization state" phase?
- Dominant language
- Python
- Stars
- 918
- Forks
- 335
- PR merge metrics
- No merged PRs in 30d
Description
## The Problem
gsutil version: 4.59
There is a much longer "Building synchronization state" phase compared to the actual file copies when using `gsutil rsync`.
This is particularly painful when the destination bucket has 100's of thousands of files and the source only has a few files to copy.
I have to wait up to 10 minutes sometimes for `gsutil` to stat 100's of thousands of files that will not be part of the operation at all.
## My Question
During the synchronization phase, is there some way to exclude all files from the destination that are not included in the source?
## Why don't I just use `gsutil cp`?
I can't use `gsutil cp` because it doesn't have a mechanism for comparing file mtime's to determine if the file should be copied or not.
My use case is that I have a fleet of client-facing servers behind a load balancer that collect info about the clients that connect to them. These files are saved with the client's name as the file name.
e.g.
```
/clients/client-aaa000.json
/clients/client-abb001.json
/clients/client-zzz123.json
```
I run `gsutil rsync` occasionally on all of the servers to send these client json files to a bucket.
Using the rsync command's ability to check mtime allows for not sending a file if some other server has already sent a newer copy (which is great!)
Any one server will only have ~100 JSON files on it at any time, but the GCS bucket has 500,000+ files in it. The rsync command is this:
```
gsutil -m rsync -u /clients gs://my-bucket/clients
```
So to send ~100 files, gsutil first has to stat 500,000+ files, 499,900 of which I don't care about (which is not great)
Contributor guide
Research direction
Start with the `gsutil -m rsync -u` entry point and reproduce the synchronization phase using a source with a few files and a destination containing many more. Trace how destination objects are discovered and compared, then define a measurable completion condition: unchanged files outside the source set should not be listed or statted while source files still receive mtime-based comparison.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python
- Domain
- cli, cloud, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100