GoogleCloudPlatform / GoogleCloudPlatform/gsutil

rsync: Is there any way to speed up the "Building synchronization state" phase?

Open
#1,206 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

## The Problem
gsutil version: 4.59

There is a much longer "Building synchronization state" phase compared to the actual file copies when using `gsutil rsync`.

This is particularly painful when the destination bucket has 100's of thousands of files and the source only has a few files to copy.
I have to wait up to 10 minutes sometimes for `gsutil` to stat 100's of thousands of files that will not be part of the operation at all.

## My Question
During the synchronization phase, is there some way to exclude all files from the destination that are not included in the source?

## Why don't I just use `gsutil cp`?

I can't use `gsutil cp` because it doesn't have a mechanism for comparing file mtime's to determine if the file should be copied or not.

My use case is that I have a fleet of client-facing servers behind a load balancer that collect info about the clients that connect to them. These files are saved with the client's name as the file name.
e.g.
```
/clients/client-aaa000.json
/clients/client-abb001.json
/clients/client-zzz123.json
```
I run `gsutil rsync` occasionally on all of the servers to send these client json files to a bucket.
Using the rsync command's ability to check mtime allows for not sending a file if some other server has already sent a newer copy (which is great!)

Any one server will only have ~100 JSON files on it at any time, but the GCS bucket has 500,000+ files in it. The rsync command is this:
```
gsutil -m rsync -u /clients gs://my-bucket/clients
```
So to send ~100 files, gsutil first has to stat 500,000+ files, 499,900 of which I don't care about (which is not great)

Contributor guide

Open the contributing guide

Research direction

Start with the `gsutil -m rsync -u` entry point and reproduce the synchronization phase using a source with a few files and a destination containing many more. Trace how destination objects are discovered and compared, then define a measurable completion condition: unchanged files outside the source set should not be listed or statted while source files still receive mtime-based comparison.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
cli, cloud, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.