GoogleCloudPlatform / GoogleCloudPlatform/gsutil

rsync multiple files speed slows as it progresses

Open
#1,366 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

I am rsync'n multiple large files usually 15 at a time with each file being over 100GB to a cloud bucket.
Things start off great. I'm talking over 1G sync speeds.

the issue is when I get to about 70% completed and there are only a few files left the speeds drop to less then 100M and the last few files take 5 times longer to finish.

I am using the -m option. The server has 80 cores and 640MB ram on it.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the slowdown with gsutil rsync -m against a cloud bucket using several files over 100GB, and measure throughput as the transfer nears completion. No source file or test is identified in the issue, so locate the rsync entry point and determine whether the late-stage slowdown is reproducible and what change or evidence would define it as resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, cloud, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.