GoogleCloudPlatform / GoogleCloudPlatform/gsutil

Slow rsync not using available bandwidth

Open
#1,023 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

I'm copying over 5000 files from GCP to AWS S3. Each file is around 870 MB in size. I run this command:

```bash
gsutil -m -o GSUtil:parallel_process_count=4 -o GSUtil:parallel_thread_count=1 -o GSUtil:sliced_object_download_max_components=8 -o Credentials:gs_service_key_file=$gcp_key_file -o s3:host=s3.$aws_region.amazonaws.com -o s3:use-sigv4=True -o Boto:https_validate_certificates=True rsync $gcs_bucket_url $s3_bucket_url
```

on c5n.9xlarge EC2 machine (32 vCPU, 96 GB memory, 50 Gbps network bandwidth).

Copying of 4 files takes about 4 minutes. NetHogs shows summary download speed at about 24 MB/sec.

`top` command shows usage:

```
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16061 ubuntu 20 0 1004732 36992 6532 S 172.1 0.0 806:58.46 gsutil
16184 ubuntu 20 0 286656 84156 11292 S 28.6 0.1 115:49.95 gsutil
16182 ubuntu 20 0 286912 84624 11180 S 27.6 0.1 115:36.53 gsutil
16194 ubuntu 20 0 286400 84084 11136 S 27.2 0.1 114:53.46 gsutil
16188 ubuntu 20 0 286656 84736 11128 S 22.3 0.1 115:51.91 gsutil
```

It looks like there is a single (master?) process that controls the rsync and 4 other processes (which corresponds to number of files synced simultaneously).

Changing `parallel_process_count` to higher number results in more files synced at once, but slower, total network usage states about the same.

When I accidentally started rsync command for the second time it doubled the cpu and network usage, so they are not a problem, but the one process that controls the sync.

`gsutil version -l` output:

```
gsutil version: 4.51
checksum: PACKAGED_GSUTIL_INSTALLS_DO_NOT_HAVE_CHECKSUMS (!= a4c57d9b2479f11efe1b0ffb6470c0c5)
boto version: 2.49.0
python version: 3.6.9 (default, Apr 18 2020, 01:56:04) [GCC 8.4.0]
OS: Linux 4.15.0-1065-aws
multiprocessing available: True
using cloud sdk: False
pass cloud sdk credentials to gsutil: False
config path(s): No config found
gsutil path: /home/ubuntu/.local/bin/gsutil
compiled crcmod: True
installed via package manager: True
editable install: False
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reported gsutil 4.51 rsync command with the listed parallel-process and parallel-thread settings on the described environment. Inspect the rsync entry point and process coordination, then compare throughput as parallel_process_count changes. Done means identifying the limiting behavior and documenting or validating a concrete improvement with a reproducible test.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, gcp, python
Domain
cli, cloud, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.