GoogleCloudPlatform / GoogleCloudPlatform/gsutil
Slow rsync not using available bandwidth
- Dominant language
- Python
- Stars
- 918
- Forks
- 335
- PR merge metrics
- No merged PRs in 30d
Description
I'm copying over 5000 files from GCP to AWS S3. Each file is around 870 MB in size. I run this command:
```bash
gsutil -m -o GSUtil:parallel_process_count=4 -o GSUtil:parallel_thread_count=1 -o GSUtil:sliced_object_download_max_components=8 -o Credentials:gs_service_key_file=$gcp_key_file -o s3:host=s3.$aws_region.amazonaws.com -o s3:use-sigv4=True -o Boto:https_validate_certificates=True rsync $gcs_bucket_url $s3_bucket_url
```
on c5n.9xlarge EC2 machine (32 vCPU, 96 GB memory, 50 Gbps network bandwidth).
Copying of 4 files takes about 4 minutes. NetHogs shows summary download speed at about 24 MB/sec.
`top` command shows usage:
```
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16061 ubuntu 20 0 1004732 36992 6532 S 172.1 0.0 806:58.46 gsutil
16184 ubuntu 20 0 286656 84156 11292 S 28.6 0.1 115:49.95 gsutil
16182 ubuntu 20 0 286912 84624 11180 S 27.6 0.1 115:36.53 gsutil
16194 ubuntu 20 0 286400 84084 11136 S 27.2 0.1 114:53.46 gsutil
16188 ubuntu 20 0 286656 84736 11128 S 22.3 0.1 115:51.91 gsutil
```
It looks like there is a single (master?) process that controls the rsync and 4 other processes (which corresponds to number of files synced simultaneously).
Changing `parallel_process_count` to higher number results in more files synced at once, but slower, total network usage states about the same.
When I accidentally started rsync command for the second time it doubled the cpu and network usage, so they are not a problem, but the one process that controls the sync.
`gsutil version -l` output:
```
gsutil version: 4.51
checksum: PACKAGED_GSUTIL_INSTALLS_DO_NOT_HAVE_CHECKSUMS (!= a4c57d9b2479f11efe1b0ffb6470c0c5)
boto version: 2.49.0
python version: 3.6.9 (default, Apr 18 2020, 01:56:04) [GCC 8.4.0]
OS: Linux 4.15.0-1065-aws
multiprocessing available: True
using cloud sdk: False
pass cloud sdk credentials to gsutil: False
config path(s): No config found
gsutil path: /home/ubuntu/.local/bin/gsutil
compiled crcmod: True
installed via package manager: True
editable install: False
```
Contributor guide
Research direction
Start by reproducing the reported gsutil 4.51 rsync command with the listed parallel-process and parallel-thread settings on the described environment. Inspect the rsync entry point and process coordination, then compare throughput as parallel_process_count changes. Done means identifying the limiting behavior and documenting or validating a concrete improvement with a reproducible test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, gcp, python
- Domain
- cli, cloud, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100