GoogleCloudPlatform / GoogleCloudPlatform/gsutil

`gcloud storage rsync` is slower than `gsutil rsync` when

Open
#1,832 0 comments 3 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

Intuitively, these steps are functionally equivalent.
In our experiments, the new method is ~12x slower. Is there a configuration problem?

deprecated builder:

```
- name: 'gcr.io/cloud-builders/gsutil'
args:
- '-m'
- 'rsync'
- '-d'
- '-r'
- '-c'
- 'dags/'
- '${_COMPOSER_BUCKET}/dags'
```

new method:

```
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk:slim'
args:
- 'storage'
- 'rsync'
- '--checksums-only'
- '--delete-unmatched-destination-objects'
- '--recursive'
- 'dags/'
- '${_COMPOSER_BUCKET}/dags'
```

Contributor guide

Open the contributing guide

Research direction

Reproduce the two Cloud Build configurations using the provided gsutil rsync and gcloud storage rsync commands, then compare their execution behavior and timing. Determine whether the differing flags or configuration explain the reported slowdown; done means documenting the cause or confirming the equivalent configuration needed for comparable performance.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud
Domain
cli, cloud, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.