GoogleCloudPlatform / GoogleCloudPlatform/gsutil

Slow gsutil rsync performance - checks legacy ACL's for object OWNER, does not check newer IAM roles

Open
#942 2 comments 8 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

https://github.com/GoogleCloudPlatform/gsutil/blob/9a62a3b73530e1995b0eb3b94307f9cacccf399f/gslib/commands/rsync.py#L1467

Steps to reproduce:

1. Remove user from project-level Owner and Editor roles. Grant them only storage.admin at the project level. (Also try granting them legacyObjectOwner permission on the bucket).
2. Run `gsutil rsync` on a large file (>1GB).
3. In a separate directory, download the large file, then re-upload it.
4. Go back to the original rsync directory and run `gsutil rsync` again.

Expected behavior:
Gsutil should calculate local hash, compare it to the hash in GCS, and run an object patch operation to update the object's MTIME metadata value only.

Actual behavior:
Gsutil reports that you don't have OWNER permissions on the object, and so it proceeds to upload the entire object.

But I do, in fact, have the appropriate permissions to perform an object patch! It's just that gsutil's validation looks at the old-school ACL's, instead of the newer IAM permissions.

Suggested solution: gsutil should detect if appropriate IAM permission exists, and attempt to perform the patch if the IAM permission exists. OR, gsutil can attempt the patch without doing the permission check, and if it fails, fall back to uploading the entire object.

Contributor guide

Open the contributing guide

Research direction

Start at gslib/commands/rsync.py around line 1467 and inspect the permission check used by rsync before patching object metadata. Reproduce with a large file and a storage.admin-only account, then run gsutil rsync twice. Done means IAM-authorized users take the metadata patch path instead of re-uploading the object, with failed patches falling back as described.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
authorization, cli, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.