GoogleCloudPlatform / GoogleCloudPlatform/gsutil
Slow gsutil rsync performance - checks legacy ACL's for object OWNER, does not check newer IAM roles
- Dominant language
- Python
- Stars
- 918
- Forks
- 335
- PR merge metrics
- No merged PRs in 30d
Description
https://github.com/GoogleCloudPlatform/gsutil/blob/9a62a3b73530e1995b0eb3b94307f9cacccf399f/gslib/commands/rsync.py#L1467
Steps to reproduce:
1. Remove user from project-level Owner and Editor roles. Grant them only storage.admin at the project level. (Also try granting them legacyObjectOwner permission on the bucket).
2. Run `gsutil rsync` on a large file (>1GB).
3. In a separate directory, download the large file, then re-upload it.
4. Go back to the original rsync directory and run `gsutil rsync` again.
Expected behavior:
Gsutil should calculate local hash, compare it to the hash in GCS, and run an object patch operation to update the object's MTIME metadata value only.
Actual behavior:
Gsutil reports that you don't have OWNER permissions on the object, and so it proceeds to upload the entire object.
But I do, in fact, have the appropriate permissions to perform an object patch! It's just that gsutil's validation looks at the old-school ACL's, instead of the newer IAM permissions.
Suggested solution: gsutil should detect if appropriate IAM permission exists, and attempt to perform the patch if the IAM permission exists. OR, gsutil can attempt the patch without doing the permission check, and if it fails, fall back to uploading the entire object.
Contributor guide
Research direction
Start at gslib/commands/rsync.py around line 1467 and inspect the permission check used by rsync before patching object metadata. Reproduce with a large file and a storage.admin-only account, then run gsutil rsync twice. Done means IAM-authorized users take the metadata patch path instead of re-uploading the object, with failed patches falling back as described.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python
- Domain
- authorization, cli, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100