GoogleCloudPlatform / GoogleCloudPlatform/gsutil
gsutil ignoring .boto configuration when doing resumable uploads
- Dominant language
- Python
- Stars
- 918
- Forks
- 335
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
we're having an on-prem machine trying to upload files to GCS. Using dedicated interconnect, we created a private service connect endpoint and redefined some of the hosts in /etc/hosts (as we cannot simply change in DNS since other servers could suffer).
Using debugging option, we were able to confirm that gsutil is working fine when a file is under the resumable_threshold value. However, when file size is over the threshold, gsutil ignores the .boto configuration and tries using _storage.googleapis.com_.
We also had to redefine the _oauth2_ endpoint to work with our private service connect IP.
My .boto and hosts file are below.
BR,
Daniel
---
.boto
```
[Credentials]
gs_host = storage-mygoogleapis.p.googleapis.com
gs_host_header = storage-mygoogleapis.p.googleapis.com
gs_json_host = storage-mygoogleapis.p.googleapis.com # does not work. instead we use: storage.googleapis.com and put storage.googleapis.com to /etc/hosts
gs_json_host_header = www.googleapis.com
[Boto]
https_validate_certificates = True
[GoogleCompute]
[GSUtil]
resumable_threshold = 8388608
content_language = en
prefer_api = xml
default_api_version = 2
[OAuth2]
provider_token_uri = https://oauth2-mygoogleapis.p.googleapis.com/token
```
/etc/hosts:
```
# enabling usage of google interconnect
192.168.123.1 storage-mygoogleapis.p.googleapis.com
# 192.168.123.1 storage.googleapis.com
192.168.123.1 oauth2-mygoogleapis.p.googleapis.com
```
Contributor guide
Research direction
Start by tracing gsutil's resumable-upload path and how it reads the .boto settings, then compare behavior below and above resumable_threshold. Use the provided .boto configuration and /etc/hosts setup to reproduce the endpoint selection; done means resumable uploads honor the configured service and OAuth endpoints.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python
- Domain
- cli, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100