GoogleCloudPlatform / GoogleCloudPlatform/gsutil

gsutil ignoring .boto configuration when doing resumable uploads

Open
#1,264 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

Hi,

we're having an on-prem machine trying to upload files to GCS. Using dedicated interconnect, we created a private service connect endpoint and redefined some of the hosts in /etc/hosts (as we cannot simply change in DNS since other servers could suffer).

Using debugging option, we were able to confirm that gsutil is working fine when a file is under the resumable_threshold value. However, when file size is over the threshold, gsutil ignores the .boto configuration and tries using _storage.googleapis.com_.

We also had to redefine the _oauth2_ endpoint to work with our private service connect IP.

My .boto and hosts file are below.

BR,
Daniel

---
.boto
```
[Credentials]

gs_host = storage-mygoogleapis.p.googleapis.com
gs_host_header = storage-mygoogleapis.p.googleapis.com
gs_json_host = storage-mygoogleapis.p.googleapis.com # does not work. instead we use: storage.googleapis.com and put storage.googleapis.com to /etc/hosts
gs_json_host_header = www.googleapis.com

[Boto]
https_validate_certificates = True

[GoogleCompute]

[GSUtil]
resumable_threshold = 8388608
content_language = en
prefer_api = xml
default_api_version = 2

[OAuth2]
provider_token_uri = https://oauth2-mygoogleapis.p.googleapis.com/token
```

/etc/hosts:
```
# enabling usage of google interconnect
192.168.123.1 storage-mygoogleapis.p.googleapis.com
# 192.168.123.1 storage.googleapis.com
192.168.123.1 oauth2-mygoogleapis.p.googleapis.com
```

Contributor guide

Open the contributing guide

Research direction

Start by tracing gsutil's resumable-upload path and how it reads the .boto settings, then compare behavior below and above resumable_threshold. Use the provided .boto configuration and /etc/hosts setup to reproduce the endpoint selection; done means resumable uploads honor the configured service and OAuth endpoints.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
cli, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.