GoogleCloudPlatform / GoogleCloudPlatform/gsutil

TypeError in gsutil cp and perfdiag due to gs_port in boto configuration

Open
#1,876 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

The current boto configuration causes gsutil cp and gsutil perfdiag to raise a TypeError exception.
```
[Credentials]

gs_host = xx
gs_port = 8021

gs_json_host = xx
gs_json_port = 8020
```

```
Traceback (most recent call last):
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/boto_translation.py", line 655, in _PerformSimpleDownload
key.get_contents_to_file(download_stream,
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/gs/key.py", line 234, in get_contents_to_file
self.get_file(fp, headers, cb, num_cb, torrent=torrent,
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/gs/key.py", line 174, in get_file
self._get_file_internal(fp, headers=headers, cb=cb, num_cb=num_cb,
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/s3/key.py", line 1544, in _get_file_internal
self.open('r', headers, query_args=query_args,
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/s3/key.py", line 365, in open
self.open_read(headers=headers, query_args=query_args,
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/gs/key.py", line 164, in open_read
super(Key, self).open_read(headers=headers, query_args=query_args,
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/s3/key.py", line 313, in open_read
self.resp = self.bucket.connection.make_request(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/s3/connection.py", line 668, in make_request
host = self.calling_format.build_host(self.server_name(), bucket)
^^^^^^^^^^^^^^^^^^
File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/vendored/boto/boto/connection.py", line 661, in server_name
signature_host = '%s:%d' % (self.host, port)
~~~~~~~~^~~~~~~~~~~~~~~~~~~
TypeError: %d format: a real number is required, not str
```

The issue originates from the `gs_port` setting in the boto configuration. In `gslib/boto_translation.py`, the `_AddCustomEndpointToKey` function retrieves the port using `config.get`, which returns a string. This leads to a **TypeError** when an integer is expected.
https://github.com/GoogleCloudPlatform/gsutil/blob/141e2a098f56e3ba4bcebb1a5cf89947efece4a4/gslib/boto_translation.py#L131C1-L141C72
```
def _AddCustomEndpointToKey(key):
"""Update Boto Key object with user config's custom endpoint."""
user_setting_to_key_attribute = {
'gs_host': 'host',
'gs_port': 'port',
'gs_host_header': 'host_header',
}
for user_setting, key_attribute in user_setting_to_key_attribute.items():
user_setting_value = config.get('Credentials', user_setting, None)
if user_setting_value is not None:
setattr(key.bucket.connection, key_attribute, user_setting_value)
```

I believe replacing `config.get` with the `config.getint` function will fix the issue (for `gs_port` only!)
For example, `config.getint` is used for the same `gs_port` here: https://github.com/GoogleCloudPlatform/gsutil/blob/141e2a098f56e3ba4bcebb1a5cf89947efece4a4/gslib/commands/perfdiag.py#L521

If you think this is the right way to fix the problem, I can make a PR

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.