GoogleCloudPlatform / GoogleCloudPlatform/gsutil

gsutil doesn't work with proxy on python3

Open
#880 4 comments 4 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

Tested the same setup with and without a proxy. Without a proxy it works. With a proxy I get this:
```
Traceback (most recent call last):
File "/usr/local/bin/gsutil", line 10, in
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/gslib/__main__.py", line 431, in main
user_project=user_project)
File "/usr/local/lib/python3.6/site-packages/gslib/__main__.py", line 760, in _RunNamedCommandAndHandleExceptions
_HandleUnknownFailure(e)
File "/usr/local/lib/python3.6/site-packages/gslib/__main__.py", line 626, in _RunNamedCommandAndHandleExceptions
user_project=user_project)
File "/usr/local/lib/python3.6/site-packages/gslib/command_runner.py", line 411, in RunNamedCommand
return_code = command_inst.RunCommand()
File "/usr/local/lib/python3.6/site-packages/gslib/commands/ls.py", line 640, in RunCommand
listing_helper.ExpandUrlAndPrint(storage_url))
File "/usr/local/lib/python3.6/site-packages/gslib/utils/ls_helper.py", line 369, in ExpandUrlAndPrint
print_initial_newline=False)
File "/usr/local/lib/python3.6/site-packages/gslib/utils/ls_helper.py", line 443, in _RecurseExpandUrlAndPrint
bucket_listing_fields=self.bucket_listing_fields):
File "/usr/local/lib/python3.6/site-packages/gslib/wildcard_iterator.py", line 497, in IterAll
expand_top_level_buckets=expand_top_level_buckets):
File "/usr/local/lib/python3.6/site-packages/gslib/wildcard_iterator.py", line 173, in __iter__
fields=bucket_listing_fields):
File "/usr/local/lib/python3.6/site-packages/gslib/gcs_json_api.py", line 715, in ListObjects
global_params=global_params)
File "/usr/local/lib/python3.6/site-packages/gslib/third_party/storage_apitools/storage_v1_client.py", line 1217, in List
config, request, global_params=global_params)
File "/usr/local/lib/python3.6/site-packages/apitools/base/py/base_api.py", line 729, in _RunMethod
http, http_request, **opts)
File "/usr/local/lib/python3.6/site-packages/apitools/base/py/http_wrapper.py", line 356, in MakeRequest
max_retry_wait, total_wait_sec))
File "/usr/local/lib/python3.6/site-packages/gslib/utils/retry_util.py", line 65, in WarnAfterManyRetriesHandler
http_wrapper.HandleExceptionsAndRebuildHttpConnections(retry_args)
File "/usr/local/lib/python3.6/site-packages/apitools/base/py/http_wrapper.py", line 304, in HandleExceptionsAndRebuildHttpConnections
raise retry_args.exc
File "/usr/local/lib/python3.6/site-packages/apitools/base/py/http_wrapper.py", line 346, in MakeRequest
check_response_func=check_response_func)
File "/usr/local/lib/python3.6/site-packages/apitools/base/py/http_wrapper.py", line 396, in _MakeRequestNoRetry
redirections=redirections, connection_type=connection_type)
File "/usr/local/lib/python3.6/site-packages/oauth2client/transport.py", line 159, in new_request
credentials._refresh(orig_request_method)
File "/usr/local/lib/python3.6/site-packages/oauth2client/client.py", line 761, in _refresh
self._do_refresh_request(http)
File "/usr/local/lib/python3.6/site-packages/oauth2client/client.py", line 780, in _do_refresh_request
body=body, headers=headers)
File "/usr/local/lib/python3.6/site-packages/oauth2client/transport.py", line 282, in request
connection_type=connection_type)
File "/usr/local/lib/python3.6/site-packages/httplib2/__init__.py", line 1953, in request
cachekey,
File "/usr/local/lib/python3.6/site-packages/httplib2/__init__.py", line 1618, in _request
conn, request_uri, method, body, headers
File "/usr/local/lib/python3.6/site-packages/httplib2/__init__.py", line 1524, in _conn_request
conn.connect()
File "/usr/local/lib/python3.6/site-packages/httplib2/__init__.py", line 1305, in connect
sock.connect((self.host, self.port))
File "/usr/local/lib/python3.6/site-packages/socks.py", line 406, in connect
self.__negotiatehttp(destpair[0],destpair[1])
File "/usr/local/lib/python3.6/site-packages/socks.py", line 357, in __negotiatehttp
while resp.find("\r\n\r\n")==-1:
TypeError: a bytes-like object is required, not 'str'
```

`gsutil -D version` output:
```
gsutil version: 4.42
checksum: PACKAGED_GSUTIL_INSTALLS_DO_NOT_HAVE_CHECKSUMS (!= d901cf5a002200f4a5526a372681677c)
boto version: 2.49.0
python version: 3.6.9 (default, Jul 13 2019, 14:58:50) [GCC 8.3.0]
OS: Linux 4.4.0-142-generic
multiprocessing available: True
using cloud sdk: False
pass cloud sdk credentials to gsutil: False
config path(s): /usr/local//.boto
gsutil path: /usr/local/bin/gsutil
compiled crcmod: True
installed via package manager: True
editable install: False
Command being run: /usr/local/bin/gsutil -D version
config_file_list: ['/usr/local//.boto']
config: [('working_dir', '/mnt/pyami'), ('debug', '0'), ('proxy', 'REDACTED'), ('proxy_port', 'REDACTED'), ('https_validate_certificates', 'True'), ('working_dir', '/mnt/pyami'), ('debug', '0'), ('content_language', 'en'), ('default_api_version', '2'), ('default_project_id', '')]
gsutil version: 4.42
```
`gsutil` is running in a Docker container made from `python-slim` base image and was installed using `pip install gsutil`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.