GoogleCloudPlatform / GoogleCloudPlatform/gsutil

gsutils NO_PROXY not working

Open
#1,503 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

Hello,

no_proxy seems to not be respected:

```
# env | grep -i prox
no_proxy=*.googleapis.com,*google*
https_proxy=http://172.24.0.0:3128
```

```
gsutil -D cp gs://mybucket_test/readme.adoc .
```

Error:

```
# gsutil -D cp gs://testbucket/readme.adoc .
***************************** WARNING *****************************
*** You are running gsutil with debug output enabled.
*** Be aware that debug output includes authentication credentials.
*** Make sure to remove the value of the Authorization header for
*** each HTTP request printed to the console prior to posting to
*** a public medium such as a forum post or Stack Overflow.
***************************** WARNING *****************************
gsutil version: 5.9
checksum: dc0f1c0e775aa93031be8ec82ce7dd84 (OK)
boto version: 2.49.0
python version: 3.9.7 (default, Nov 24 2021, 21:15:59) [GCC 10.3.1 20211027]
OS: Linux 5.4.170+
multiprocessing available: False
using cloud sdk: True
pass cloud sdk credentials to gsutil: True
config path(s): No config found
gsutil path: /google-cloud-sdk/bin/gsutil
compiled crcmod: True
installed via package manager: False
editable install: False
Command being run: /google-cloud-sdk/platform/gsutil/gsutil -o GSUtil:default_project_id=myproj -o GoogleCompute:service_account=default -D cp gs://testbucket/readme.adoc .
config_file_list: ['No config found']
config: [('working_dir', '/mnt/pyami'), ('debug', '0'), ('https_validate_certificates', 'true'), ('working_dir', '/mnt/pyami'), ('debug', '0'), ('default_project_id', 'myproj')]
DEBUG 0422 14:40:20.919526 multiprocess_file_storage.py] Read credential file
INFO 0422 14:40:20.920960 base_api.py] Calling method storage.objects.get with StorageObjectsGetRequest:
INFO 0422 14:40:20.922105 base_api.py] Making http GET to https://storage.googleapis.com/storage/v1/b/mytestbucket/o/readme.adoc?alt=json&fields=etag%2Cname%2CmediaLink%2Csize%2CcontentEncoding%2Cgeneration%2CcontentType%2Cmd5Hash%2Ccrc32c%2CcustomerEncryption&projection=noAcl
INFO 0422 14:40:20.922641 base_api.py] Headers: {'accept': 'application/json',
'accept-encoding': 'gzip, deflate',
'content-length': '0',
'user-agent': 'apitools Python/3.9.7 gsutil/5.9 (linux) analytics/disabled '
'interactive/True command/cp google-cloud-sdk/381.0.0'}
INFO 0422 14:40:20.922901 base_api.py] Body: (none)
process count: 1
thread count: 1
Traceback (most recent call last):
File "/google-cloud-sdk/platform/gsutil/gsutil", line 21, in
gsutil.RunMain()
File "/google-cloud-sdk/platform/gsutil/gsutil.py", line 151, in RunMain
sys.exit(gslib.__main__.main())
File "/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 435, in main
return _RunNamedCommandAndHandleExceptions(
File "/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 783, in _RunNamedCommandAndHandleExceptions
_HandleUnknownFailure(e)
File "/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 632, in _RunNamedCommandAndHandleExceptions
return command_runner.RunNamedCommand(command_name,
File "/google-cloud-sdk/platform/gsutil/gslib/command_runner.py", line 421, in RunNamedCommand
return_code = command_inst.RunCommand()
File "/google-cloud-sdk/platform/gsutil/gslib/commands/cp.py", line 1138, in RunCommand
self.Apply(_CopyFuncWrapper,
File "/google-cloud-sdk/platform/gsutil/gslib/command.py", line 1556, in Apply
self._SequentialApply(func, args_iterator, exception_handler, caller_id,
File "/google-cloud-sdk/platform/gsutil/gslib/command.py", line 1603, in _SequentialApply
args = next(args_iterator)
File "/google-cloud-sdk/platform/gsutil/gslib/name_expansion.py", line 687, in __next__
name_expansion_result = next(self.current_expansion_iter)
File "/google-cloud-sdk/platform/gsutil/gslib/plurality_checkable_iterator.py", line 96, in __next__
raise six.reraise(item_tuple[1].__class__, item_tuple[1], item_tuple[2])
File "/google-cloud-sdk/platform/gsutil/third_party/six/six.py", line 693, in reraise
raise value
File "/google-cloud-sdk/platform/gsutil/gslib/plurality_checkable_iterator.py", line 70, in _PopulateHead
e = next(self.base_iterator)
File "/google-cloud-sdk/platform/gsutil/gslib/name_expansion.py", line 291, in __iter__
for (names_container, blr) in post_step3_iter:
File "/google-cloud-sdk/platform/gsutil/gslib/plurality_checkable_iterator.py", line 96, in __next__
raise six.reraise(item_tuple[1].__class__, item_tuple[1], item_tuple[2])
File "/google-cloud-sdk/platform/gsutil/third_party/six/six.py", line 693, in reraise
raise value
File "/google-cloud-sdk/platform/gsutil/gslib/plurality_checkable_iterator.py", line 70, in _PopulateHead
e = next(self.base_iterator)
File "/google-cloud-sdk/platform/gsutil/gslib/name_expansion.py", line 533, in __iter__
for (names_container, blr) in self.tuple_iter:
File "/google-cloud-sdk/platform/gsutil/gslib/plurality_checkable_iterator.py", line 96, in __next__
raise six.reraise(item_tuple[1].__class__, item_tuple[1], item_tuple[2])
File "/google-cloud-sdk/platform/gsutil/third_party/six/six.py", line 693, in reraise
raise value
File "/google-cloud-sdk/platform/gsutil/gslib/plurality_checkable_iterator.py", line 70, in _PopulateHead
e = next(self.base_iterator)
File "/google-cloud-sdk/platform/gsutil/gslib/name_expansion.py", line 497, in __iter__
for blr in self.blr_iter:
File "/google-cloud-sdk/platform/gsutil/gslib/plurality_checkable_iterator.py", line 96, in __next__
raise six.reraise(item_tuple[1].__class__, item_tuple[1], item_tuple[2])
File "/google-cloud-sdk/platform/gsutil/third_party/six/six.py", line 693, in reraise
raise value
File "/google-cloud-sdk/platform/gsutil/gslib/plurality_checkable_iterator.py", line 70, in _PopulateHead
e = next(self.base_iterator)
File "/google-cloud-sdk/platform/gsutil/gslib/wildcard_iterator.py", line 539, in IterAll
for blr in self.__iter__(bucket_listing_fields=bucket_listing_fields,
File "/google-cloud-sdk/platform/gsutil/gslib/wildcard_iterator.py", line 190, in __iter__
get_object = self.gsutil_api.GetObjectMetadata(
File "/google-cloud-sdk/platform/gsutil/gslib/cloud_api_delegator.py", line 315, in GetObjectMetadata
return self._GetApi(provider).GetObjectMetadata(bucket_name,
File "/google-cloud-sdk/platform/gsutil/gslib/gcs_json_api.py", line 1014, in GetObjectMetadata
object_metadata = self._GetObjectMetadataHelper(bucket_name,
File "/google-cloud-sdk/platform/gsutil/gslib/gcs_json_api.py", line 971, in _GetObjectMetadataHelper
return self.api_client.objects.Get(apitools_request,
File "/google-cloud-sdk/platform/gsutil/gslib/third_party/storage_apitools/storage_v1_client.py", line 1124, in Get
return self._RunMethod(
File "/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/base_api.py", line 734, in _RunMethod
http_response = http_wrapper.MakeRequest(
File "/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/http_wrapper.py", line 359, in MakeRequest
retry_func(ExceptionRetryArgs(http, http_request, e, retry,
File "/google-cloud-sdk/platform/gsutil/gslib/utils/retry_util.py", line 65, in WarnAfterManyRetriesHandler
http_wrapper.HandleExceptionsAndRebuildHttpConnections(retry_args)
File "/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/http_wrapper.py", line 304, in HandleExceptionsAndRebuildHttpConnections
raise retry_args.exc
File "/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/http_wrapper.py", line 348, in MakeRequest
return _MakeRequestNoRetry(
File "/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/http_wrapper.py", line 397, in _MakeRequestNoRetry
info, content = http.request(
File "/google-cloud-sdk/platform/gsutil/gslib/vendored/oauth2client/oauth2client/transport.py", line 173, in new_request
resp, content = request(orig_request_method, uri, method, body,
File "/google-cloud-sdk/platform/gsutil/gslib/vendored/oauth2client/oauth2client/transport.py", line 280, in request
return http_callable(uri, method=method, body=body, headers=headers,
File "/google-cloud-sdk/platform/gsutil/third_party/httplib2/python3/httplib2/__init__.py", line 1701, in request
(response, content) = self._request(
File "/google-cloud-sdk/platform/gsutil/third_party/httplib2/python3/httplib2/__init__.py", line 1421, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/google-cloud-sdk/platform/gsutil/third_party/httplib2/python3/httplib2/__init__.py", line 1343, in _conn_request
conn.connect()
File "/google-cloud-sdk/platform/gsutil/third_party/httplib2/python3/httplib2/__init__.py", line 1133, in connect
sock.connect((self.host, self.port))
File "/google-cloud-sdk/platform/gsutil/third_party/httplib2/python3/httplib2/socks.py", line 504, in connect
self.__negotiatehttp(destpair[0], destpair[1])
File "/google-cloud-sdk/platform/gsutil/third_party/httplib2/python3/httplib2/socks.py", line 465, in __negotiatehttp
raise HTTPError((statuscode, statusline[2]))
httplib2.socks.HTTPError: (403, b'Forbidden')
```

If i do unset https_proxy everything works fine.

Related to #1086 or #435

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.