GoogleCloudPlatform / GoogleCloudPlatform/gsutil
gsutil always says "Reauthentication required" when using a security key
- Dominant language
- Python
- Stars
- 918
- Forks
- 335
- PR merge metrics
- No merged PRs in 30d
Description
I use a YubiKey and have to reauth once a day with `gcloud`, however even immediately after reauthenticating with `gcloud`, `gsutil` keeps prompting for it on each usage.
```
$ gsutil --debug cp gs://whatever/foo.json /tmp/foo.json
***************************** WARNING *****************************
*** You are running gsutil with debug output enabled.
*** Be aware that debug output includes authentication credentials.
*** Make sure to remove the value of the Authorization header for
*** each HTTP request printed to the console prior to posting to
*** a public medium such as a forum post or Stack Overflow.
***************************** WARNING *****************************
gsutil version: 5.23
checksum: cea0c6d70aecfa4525c7abab95b75dcf (OK)
boto version: 2.49.0
python version: 3.11.3 (main, Apr 7 2023, 19:25:52) [Clang 14.0.0 (clang-1400.0.29.202)]
OS: Darwin 22.4.0
multiprocessing available: True
using cloud sdk: True
pass cloud sdk credentials to gsutil: True
[...]
DEBUG 0512 13:25:48.249412 multiprocess_file_storage.py] Read credential file
DEBUG 0512 13:25:48.249971 multiprocess_file_storage.py] Read credential file
INFO 0512 13:25:48.251158 base_api.py] Calling method storage.objects.get with StorageObjectsGetRequest:
INFO 0512 13:25:48.252010 base_api.py] Making http GET to https://storage.googleapis.com/storage/v1/b/whatever/o/foo.json?alt=json&fields=crc32c%2CcustomerEncryption%2CcontentEncoding%2Cgeneration%2Cetag%2CmediaLink%2Cmd5Hash%2Cname%2CcontentType%2Csize&projection=noAcl
INFO 0512 13:25:48.252429 base_api.py] Headers: {'accept': 'application/json',
'accept-encoding': 'gzip, deflate',
'content-length': '0',
'user-agent': 'apitools Python/3.11.3 gsutil/5.23 (darwin) analytics/disabled '
'interactive/True command/cp google-cloud-sdk/430.0.0'}
INFO 0512 13:25:48.252702 base_api.py] Body: (none)
INFO 0512 13:25:48.252846 transport.py] Attempting refresh to obtain initial access_token
DEBUG 0512 13:25:48.253362 multiprocess_file_storage.py] Read credential file
DEBUG 0512 13:25:48.253572 multiprocess_file_storage.py] Read credential file
INFO 0512 13:25:48.253670 reauth_creds.py] Refreshing access_token
connect: (oauth2.googleapis.com, 443)
send: b'POST /token HTTP/1.1\r\nHost: oauth2.googleapis.com\r\nContent-Length: 234\r\ncontent-type: application/x-www-form-urlencoded\r\nuser-agent: Python-httplib2/0.20.4 (gzip)\r\naccept-encoding: gzip, deflate\r\n\r\n'
send: b'grant_type=refresh_token&client_id=XXX.apps.googleusercontent.com&client_secret=XXX'
reply: 'HTTP/1.1 400 Bad Request\r\n'
header: Date: Fri, 12 May 2023 11:25:48 GMT
header: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
header: Pragma: no-cache
header: Expires: Mon, 01 Jan 1990 00:00:00 GMT
header: Content-Type: application/json; charset=utf-8
header: Vary: Origin
header: Vary: X-Origin
header: Vary: Referer
header: Content-Encoding: gzip
header: Server: scaffolding on HTTPServer2
header: X-XSS-Protection: 0
header: X-Frame-Options: SAMEORIGIN
header: X-Content-Type-Options: nosniff
header: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
header: Transfer-Encoding: chunked
Reauthentication required.
[...]
Please insert your security key and press enter...^CDEBUG: Exception stack trace:
Traceback (most recent call last):
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/third_party/pyu2f/pyu2f/convenience/localauthenticator.py", line 36, in Authenticate
device = u2f.GetLocalU2FInterface(origin=self.origin)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/third_party/pyu2f/pyu2f/u2f.py", line 45, in GetLocalU2FInterface
raise errors.NoDeviceFoundError()
pyu2f.errors.NoDeviceFoundError
DEBUG: Caught CTRL-C (signal 2) - Exception stack trace:
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gsutil", line 21, in
gsutil.RunMain()
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gsutil.py", line 151, in RunMain
sys.exit(gslib.__main__.main())
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 436, in main
return _RunNamedCommandAndHandleExceptions(
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 633, in _RunNamedCommandAndHandleExceptions
return command_runner.RunNamedCommand(command_name,
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/command_runner.py", line 421, in RunNamedCommand
return_code = command_inst.RunCommand()
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/commands/cp.py", line 1054, in RunCommand
name_expansion_iterator = CopyObjectsIterator(
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/name_expansion.py", line 677, in __init__
name_expansion_dest_tuple = next(self.name_expansion_dest_iter)
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/commands/cp.py", line 990, in _ConstructNameExpansionIteratorDstTupleIterator
NameExpansionIterator(
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/name_expansion.py", line 476, in NameExpansionIterator
if name_expansion_iterator.IsEmpty():
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/plurality_checkable_iterator.py", line 100, in IsEmpty
return not self._PopulateHead()
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/plurality_checkable_iterator.py", line 70, in _PopulateHead
e = next(self.base_iterator)
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/name_expansion.py", line 242, in __iter__
src_url_expands_to_multi = post_step1_iter.HasPlurality()
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/plurality_checkable_iterator.py", line 105, in HasPlurality
return self._PopulateHead(num_elements=2) > 1
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/plurality_checkable_iterator.py", line 70, in _PopulateHead
e = next(self.base_iterator)
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/wildcard_iterator.py", line 538, in IterAll
for blr in self.__iter__(bucket_listing_fields=bucket_listing_fields,
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/wildcard_iterator.py", line 189, in __iter__
get_object = self.gsutil_api.GetObjectMetadata(
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/cloud_api_delegator.py", line 319, in GetObjectMetadata
return self._GetApi(provider).GetObjectMetadata(bucket_name,
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/gcs_json_api.py", line 1041, in GetObjectMetadata
object_metadata = self._GetObjectMetadataHelper(bucket_name,
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/gcs_json_api.py", line 998, in _GetObjectMetadataHelper
return self.api_client.objects.Get(apitools_request,
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/third_party/storage_apitools/storage_v1_client.py", line 1124, in Get
return self._RunMethod(
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/base_api.py", line 734, in _RunMethod
http_response = http_wrapper.MakeRequest(
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/http_wrapper.py", line 348, in MakeRequest
return _MakeRequestNoRetry(
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/http_wrapper.py", line 397, in _MakeRequestNoRetry
info, content = http.request(
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/vendored/oauth2client/oauth2client/transport.py", line 159, in new_request
credentials._refresh(orig_request_method)
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/vendored/oauth2client/oauth2client/client.py", line 761, in _refresh
self._do_refresh_request(http)
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/third_party/google-reauth-python/google_reauth/reauth_creds.py", line 112, in _do_refresh_request
self._update(*reauth.refresh_access_token(
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/third_party/google-reauth-python/google_reauth/reauth.py", line 283, in refresh_access_token
rapt = get_rapt_token(
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/third_party/google-reauth-python/google_reauth/reauth.py", line 194, in get_rapt_token
rapt_token = _obtain_rapt(
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/third_party/google-reauth-python/google_reauth/reauth.py", line 144, in _obtain_rapt
msg = _run_next_challenge(msg, http_request, access_token)
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/third_party/google-reauth-python/google_reauth/reauth.py", line 93, in _run_next_challenge
client_input = c.obtain_challenge_input(challenge)
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/third_party/google-reauth-python/google_reauth/challenges.py", line 110, in obtain_challenge_input
response = api.Authenticate(app_id, challenge_data,
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/third_party/pyu2f/pyu2f/convenience/authenticator.py", line 44, in Authenticate
result = authenticator.Authenticate(app_id,
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/third_party/pyu2f/pyu2f/convenience/localauthenticator.py", line 39, in Authenticate
six.input()
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/sig_handling.py", line 92, in _SignalHandler
_final_signal_handlers[signal_num](signal_num, cur_stack_frame)
File "/Users/tsuna/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 520, in _HandleControlC
stack_trace = ''.join(traceback.format_list(traceback.extract_stack()))
```
This makes `gsutil` super annoying to work with. Other people on my team are reporting the same problem since we turned on security key with daily refreshes for `gcloud` and this problem only seems to impact `gsutil`.
Contributor guide
Research direction
Trace the refresh path from gslib/third_party/google-reauth-python/google_reauth/reauth_creds.py and google_reauth/reauth.py, using the reported gsutil --debug cp flow as the starting point. Compare how gcloud and gsutil handle the refreshed credentials; done means gsutil no longer repeatedly requests security-key reauthentication immediately after gcloud reauthenticates.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python
- Domain
- authentication, cli
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100