tensorflow / tensorflow/tensorboard

`tensorboard dev upload` fails with unhelpful `invalid_grant: Token has been expired or revoked.` when auth has expired

Open
#5,552 3 comments 0 reactions 1 assignee View on GitHub

@bmd3k is already working on this.

Since Feb 3, 2022.

tbdev type:bug
Dominant language
TypeScript
Stars
7.2k
Forks
1.7k
Avg merge
4d 22h
Merged PRs (30d)
1

Description

Diagnostics
Diagnostics output 
--- check: autoidentify
INFO: diagnose_tensorboard.py version e43767ef2b648d0d5d57c00f38ccbd38390e38da

--- check: general
INFO: sys.version_info: sys.version_info(major=3, minor=9, micro=7, releaselevel='final', serial=0)
INFO: os.name: posix
INFO: os.uname(): posix.uname_result(sysname='Darwin', nodename='NDPH8334.local', release='21.1.0', version='Darwin Kernel Version 21.1.0: Wed Oct 13 17:33:24 PDT 2021; root:xnu-8019.41.5~1/RELEASE_ARM64_T8101', machine='x86_64')
INFO: sys.getwindowsversion(): N/A

--- check: package_management
INFO: has conda-meta: True
INFO: $VIRTUAL_ENV: None

--- check: installed_packages
INFO: installed: tensorboard==2.7.0
WARNING: no installation among: ['tensorflow', 'tensorflow-gpu', 'tf-nightly', 'tf-nightly-2.0-preview', 'tf-nightly-gpu', 'tf-nightly-gpu-2.0-preview']
WARNING: no installation among: ['tensorflow-estimator', 'tensorflow-estimator-2.0-preview', 'tf-estimator-nightly']
INFO: installed: tensorboard-data-server==0.6.1

--- check: tensorboard_python_version
INFO: tensorboard.version.VERSION: '2.7.0'

--- check: tensorflow_python_version
Traceback (most recent call last):
  File "/Users/hangy/Desktop/diagnose_tensorboard.py", line 522, in main
    suggestions.extend(check())
  File "/Users/hangy/Desktop/diagnose_tensorboard.py", line 75, in wrapper
    result = fn()
  File "/Users/hangy/Desktop/diagnose_tensorboard.py", line 278, in tensorflow_python_version
    import tensorflow as tf
ModuleNotFoundError: No module named 'tensorflow'

--- check: tensorboard_data_server_version
INFO: data server binary: '/Users/hangy/opt/miniconda3/envs/home_Env/lib/python3.9/site-packages/tensorboard_data_server/bin/server'
INFO: data server binary version: b'rustboard 0.6.1'

--- check: tensorboard_binary_path
INFO: which tensorboard: b'/Users/hangy/opt/miniconda3/envs/home_Env/bin/tensorboard\n'

--- check: addrinfos
socket.has_ipv6 = True
socket.AF_UNSPEC = <AddressFamily.AF_UNSPEC: 0>
socket.SOCK_STREAM = <SocketKind.SOCK_STREAM: 1>
socket.AI_ADDRCONFIG = <AddressInfo.AI_ADDRCONFIG: 1024>
socket.AI_PASSIVE = <AddressInfo.AI_PASSIVE: 1>
Loopback flags: <AddressInfo.AI_ADDRCONFIG: 1024>
Loopback infos: [(<AddressFamily.AF_INET6: 30>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('::1', 0, 0, 0)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 0))]
Wildcard flags: <AddressInfo.AI_PASSIVE: 1>
Wildcard infos: [(<AddressFamily.AF_INET6: 30>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('::', 0, 0, 0)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('0.0.0.0', 0))]

--- check: readable_fqdn
INFO: socket.getfqdn(): '1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa'

--- check: stat_tensorboardinfo
INFO: directory: /var/folders/4c/39j9ycnx4wb4v4bff1pb657cjhd657/T/.tensorboard-info
INFO: .tensorboard-info directory does not exist

--- check: source_trees_without_genfiles
INFO: tensorboard_roots (1): ['/Users/hangy/opt/miniconda3/envs/home_Env/lib/python3.9/site-packages']; bad_roots (0): []

--- check: full_pip_freeze
INFO: pip freeze --all:
absl-py==0.15.0
accelerometer==4.0.0
alembic==1.7.4
antlr4-python3-runtime==4.8
anyio @ file:///Users/runner/miniforge3/conda-bld/anyio_1626998615142/work/dist
appnope @ file:///Users/runner/miniforge3/conda-bld/appnope_1610094673755/work
argon2-cffi @ file:///Users/runner/miniforge3/conda-bld/argon2-cffi_1625821234041/work
astroid==2.8.4
async-generator==1.10
attrs @ file:///home/conda/feedstock_root/build_artifacts/attrs_1620387926260/work
autopage==0.4.0
Babel @ file:///home/conda/feedstock_root/build_artifacts/babel_1619719576210/work
backcall @ file:///home/conda/feedstock_root/build_artifacts/backcall_1592338393461/work
backports.entry-points-selectable==1.1.0
backports.functools-lru-cache @ file:///home/conda/feedstock_root/build_artifacts/backports.functools_lru_cache_1618230623929/work
bleach @ file:///home/conda/feedstock_root/build_artifacts/bleach_1629908509068/work
brotlipy==0.7.0
cachetools==4.2.4
catch22==0.2.0
certifi==2021.5.30
cffi @ file:///Users/runner/miniforge3/conda-bld/cffi_1631636467320/work
cfgv==3.3.1
chardet @ file:///Users/runner/miniforge3/conda-bld/chardet_1610093613037/work
charset-normalizer @ file:///home/conda/feedstock_root/build_artifacts/charset-normalizer_1626371162869/work
click==8.0.3
cliff==3.9.0
cmaes==0.8.2
cmd2==2.2.0
colorama==0.4.4
colorlog==6.6.0
cryptography @ file:///Users/runner/miniforge3/conda-bld/cryptography_1616851523607/work
cycler==0.10.0
DateTime==4.3
debugpy @ file:///Users/runner/miniforge3/conda-bld/debugpy_1627075104865/work
decorator @ file:///home/conda/feedstock_root/build_artifacts/decorator_1631346842025/work
defusedxml @ file:///home/conda/feedstock_root/build_artifacts/defusedxml_1615232257335/work
distlib==0.3.3
docopt==0.6.2
docutils==0.18
entrypoints @ file:///home/conda/feedstock_root/build_artifacts/entrypoints_1605121927639/work/dist/entrypoints-0.3-py2.py3-none-any.whl
filelock==3.3.0
Flask==2.0.2
fonttools==4.25.0
google-auth==2.3.3
google-auth-oauthlib==0.4.6
greenlet==1.1.2
grip==4.5.2
grpcio==1.41.1
hydra-core==1.1.1
identify==2.3.0
idna @ file:///home/conda/feedstock_root/build_artifacts/idna_1609836280497/work
importlib-metadata @ file:///Users/runner/miniforge3/conda-bld/importlib-metadata_1630267507666/work
iniconfig @ file:///home/linux1/recipes/ci/iniconfig_1610983019677/work
ipykernel @ file:///Users/runner/miniforge3/conda-bld/ipykernel_1631291142144/work/dist/ipykernel-6.4.1-py3-none-any.whl
ipython @ file:///Users/runner/miniforge3/conda-bld/ipython_1630172558090/work
ipython-genutils==0.2.0
isort==5.10.1
itsdangerous==2.0.1
jedi @ file:///Users/runner/miniforge3/conda-bld/jedi_1610146808808/work
Jinja2 @ file:///home/conda/feedstock_root/build_artifacts/jinja2_1621419064915/work
joblib==1.1.0
json5 @ file:///home/conda/feedstock_root/build_artifacts/json5_1600692310011/work
jsonschema @ file:///home/conda/feedstock_root/build_artifacts/jsonschema_1614815863336/work
jupyter-client @ file:///home/conda/feedstock_root/build_artifacts/jupyter_client_1631893672246/work
jupyter-core @ file:///Users/runner/miniforge3/conda-bld/jupyter_core_1631852747767/work
jupyter-server @ file:///home/conda/feedstock_root/build_artifacts/jupyter_server_1631218027728/work
jupyterlab @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_1632323612590/work
jupyterlab-pygments @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_pygments_1601375948261/work
jupyterlab-server @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_server_1631050937958/work
keyring==23.2.1
kiwisolver @ file:///opt/concourse/worker/volumes/live/f8867fdb-2fa2-4145-73d9-4d6f6dad5f7c/volume/kiwisolver_1612282424136/work
lazy-object-proxy==1.6.0
Mako==1.1.5
Markdown==3.3.4
MarkupSafe @ file:///Users/runner/miniforge3/conda-bld/markupsafe_1621455730541/work
matplotlib @ file:///opt/concourse/worker/volumes/live/ea31f923-d5bd-4af7-6d95-b910b4a0fa5f/volume/matplotlib-suite_1628011455335/work
matplotlib-inline @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-inline_1631080358261/work
mccabe==0.6.1
mistune @ file:///Users/runner/miniforge3/conda-bld/mistune_1624941339085/work
mkl-fft==1.3.0
mkl-random @ file:///opt/concourse/worker/volumes/live/0cda23d8-7460-44b2-7e5d-3c76a8a0ca7e/volume/mkl_random_1626186083266/work
mkl-service==2.4.0
more-itertools @ file:///tmp/build/80754af9/more-itertools_1622818384463/work
munkres==1.1.4
nbclassic @ file:///home/conda/feedstock_root/build_artifacts/nbclassic_1631880505492/work
nbclient @ file:///home/conda/feedstock_root/build_artifacts/nbclient_1629120697898/work
nbconvert @ file:///Users/runner/miniforge3/conda-bld/nbconvert_1631672239611/work
nbformat @ file:///home/conda/feedstock_root/build_artifacts/nbformat_1617383142101/work
nest-asyncio @ file:///home/conda/feedstock_root/build_artifacts/nest-asyncio_1617163391303/work
nodeenv==1.6.0
notebook @ file:///home/conda/feedstock_root/build_artifacts/notebook_1631733685426/work
numpy==1.21.2
oauthlib==3.1.1
olefile @ file:///Users/ktietz/demo/mc3/conda-bld/olefile_1629805411829/work
omegaconf==2.1.1
optuna==2.10.0
packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1625323647219/work
pandas==1.3.3
pandocfilters @ file:///home/conda/feedstock_root/build_artifacts/pandocfilters_1631603243851/work
parso @ file:///home/conda/feedstock_root/build_artifacts/parso_1617148930513/work
path-and-address==2.0.1
patsy==0.5.2
pbr==5.7.0
pexpect @ file:///home/conda/feedstock_root/build_artifacts/pexpect_1602535608087/work
pickleshare @ file:///home/conda/feedstock_root/build_artifacts/pickleshare_1602536217715/work
Pillow @ file:///opt/concourse/worker/volumes/live/140f8506-0993-42b0-466b-ccbbcb5df83e/volume/pillow_1625655825783/work
pip==21.2.4
pkginfo==1.7.1
platformdirs==2.4.0
pluggy==1.0.0
pre-commit==2.15.0
prettytable==2.4.0
prometheus-client @ file:///home/conda/feedstock_root/build_artifacts/prometheus_client_1622586138406/work
prompt-toolkit @ file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1629903925368/work
protobuf==3.19.1
ptyprocess @ file:///home/conda/feedstock_root/build_artifacts/ptyprocess_1609419310487/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
py @ file:///tmp/build/80754af9/py_1607971587848/work
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1593275161868/work
pyfew==0.0.4
Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1629119114968/work
pylint==2.11.1
pyOpenSSL @ file:///home/conda/feedstock_root/build_artifacts/pyopenssl_1608055815057/work
pyparsing==2.4.7
pyperclip==1.8.2
pyrsistent @ file:///Users/runner/miniforge3/conda-bld/pyrsistent_1624984762901/work
PySocks @ file:///Users/runner/miniforge3/conda-bld/pysocks_1610291470465/work
pytest==6.2.5
pytest-datafiles==2.0
python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1626286286081/work
pytz @ file:///home/conda/feedstock_root/build_artifacts/pytz_1612179539967/work
PyYAML==5.4.1
pyzmq @ file:///Users/runner/miniforge3/conda-bld/pyzmq_1631793531263/work
readme-renderer==30.0
requests @ file:///home/conda/feedstock_root/build_artifacts/requests_1626393743643/work
requests-oauthlib==1.3.0
requests-toolbelt==0.9.1
requests-unixsocket==0.2.0
rfc3986==1.5.0
rsa==4.7.2
scikit-learn==1.0
scipy @ file:///opt/concourse/worker/volumes/live/8e481b15-67b8-4160-7a8b-26cd6e8fe54f/volume/scipy_1630606804826/work
seaborn==0.11.2
Send2Trash @ file:///home/conda/feedstock_root/build_artifacts/send2trash_1628511208346/work
setuptools==58.0.4
six @ file:///home/conda/feedstock_root/build_artifacts/six_1620240208055/work
sniffio @ file:///Users/runner/miniforge3/conda-bld/sniffio_1610318427209/work
SQLAlchemy==1.4.26
statsmodels==0.13.0
stevedore==3.5.0
tensorboard==2.7.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.0
terminado @ file:///Users/runner/miniforge3/conda-bld/terminado_1631192978302/work
testpath @ file:///home/conda/feedstock_root/build_artifacts/testpath_1621261527237/work
threadpoolctl==3.0.0
toml @ file:///tmp/build/80754af9/toml_1616166611790/work
torch==1.10.0
torchvision==0.11.1
tornado @ file:///Users/runner/miniforge3/conda-bld/tornado_1625488906146/work
tqdm==4.62.3
traitlets @ file:///home/conda/feedstock_root/build_artifacts/traitlets_1630423529112/work
transforms3d==0.3.1
twine==3.6.0
typing-extensions==3.10.0.2
urllib3 @ file:///home/conda/feedstock_root/build_artifacts/urllib3_1632350318291/work
virtualenv==20.8.1
wcwidth @ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1600965781394/work
webencodings==0.5.1
websocket-client @ file:///Users/runner/miniforge3/conda-bld/websocket-client_1610127894508/work
Werkzeug==2.0.2
wheel==0.37.0
wrapt==1.13.3
zipp @ file:///home/conda/feedstock_root/build_artifacts/zipp_1625284368454/work
zope.interface==5.4.0

Issue description

Tensorboard dev upload command fails with a simple example:

 tensorboard dev upload --logdir weighted_sample_exp

I keep getting the same error which says

E0203 15:37:47.877418 139926947493632 _plugin_wrapping.py:82] AuthMetadataPluginCallback "<tensorboard.uploader.auth.IdTokenAuthMetadataPlugin object at 0x7f435667e2d0>" raised exception!
Traceback (most recent call last):
  File "/home/cxx579/anaconda3/envs/ssw_env/lib/python3.7/site-packages/grpc/_plugin_wrapping.py", line 78, in __call__
    context, _AuthMetadataPluginCallback(callback_state, callback))
  File "/home/cxx579/anaconda3/envs/ssw_env/lib/python3.7/site-packages/tensorboard/uploader/auth.py", line 222, in __call__
    self._request, context.method_name, context.service_url, headers
  File "/home/cxx579/anaconda3/envs/ssw_env/lib/python3.7/site-packages/google/auth/credentials.py", line 133, in before_request
    self.refresh(request)
  File "/home/cxx579/anaconda3/envs/ssw_env/lib/python3.7/site-packages/google/oauth2/credentials.py", line 206, in refresh
    self._scopes,
  File "/home/cxx579/anaconda3/envs/ssw_env/lib/python3.7/site-packages/google/oauth2/_client.py", line 248, in refresh_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/home/cxx579/anaconda3/envs/ssw_env/lib/python3.7/site-packages/google/oauth2/_client.py", line 124, in _token_endpoint_request
    _handle_error_response(response_body)
  File "/home/cxx579/anaconda3/envs/ssw_env/lib/python3.7/site-packages/google/oauth2/_client.py", line 60, in _handle_error_response
    raise exceptions.RefreshError(error_details, response_body)
google.auth.exceptions.RefreshError: ('invalid_grant: Token has been expired or revoked.', '{\n  "error": "invalid_grant",\n  "error_description": "Token has been expired or revoked."\n}')

Seems like something might have gone wrong with the Google API for uploading. This issue can be replicated on both Mac and Linux.

The attached in the log that I wanted to upload.
test.zip

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.