tensorflow / tensorflow/tensorboard

"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc", "failed to connect to all addresses"

Open
#5,384 5 comments 0 reactions 1 assignee View on GitHub

@bileschi is already working on this.

Since Jan 13, 2022.

stat:awaiting tensorflower type:bug
Dominant language
TypeScript
Stars
7.2k
Forks
1.7k
Avg merge
4d 22h
Merged PRs (30d)
1

Description

Consider Stack Overflow for getting support using TensorBoard—they have
a larger community with better searchability:

https://stackoverflow.com/questions/tagged/tensorboard

Do not use this template for for setup, installation, or configuration
issues. Instead, use the “installation problem” issue template:

https://github.com/tensorflow/tensorboard/issues/new?template=installation_problem.md

To report a problem with TensorBoard itself, please fill out the
remainder of this template.

Environment information (required)

Please run diagnose_tensorboard.py (link below) in the same
environment from which you normally run TensorFlow/TensorBoard, and
paste the output here:

Diagnostics
Diagnostics output 
--- check: autoidentify
INFO: diagnose_tensorboard.py version e43767ef2b648d0d5d57c00f38ccbd38390e38da

--- check: general
INFO: sys.version_info: sys.version_info(major=3, minor=8, micro=10, releaselevel='final', serial=0)
INFO: os.name: posix
INFO: os.uname(): posix.uname_result(sysname='Linux', nodename='43bf443dec00', release='5.8.0-1036-azure', version='#38~20.04.1-Ubuntu SMP Thu Jun 17 14:14:18 UTC 2021', machine='x86_64')
INFO: sys.getwindowsversion(): N/A

--- check: package_management
INFO: has conda-meta: False
INFO: $VIRTUAL_ENV: None

--- check: installed_packages
INFO: installed: tensorboard==2.6.0
WARNING: no installation among: ['tensorflow', 'tensorflow-gpu', 'tf-nightly', 'tf-nightly-2.0-preview', 'tf-nightly-gpu', 'tf-nightly-gpu-2.0-preview']
INFO: installed: tensorflow-estimator==2.6.0
INFO: installed: tensorboard-data-server==0.6.1

--- check: tensorboard_python_version
INFO: tensorboard.version.VERSION: '2.6.0'

--- check: tensorflow_python_version
INFO: tensorflow.__version__: '2.7.0-rc0'
INFO: tensorflow.__git_version__: 'v1.12.1-64316-gce35e5c3a8e'

--- check: tensorboard_data_server_version
INFO: data server binary: '/usr/local/lib/python3.8/dist-packages/tensorboard_data_server/bin/server'
INFO: data server binary version: b'rustboard 0.6.1'

--- check: tensorboard_binary_path
INFO: which tensorboard: b'/usr/local/bin/tensorboard\n'

--- check: addrinfos
socket.has_ipv6 = True
socket.AF_UNSPEC = <AddressFamily.AF_UNSPEC: 0>
socket.SOCK_STREAM = <SocketKind.SOCK_STREAM: 1>
socket.AI_ADDRCONFIG = <AddressInfo.AI_ADDRCONFIG: 32>
socket.AI_PASSIVE = <AddressInfo.AI_PASSIVE: 1>
Loopback flags: <AddressInfo.AI_ADDRCONFIG: 32>
Loopback infos: [(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 0))]
Wildcard flags: <AddressInfo.AI_PASSIVE: 1>
Wildcard infos: [(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('0.0.0.0', 0)), (<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('::', 0, 0, 0))]

--- check: readable_fqdn
INFO: socket.getfqdn(): '43bf443dec00'

--- check: stat_tensorboardinfo
INFO: directory: /tmp/.tensorboard-info
INFO: os.stat(...): os.stat_result(st_mode=16895, st_ino=1067657, st_dev=54, st_nlink=2, st_uid=0, st_gid=0, st_size=4096, st_atime=1634840828, st_mtime=1634840828, st_ctime=1634840828)
INFO: mode: 0o40777

--- check: source_trees_without_genfiles
INFO: tensorboard_roots (1): ['/usr/local/lib/python3.8/dist-packages']; bad_roots (0): []

--- check: full_pip_freeze
INFO: pip freeze --all:
absl-py==0.14.1
astunparse==1.6.3
cachetools==4.2.4
certifi==2021.10.8
charset-normalizer==2.0.6
flatbuffers==2.0
gast==0.4.0
google-auth==1.35.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.41.0
h5py==3.1.0
idna==3.2
keras==2.6.0
Keras-Preprocessing==1.1.2
libclang==11.1.0
Markdown==3.3.4
numpy==1.19.5
oauthlib==3.1.1
opt-einsum==3.3.0
pip==20.2.4
protobuf==3.18.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
requests==2.26.0
requests-oauthlib==1.3.0
rsa==4.7.2
setuptools==58.2.0
six==1.15.0
tensorboard==2.6.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.0
tensorflow-cpu==2.7.0rc0
tensorflow-estimator==2.6.0
tensorflow-io-gcs-filesystem==0.21.0
termcolor==1.1.0
typing-extensions==3.7.4.3
urllib3==1.26.7
Werkzeug==2.0.2
wheel==0.37.0
wrapt==1.12.1

https://raw.githubusercontent.com/tensorflow/tensorboard/master/tensorboard/tools/diagnose_tensorboard.py

For browser-related issues, please additionally specify:

  • Browser type and version (e.g., Chrome 64.0.3282.140):
  • Screenshot, if it’s a visual issue:

Issue description

The tensorboard fails with Data could not be loaded. error message after loading data for the first few requests. It was working for few months, something must have changed to cause this error, but its unclear how to find the root cause of this issue. Please help.

Error
Error output 

I1021 18:27:40.527237 140419692422912 timing.py:122] Thread-26[7fb601e2e700] LEAVE GrpcDataProvider.read_scalars - 0.004939s elapsed
I1021 18:27:40.528655 140419692422912 _internal.py:225] 172.17.0.1 - - [21/Oct/2021 18:27:40] "POST /data/plugin/scalars/scalars_multirun HTTP/1.0" 200 -
I1021 18:27:40.570473 140419683768064 timing.py:122] Thread-27[7fb6015ed700]   LEAVE _stub.ReadScalars - 0.044712s elapsed
I1021 18:27:40.570747 140419683768064 timing.py:122] Thread-27[7fb6015ed700]   ENTER build result
I1021 18:27:40.571971 140419683768064 timing.py:122] Thread-27[7fb6015ed700]   LEAVE build result - 0.001219s elapsed
I1021 18:27:40.572121 140419683768064 timing.py:122] Thread-27[7fb6015ed700] LEAVE GrpcDataProvider.read_scalars - 0.047378s elapsed
I1021 18:27:40.577262 140419683768064 _internal.py:225] 172.17.0.1 - - [21/Oct/2021 18:27:40] "POST /data/plugin/scalars/scalars_multirun HTTP/1.0" 200 -
I1021 18:28:29.129058 140419683768064 _internal.py:225] 172.17.0.1 - - [21/Oct/2021 18:28:29] "GET / HTTP/1.0" 200 -
I1021 18:28:29.709144 140419683768064 _internal.py:225] 172.17.0.1 - - [21/Oct/2021 18:28:29] "GET /index.js?_file_hash=8f39b027 HTTP/1.0" 200 -
I1021 18:28:33.277612 140419683768064 _internal.py:225] 172.17.0.1 - - [21/Oct/2021 18:28:33] "GET /icon_bundle.svg HTTP/1.0" 200 -
I1021 18:28:33.374844 140419700815616 _internal.py:225] 172.17.0.1 - - [21/Oct/2021 18:28:33] "GET /data/environment HTTP/1.0" 500 -
I1021 18:28:33.376072 140419692422912 _internal.py:225] 172.17.0.1 - - [21/Oct/2021 18:28:33] "GET /data/runs HTTP/1.0" 500 -
I1021 18:28:33.376559 140419675113216 _internal.py:225] 172.17.0.1 - - [21/Oct/2021 18:28:33] "GET /data/environment HTTP/1.0" 500 -
I1021 18:28:33.376826 140419683768064 _internal.py:225] 172.17.0.1 - - [21/Oct/2021 18:28:33] "GET /data/plugins_listing HTTP/1.0" 500 -
E1021 18:28:33.389955 140419692422912 _internal.py:225] Error on request:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/werkzeug/serving.py", line 319, in run_wsgi
    execute(self.server.app)
  File "/usr/local/lib/python3.8/dist-packages/werkzeug/serving.py", line 308, in execute
    application_iter = app(environ, start_response)
  File "/usr/local/lib/python3.8/dist-packages/tensorboard/backend/application.py", line 525, in __call__
    return self._app(environ, start_response)
  File "/usr/local/lib/python3.8/dist-packages/tensorboard/backend/application.py", line 566, in wrapper
    return wsgi_app(environ, start_response)
  File "/usr/local/lib/python3.8/dist-packages/tensorboard/backend/security_validator.py", line 77, in __call__
    return self._application(environ, start_response_proxy)
  File "/usr/local/lib/python3.8/dist-packages/tensorboard/backend/path_prefix.py", line 68, in __call__
    return self._application(environ, start_response)
  File "/usr/local/lib/python3.8/dist-packages/tensorboard/backend/experiment_id.py", line 73, in __call__
    return self._application(environ, start_response)
  File "/usr/local/lib/python3.8/dist-packages/tensorboard/backend/empty_path_redirect.py", line 43, in __call__
    return self._application(environ, start_response)
  File "/usr/local/lib/python3.8/dist-packages/tensorboard/backend/application.py", line 589, in wrapper
    return wsgi_app(environ, start_response)
  File "/usr/local/lib/python3.8/dist-packages/tensorboard/backend/application.py", line 548, in _route_request
    return self.exact_routes[clean_path](environ, start_response)
  File "/usr/local/lib/python3.8/dist-packages/werkzeug/wrappers/request.py", line 206, in application
    resp = f(*args[:-2] + (request,))
  File "/usr/local/lib/python3.8/dist-packages/tensorboard/plugins/core/core_plugin.py", line 253, in _serve_runs
    self._data_provider.list_runs(ctx, experiment_id=experiment),
  File "/usr/local/lib/python3.8/dist-packages/tensorboard/data/grpc_provider.py", line 76, in list_runs
    res = self._stub.ListRuns(req)
  File "/usr/local/lib/python3.8/dist-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.8/dist-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses"
        debug_error_string = "{"created":"@1634840913.346693130","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3158,"referenced_errors":[{"created":"@1634840913.346692130","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":147,"grpc_status":14}]}"
>
Dockerfile to create the tensorboard
Error output 
FROM tensorflow/tensorflow:2.7.0rc0
CMD tensorboard --logdir /tf_logs/ --host 0.0.0.0 --verbosity 0

Related issues, https://github.com/tensorflow/tensorboard/issues/2601#

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.