tensorflow / tensorflow/tensorboard

Tensorboard frontend does not update when the log directory changes.

Open
#2,690 13 comments 2 reactions 1 assignee View on GitHub

Nobody has claimed this yet.

core:frontend stat:awaiting tensorflower type:bug
Dominant language
TypeScript
Stars
7.2k
Forks
1.7k
Avg merge
4d 22h
Merged PRs (30d)
1

Description

Environment information (required)

Diagnostics
Diagnostics output
--- check: autoidentify
INFO: diagnose_tensorboard.py version b5843ba83bb708385ff54baaab4b2c70c39f7a4f

--- check: general
INFO: sys.version_info: sys.version_info(major=3, minor=7, micro=4, releaselevel='final', serial=0)
INFO: os.name: posix
INFO: os.uname(): posix.uname_result(sysname='Linux', nodename='c8278555cf9a', release='4.19.69-1-MANJARO', version='#1 SMP PREEMPT Thu Aug 29 08:51:46 UTC 2019', machine='x86_64')
INFO: sys.getwindowsversion(): N/A

--- check: package_management
INFO: has conda-meta: False
INFO: $VIRTUAL_ENV: None

--- check: installed_packages
INFO: installed: tensorboard==1.14.0
diagnose_tensorboard.py:197: DeprecationWarning: The 'warn' function is deprecated, use 'warning' instead
  logging.warn("no installation among: %s", sorted(family))
WARNING: no installation among: ['tensorflow', 'tensorflow-gpu', 'tf-nightly', 'tf-nightly-2.0-preview', 'tf-nightly-gpu', 'tf-nightly-gpu-2.0-preview']
WARNING: no installation among: ['tensorflow-estimator', 'tensorflow-estimator-2.0-preview', 'tf-estimator-nightly']

--- check: tensorboard_python_version
INFO: tensorboard.version.VERSION: '1.14.0'

--- check: tensorflow_python_version
Traceback (most recent call last):
  File "diagnose_tensorboard.py", line 419, in main
    suggestions.extend(check())
  File "diagnose_tensorboard.py", line 77, in wrapper
    result = fn()
  File "diagnose_tensorboard.py", line 236, in tensorflow_python_version
    import tensorflow as tf
ModuleNotFoundError: No module named 'tensorflow'

--- check: tensorboard_binary_path
INFO: which tensorboard: b'/usr/local/bin/tensorboard\n'

--- check: readable_fqdn
INFO: socket.getfqdn(): 'c8278555cf9a'

--- check: stat_tensorboardinfo
INFO: directory: /tmp/.tensorboard-info
INFO: os.stat(...): os.stat_result(st_mode=16895, st_ino=19925017, st_dev=48, st_nlink=2, st_uid=0, st_gid=0, st_size=4096, st_atime=1569252142, st_mtime=1569252142, st_ctime=1569252142)
INFO: mode: 0o40777

--- check: source_trees_without_genfiles
INFO: tensorboard_roots (1): ['/usr/local/lib/python3.7/site-packages']; bad_roots (0): []

--- check: full_pip_freeze
INFO: pip freeze --all:
absl-py==0.8.0
grpcio==1.23.0
Markdown==3.1.1
numpy==1.17.2
pip==19.2.3
protobuf==3.9.1
setuptools==41.2.0
six==1.12.0
tensorboard==1.14.0
Werkzeug==0.16.0
wheel==0.33.6

Issue description

Please describe the bug as clearly as possible. How can we reproduce the
problem without additional resources (including external data files and
proprietary Python modules)?

I am trying to run the tensorboard server in a docker container but there seems to be problems with updating the log directory. Tensorboard will load a single file and then be stuck in that state even if I manually go into the container and delete the files. For example, I have 4 folders each with a tfevents file and this is what the resulting frontend is stuck at:

image

An exception is raised in the Reloader thread during spin up of the server which seems related to the issue. Output of docker run:

/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/usr/local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
TensorFlow installation not found - running with reduced feature set.
Exception in thread Reloader:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.7/site-packages/tensorboard/backend/application.py", line 430, in _reload
    multiplexer.Reload()
  File "/usr/local/lib/python3.7/site-packages/tensorboard/backend/event_processing/plugin_event_multiplexer.py", line 240, in Reload
    Worker()
  File "/usr/local/lib/python3.7/site-packages/tensorboard/backend/event_processing/plugin_event_multiplexer.py", line 218, in Worker
    accumulator.Reload()
  File "/usr/local/lib/python3.7/site-packages/tensorboard/backend/event_processing/plugin_event_accumulator.py", line 177, in Reload
    for event in self._generator.Load():
  File "/usr/local/lib/python3.7/site-packages/tensorboard/backend/event_processing/directory_watcher.py", line 89, in Load
    for event in self._LoadInternal():
  File "/usr/local/lib/python3.7/site-packages/tensorboard/backend/event_processing/directory_watcher.py", line 113, in _LoadInternal
    for event in self._loader.Load():
  File "/usr/local/lib/python3.7/site-packages/tensorboard/backend/event_processing/event_file_loader.py", line 95, in Load
    yield event_pb2.Event.FromString(record)
google.protobuf.message.DecodeError: Error parsing message

TensorBoard 1.14.0 at http://c8278555cf9a:6006/ (Press CTRL+C to quit)

If i use the official tensorflow image everything works fine. However, the problem with that solution is that the image is too big at 1.2 gigs. I really just need a barebones tensorboard.

Steps to reproduce:

  1. Use this Dockerfile:
FROM python:3.7-slim

RUN pip install --upgrade pip
RUN pip install tensorboard==1.14

COPY diagnose_tensorboard.py .

VOLUME /logs

EXPOSE 6006

ENTRYPOINT tensorboard --logdir /logs

  1. Build the image:
    docker build -t my-tb .

  2. Run a container with mounted tfevent files:
    docker run -it --rm -p 6006:6006 --name tb_test -v $(realpath tb_files):/logs my-tb /bin/bash -c "tensorboard --logdir /logs"

  3. Go to localhost:6006

  4. Observe issue. Try deleting tfevent files in the container and notice that the frontend never updates.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.