tensorflow / tensorflow/tensorboard

Better error handling for duplicate TB installs

Open
#2,919 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

core:backend type:bug
Dominant language
TypeScript
Stars
7.2k
Forks
1.7k
Avg merge
4d 22h
Merged PRs (30d)
1

Description

Duplicate installations of TensorBoard are pretty easy to encounter in the wild, especially if you install tb-nightly into an environment that already has tensorboard installed, or if you're actually developing TensorBoard and you run a locally built binary within a virtualenv in which TensorFlow was installed, which automatically pulls in a copy of TensorBoard.

Duplicate pip installs are never entirely safe since pip will overwrite files under the "tensorboard" package and result in a mishmash of the two TensorBoard versions, and even a bazel + pip double install is structurally a bit fragile, but in practice it's often been fine. However, the dynamic plugin traversal logic now makes this fail more often and in ways that are pretty confusing.

Failing hard with an explicit error on duplicate installations is one option; it makes local development a bit more annoying (since you have to uninstall the TF-provided TB) but is otherwise a strict improvement relative to the status quo of a non-explicit error. However, if we wanted to return to being robust to multiple installs, we'd have to fix the specific cases of problem:

  1. The TensorBoard pip package currently bundles one dynamic plugin, the projector plugin. Duplicate TensorBoard installs cause an error about duplicate "projector" plugins, as discussed here: https://github.com/tensorflow/tensorboard/issues/2748

    If we wanted to avoid this issue, we could possibly adjust dynamic plugin loading so that plugins bundled with TensorBoard are only found by that TensorBoard instance. Another way to resolve it would be to stop bundling dynamic plugins with TensorBoard at all, and instead release a separate pip package for them. Or we could just allow duplicate plugin registrations and maybe print a warning but then pick only one of the plugins to load, which would handle a more general case of duplication as well.

  2. Calling pkg_resources.iter_entry_points() will fail if any of the TensorBoard installations is in violation of a version constraint from its setup.py. This is perhaps okay if it's the active TensorBoard installation that's in violation (although in practice it might work fine), but when it's actually the old/overwritten TensorBoard installation that's in violation, this is surprising, since the setup.py requirement may be outdated.

    Shown below is a concrete example of the failure mode: removing requests which is only a dependency of the new tb-nightly that is now inactive will cause the active installation to fail at plugin discovery time. As a mitigation we might be able to iterate over the entry points such that we just skip any installations for which their constraints aren't met, rather than failing on them.

$ pip install tb-nightly==2.1.0a20191108
$ pip install tensorboard==2.0.0
$ tensorboard --version
TensorFlow installation not found - running with reduced feature set.
2.0.0
$ pip uninstall requests
$ tensorboard --version
TensorFlow installation not found - running with reduced feature set.
Traceback (most recent call last):
  File "/usr/local/google/home/nickfelt/temp/bin/tensorboard", line 8, in <module>
    sys.exit(run_main())
  File "/usr/local/google/home/nickfelt/temp/local/lib/python2.7/site-packages/tensorboard/main.py", line 58, in run_main
    default.get_plugins() + default.get_dynamic_plugins(),
  File "/usr/local/google/home/nickfelt/temp/local/lib/python2.7/site-packages/tensorboard/default.py", line 110, in get_dynamic_plugins
    for entry_point in pkg_resources.iter_entry_points('tensorboard_plugins')
  File "/usr/local/google/home/nickfelt/temp/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2410, in load
    self.require(*args, **kwargs)
  File "/usr/local/google/home/nickfelt/temp/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2433, in require
    items = working_set.resolve(reqs, env, installer, extras=self.extras)
  File "/usr/local/google/home/nickfelt/temp/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'requests<3,>=2.22.0' distribution was not found and is required by the application

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tensorboard/default.py, especially get_dynamic_plugins(), and reproduce the failure using the pip install and tensorboard --version commands in the issue. Trace pkg_resources.iter_entry_points() and its handling of duplicate installations and unmet constraints. Done means duplicate installs produce explicit, non-confusing behavior according to a chosen approach.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.