tensorflow / tensorflow/tensorboard
Better error handling for duplicate TB installs
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 7.2k
- Forks
- 1.7k
- Avg merge
- 4d 22h
- Merged PRs (30d)
- 1
Description
Duplicate installations of TensorBoard are pretty easy to encounter in the wild, especially if you install tb-nightly into an environment that already has tensorboard installed, or if you're actually developing TensorBoard and you run a locally built binary within a virtualenv in which TensorFlow was installed, which automatically pulls in a copy of TensorBoard.
Duplicate pip installs are never entirely safe since pip will overwrite files under the "tensorboard" package and result in a mishmash of the two TensorBoard versions, and even a bazel + pip double install is structurally a bit fragile, but in practice it's often been fine. However, the dynamic plugin traversal logic now makes this fail more often and in ways that are pretty confusing.
Failing hard with an explicit error on duplicate installations is one option; it makes local development a bit more annoying (since you have to uninstall the TF-provided TB) but is otherwise a strict improvement relative to the status quo of a non-explicit error. However, if we wanted to return to being robust to multiple installs, we'd have to fix the specific cases of problem:
-
The TensorBoard pip package currently bundles one dynamic plugin, the projector plugin. Duplicate TensorBoard installs cause an error about duplicate "projector" plugins, as discussed here: https://github.com/tensorflow/tensorboard/issues/2748
If we wanted to avoid this issue, we could possibly adjust dynamic plugin loading so that plugins bundled with TensorBoard are only found by that TensorBoard instance. Another way to resolve it would be to stop bundling dynamic plugins with TensorBoard at all, and instead release a separate pip package for them. Or we could just allow duplicate plugin registrations and maybe print a warning but then pick only one of the plugins to load, which would handle a more general case of duplication as well.
-
Calling
pkg_resources.iter_entry_points()will fail if any of the TensorBoard installations is in violation of a version constraint from itssetup.py. This is perhaps okay if it's the active TensorBoard installation that's in violation (although in practice it might work fine), but when it's actually the old/overwritten TensorBoard installation that's in violation, this is surprising, since thesetup.pyrequirement may be outdated.Shown below is a concrete example of the failure mode: removing
requestswhich is only a dependency of the newtb-nightlythat is now inactive will cause the active installation to fail at plugin discovery time. As a mitigation we might be able to iterate over the entry points such that we just skip any installations for which their constraints aren't met, rather than failing on them.
$ pip install tb-nightly==2.1.0a20191108
$ pip install tensorboard==2.0.0
$ tensorboard --version
TensorFlow installation not found - running with reduced feature set.
2.0.0
$ pip uninstall requests
$ tensorboard --version
TensorFlow installation not found - running with reduced feature set.
Traceback (most recent call last):
File "/usr/local/google/home/nickfelt/temp/bin/tensorboard", line 8, in <module>
sys.exit(run_main())
File "/usr/local/google/home/nickfelt/temp/local/lib/python2.7/site-packages/tensorboard/main.py", line 58, in run_main
default.get_plugins() + default.get_dynamic_plugins(),
File "/usr/local/google/home/nickfelt/temp/local/lib/python2.7/site-packages/tensorboard/default.py", line 110, in get_dynamic_plugins
for entry_point in pkg_resources.iter_entry_points('tensorboard_plugins')
File "/usr/local/google/home/nickfelt/temp/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2410, in load
self.require(*args, **kwargs)
File "/usr/local/google/home/nickfelt/temp/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2433, in require
items = working_set.resolve(reqs, env, installer, extras=self.extras)
File "/usr/local/google/home/nickfelt/temp/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 786, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'requests<3,>=2.22.0' distribution was not found and is required by the application
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with tensorboard/default.py, especially get_dynamic_plugins(), and reproduce the failure using the pip install and tensorboard --version commands in the issue. Trace pkg_resources.iter_entry_points() and its handling of duplicate installations and unmet constraints. Done means duplicate installs produce explicit, non-confusing behavior according to a chosen approach.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100