tensorflow / tensorflow/tensorboard

TensorBoard: gracefully handle deleted event files

Open
#2,634 11 comments 2 reactions 1 assignee View on GitHub

@nfelt is already working on this.

Since Dec 4, 2019.

core:backend core:notf stat:awaiting tensorflower theme:usability
Dominant language
TypeScript
Stars
7.2k
Forks
1.7k
Avg merge
4d 22h
Merged PRs (30d)
1

Description

as far as i can tell this is exactly the same as this issue https://github.com/tensorflow/tensorflow/issues/3267

if i delete files from the logdir while tensorboard is running i get things like

E0911 11:27:19.441699 139989077399296 plugin_event_multiplexer.py:226] Unable to reload accumulator 'srresnet_voc_2x': [Errno 2] No such file or directory: b'/home/maksim/data/tensorboard/srresnet_voc_2x/events.out.tfevents.1568215513.maksim-desktop.105092.0'
E0911 11:27:24.447706 139989077399296 plugin_event_multiplexer.py:226] Unable to reload accumulator 'srresnet_voc_2x': [Errno 2] No such file or directory: b'/home/maksim/data/tensorboard/srresnet_voc_2x/events.out.tfevents.1568215513.maksim-desktop.105092.0'
E0911 11:27:29.453577 139989077399296 plugin_event_multiplexer.py:226] Unable to reload accumulator 'srresnet_voc_2x': [Errno 2] No such file or directory: b'/home/maksim/data/tensorboard/srresnet_voc_2x/events.out.tfevents.1568215513.maksim-desktop.105092.0'
E0911 11:27:34.459517 139989077399296 plugin_event_multiplexer.py:226] Unable to reload accumulator 'srresnet_voc_2x': [Errno 2] No such file or directory: b'/home/maksim/data/tensorboard/srresnet_voc_2x/events.out.tfevents.1568215513.maksim-desktop.105092.0'

i'm using tb-nightly==1.15.0a20190911 through pytorch.

i'm not sure when reaping is supposed to happen e.g. as in https://github.com/tensorflow/tensorflow/issues/3267#issuecomment-292911229
or how to manually force

WARNING:tensorflow:Deleting accumulator 'run1/test'
WARNING:tensorflow:Deleting accumulator 'run1'
WARNING:tensorflow:Deleting accumulator 'run2/test'
WARNING:tensorflow:Deleting accumulator 'run2'

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.