tensorflow / tensorflow/tensorboard
TensorBoard: gracefully handle deleted event files
Open
@nfelt is already working on this.
Since Dec 4, 2019.
core:backend
core:notf
stat:awaiting tensorflower
theme:usability
- Dominant language
- TypeScript
- Stars
- 7.2k
- Forks
- 1.7k
- Avg merge
- 4d 22h
- Merged PRs (30d)
- 1
Description
as far as i can tell this is exactly the same as this issue https://github.com/tensorflow/tensorflow/issues/3267
if i delete files from the logdir while tensorboard is running i get things like
E0911 11:27:19.441699 139989077399296 plugin_event_multiplexer.py:226] Unable to reload accumulator 'srresnet_voc_2x': [Errno 2] No such file or directory: b'/home/maksim/data/tensorboard/srresnet_voc_2x/events.out.tfevents.1568215513.maksim-desktop.105092.0'
E0911 11:27:24.447706 139989077399296 plugin_event_multiplexer.py:226] Unable to reload accumulator 'srresnet_voc_2x': [Errno 2] No such file or directory: b'/home/maksim/data/tensorboard/srresnet_voc_2x/events.out.tfevents.1568215513.maksim-desktop.105092.0'
E0911 11:27:29.453577 139989077399296 plugin_event_multiplexer.py:226] Unable to reload accumulator 'srresnet_voc_2x': [Errno 2] No such file or directory: b'/home/maksim/data/tensorboard/srresnet_voc_2x/events.out.tfevents.1568215513.maksim-desktop.105092.0'
E0911 11:27:34.459517 139989077399296 plugin_event_multiplexer.py:226] Unable to reload accumulator 'srresnet_voc_2x': [Errno 2] No such file or directory: b'/home/maksim/data/tensorboard/srresnet_voc_2x/events.out.tfevents.1568215513.maksim-desktop.105092.0'
i'm using tb-nightly==1.15.0a20190911 through pytorch.
i'm not sure when reaping is supposed to happen e.g. as in https://github.com/tensorflow/tensorflow/issues/3267#issuecomment-292911229
or how to manually force
WARNING:tensorflow:Deleting accumulator 'run1/test'
WARNING:tensorflow:Deleting accumulator 'run1'
WARNING:tensorflow:Deleting accumulator 'run2/test'
WARNING:tensorflow:Deleting accumulator 'run2'
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.