tensorflow / tensorflow/tensorboard

Allow async flush in EventFileWriter creation

Open
#4,036 2 comments 0 reactions 1 assignee View on GitHub

@wchargin is already working on this.

Since Aug 17, 2020.

core:notf
Dominant language
TypeScript
Stars
7.2k
Forks
1.7k
Avg merge
4d 22h
Merged PRs (30d)
1

Description

Issue

In TensorBoard logging, by default flush is done asynchronously
https://github.com/tensorflow/tensorboard/blob/67f8ab9c1583a9a14e6c52b2b30fb62234c4afe0/tensorboard/summary/writer/event_file_writer.py#L160-L162

which is nice not only because of perf win, but also people can rely on that any TensorBoard failure will never fail training jobs.

But there's one synchronous flush called from EventFileWriter creation:

https://github.com/tensorflow/tensorboard/blob/67f8ab9c1583a9a14e6c52b2b30fb62234c4afe0/tensorboard/summary/writer/event_file_writer.py#L101

which will be called from SummaryWriter creation as well.

This has caused a SEV for us recently, when a filesystem failure caused a lot of training workflows to fail, even when some workflow only created SummaryWriter without explicitly logging anything. People are surprised to find out that:

  • Logging code can throw unhandled error
  • There're IO in python class initialization

Question

  • What's the motivation to have this explicit flush call, instead of relying on existing implicit flushing?
  • What's the risk of bypassing this call (maybe via a flag), I'm not sure how the data (writer creation timestamp) is used in frontend and how bad it is if it's missing

Another related problem is, we have many clients that'll create SummaryWriter with same logdir multiple times (e.g. in every logging call or in every trainer), and with this flush callsite, we'll end up creating a lot of empty files even when nothing is eventually logged, which can slow down data loading. So if this timestamp data turns out to be not essential, then we have big incentive to bypass it completely.

Many thanks!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.