tensorflow / tensorflow/tensorboard
Allow async flush in EventFileWriter creation
@wchargin is already working on this.
Since Aug 17, 2020.
- Dominant language
- TypeScript
- Stars
- 7.2k
- Forks
- 1.7k
- Avg merge
- 4d 22h
- Merged PRs (30d)
- 1
Description
Issue
In TensorBoard logging, by default flush is done asynchronously
https://github.com/tensorflow/tensorboard/blob/67f8ab9c1583a9a14e6c52b2b30fb62234c4afe0/tensorboard/summary/writer/event_file_writer.py#L160-L162
which is nice not only because of perf win, but also people can rely on that any TensorBoard failure will never fail training jobs.
But there's one synchronous flush called from EventFileWriter creation:
which will be called from SummaryWriter creation as well.
This has caused a SEV for us recently, when a filesystem failure caused a lot of training workflows to fail, even when some workflow only created SummaryWriter without explicitly logging anything. People are surprised to find out that:
- Logging code can throw unhandled error
- There're IO in python class initialization
Question
- What's the motivation to have this explicit flush call, instead of relying on existing implicit flushing?
- What's the risk of bypassing this call (maybe via a flag), I'm not sure how the data (writer creation timestamp) is used in frontend and how bad it is if it's missing
Another related problem is, we have many clients that'll create SummaryWriter with same logdir multiple times (e.g. in every logging call or in every trainer), and with this flush callsite, we'll end up creating a lot of empty files even when nothing is eventually logged, which can slow down data loading. So if this timestamp data turns out to be not essential, then we have big incentive to bypass it completely.
Many thanks!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.