tensorflow / tensorflow/tensorboard

Support multiple event files that contain events not in chronological order

Open
#5,958 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

core:rustboard core:summaries type:bug type:feature type:support
Dominant language
TypeScript
Stars
7.2k
Forks
1.7k
Avg merge
4d 22h
Merged PRs (30d)
1

Description

Use case

Sometimes, a model can be iteratively trained on multiple workers (e.g., using mutliprocessing). In that case, there is an tfevent_* file for each of the workers. So when the model is trained on worker_1, the events are persisted in the tfevent_1 file. Then, the model training can be dedicated to a different worker (worker_2) and the events are persisted in the tfevent_2 file. For next iterations, the model can be trained on worker_1 again, which means that the tfevent_1 is used.

The issue is that the we end up having tfevent files that contain events in non-chronological order:

tfevent_1 file:

  • event 1
  • event 3

tfevent_2 file:

  • event 2

And rustboard fails to read them properly.

This is not a bug since this is a documented behaviour: https://github.com/tensorflow/tensorboard/blob/f45fdd667a63d0e6b4aac471556583e5f2b7dac2/tensorboard/data/server/logdir.rs#L36

Feature request

Allow rustboard to parse data in partially-chronological order.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the documented event-ordering behavior in tensorboard/data/server/logdir.rs and then inspect rustboard's event-file parsing path. Reproduce the worker-file ordering described in the issue; done means rustboard can parse multiple event files whose events are only partially chronological.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-visualization
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.