tensorflow / tensorflow/tensorboard
Support multiple event files that contain events not in chronological order
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 7.2k
- Forks
- 1.7k
- Avg merge
- 4d 22h
- Merged PRs (30d)
- 1
Description
Use case
Sometimes, a model can be iteratively trained on multiple workers (e.g., using mutliprocessing). In that case, there is an tfevent_* file for each of the workers. So when the model is trained on worker_1, the events are persisted in the tfevent_1 file. Then, the model training can be dedicated to a different worker (worker_2) and the events are persisted in the tfevent_2 file. For next iterations, the model can be trained on worker_1 again, which means that the tfevent_1 is used.
The issue is that the we end up having tfevent files that contain events in non-chronological order:
tfevent_1 file:
- event 1
- event 3
tfevent_2 file:
- event 2
And rustboard fails to read them properly.
This is not a bug since this is a documented behaviour: https://github.com/tensorflow/tensorboard/blob/f45fdd667a63d0e6b4aac471556583e5f2b7dac2/tensorboard/data/server/logdir.rs#L36
Feature request
Allow rustboard to parse data in partially-chronological order.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the documented event-ordering behavior in tensorboard/data/server/logdir.rs and then inspect rustboard's event-file parsing path. Reproduce the worker-file ordering described in the issue; done means rustboard can parse multiple event files whose events are only partially chronological.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-visualization
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100