tensorflow / tensorflow/tensorboard
Restarting training job doesn't visualize earlier log files
Open
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 7.2k
- Forks
- 1.7k
- Avg merge
- 4d 22h
- Merged PRs (30d)
- 1
Description
I have a training job on a spot instance and a TB log on GCS-CSI fuse.
As it is a spot instance when the job it is going to reastart it going to create a new log file.
If I don't restart the TB the curves are ok but when I restart the TB service it going to read only then last log file.
Is there a ways to workaround this?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported spot-instance restart with TensorBoard and a GCS-CSI Fuse log directory, then check which log files are loaded after the service restarts. Done means earlier log files remain available in the visualization alongside logs created after the restart.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud
- Domain
- cloud, data-visualization
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100