tensorflow / tensorflow/tensorboard
TensorBoard histograms NaN value handling
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 7.2k
- Forks
- 1.7k
- Avg merge
- 4d 22h
- Merged PRs (30d)
- 1
Description
This issue had been migrated from tensorflow/tensorflow#5276.
Recently I had to fight a lot with exploding gradients and thereby weights turning NaN after only several iterations event hough I was using tf.contrib.layers.variance_scaling_initializer correctly. To monitor the problem I started saving weight histograms which seemed totally messed up. I thought this was due to me using the initializer wrongly and only after some debugging I realised my initialization was totally fine. The NaN values appearing in iteration n seem to mess with the visualization of values in previous iterations which seems like a very misleading behaviour to me. Histograms and distributions should be visualized correctly until this is not really possible because of ill values.
I can easily provide tensorflow summary data to reproduce this behaviour.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no files, tests, or entry points, and no reproducing TensorFlow summary data is attached. Start by obtaining or constructing summary data with NaN histogram values and tracing how TensorBoard loads and renders histograms. Done means later NaN values no longer corrupt the visualization of earlier valid iterations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- data-visualization
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100