carpedm20 / carpedm20/DCGAN-tensorflow

InvalidArgumentError (see above for traceback): Nan in summary histogram for: HistogramSummary_2 [[Node: HistogramSummary_2 = HistogramSummary[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](HistogramSummary_2/tag, discriminator_1/Sigmoid)]]

Open
#398 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
7.2k
Forks
2.6k
PR merge metrics
No merged PRs in 30d

Description

when I trying to train a model,it only run for 940 batchs,and i got an error.
`Traceback (most recent call last):
File "main.py", line 147, in
tf.app.run()
File "/Users/xiaochenyue/opt/anaconda3/envs/py33/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 43, in run
sys.exit(main(sys.argv[:1] + flags_passthrough))
File "main.py", line 120, in main
dcgan.train(FLAGS)
File "/Users/xiaochenyue/Desktop/zjy/DCGAN-tensorflow/model.py", line 284, in train
feed_dict={ self.z: batch_z })
File "/Users/xiaochenyue/opt/anaconda3/envs/py33/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 766, in run
run_metadata_ptr)
File "/Users/xiaochenyue/opt/anaconda3/envs/py33/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 964, in _run
feed_dict_string, options, run_metadata)
File "/Users/xiaochenyue/opt/anaconda3/envs/py33/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1014, in _do_run
target_list, options, run_metadata)
File "/Users/xiaochenyue/opt/anaconda3/envs/py33/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1034, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Nan in summary histogram for: HistogramSummary_2
[[Node: HistogramSummary_2 = HistogramSummary[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](HistogramSummary_2/tag, discriminator_1/Sigmoid)]]

Caused by op u'HistogramSummary_2', defined at:
File "main.py", line 147, in
tf.app.run()
File "/Users/xiaochenyue/opt/anaconda3/envs/py33/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 43, in run
sys.exit(main(sys.argv[:1] + flags_passthrough))
File "main.py", line 115, in main
max_to_keep=FLAGS.max_to_keep)
File "/Users/xiaochenyue/Desktop/zjy/DCGAN-tensorflow/model.py", line 104, in __init__
self.build_model()
File "/Users/xiaochenyue/Desktop/zjy/DCGAN-tensorflow/model.py", line 132, in build_model
self.d__sum = histogram_summary("d_", self.D_)
File "/Users/xiaochenyue/opt/anaconda3/envs/py33/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 110, in new_func
return func(*args, **kwargs)
File "/Users/xiaochenyue/opt/anaconda3/envs/py33/lib/python2.7/site-packages/tensorflow/python/ops/logging_ops.py", line 101, in histogram_summary
tag=tag, values=values, name=scope)
File "/Users/xiaochenyue/opt/anaconda3/envs/py33/lib/python2.7/site-packages/tensorflow/python/ops/gen_logging_ops.py", line 139, in _histogram_summary
name=name)
File "/Users/xiaochenyue/opt/anaconda3/envs/py33/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 759, in apply_op
op_def=op_def)
File "/Users/xiaochenyue/opt/anaconda3/envs/py33/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2240, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/Users/xiaochenyue/opt/anaconda3/envs/py33/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1128, in __init__
self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): Nan in summary histogram for: HistogramSummary_2
[[Node: HistogramSummary_2 = HistogramSummary[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](HistogramSummary_2/tag, discriminator_1/Sigmoid)]]`
when I rerun it,it will run for a while,and raise same error.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with main.py lines 120 and 147 and model.py lines 104, 132, and 284, then reproduce the failure near batch 940. Trace the values reaching discriminator_1/Sigmoid and HistogramSummary_2; done means identifying the source of the NaN and confirming whether the failure can be reliably reproduced and resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.