alibaba / alibaba/x-deeplearning
tdm tdm_ub_att_ubuntu train error
- Dominant language
- PureBasic
- Stars
- 4.3k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
I built the XDL and the *.json files follows the instruction `https://github.com/alibaba/x-deeplearning/wiki/%E6%B7%B1%E5%BA%A6%E6%A0%91%E5%8C%B9%E9%85%8D%E6%A8%A1%E5%9E%8B(TDM)`
But When I run `python train.py --run_mode=local --config=config.train.json`, the error was
```
if mx.symbol.FullyConnected
[WARNING]: no gradient found for var:bn_1_d_moving_mean under scope: , maybe not used?
[WARNING]: no gradient found for var:bn_1_d_moving_var under scope: , maybe not used?
[WARNING]: no gradient found for var:bn_2_d_moving_mean under scope: , maybe not used?
[WARNING]: no gradient found for var:bn_2_d_moving_var under scope: , maybe not used?
[WARNING]: no gradient found for var:bn_3_d_moving_mean under scope: , maybe not used?
[WARNING]: no gradient found for var:bn_3_d_moving_var under scope: , maybe not used?
>>>>>>>>>>>> 0 >>>>>>>>>>>
Traceback (most recent call last):
File "train.py", line 314, in
train(is_training=True)
File "train.py", line 274, in train
result = sess.run([train_op, xdl.get_collection(xdl.UPDATE_OPS)])
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/training/train_session.py", line 268, in run
return self._session.run(v, run_option, run_statistic, feed_dict=feed_dict)
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/training/train_session.py", line 241, in run
return self._session.run(v, run_option, run_statistic, feed_dict=feed_dict)
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/framework/session.py", line 54, in run
results = execute(run_item, run_option, run_statistic)
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/lib/graph.py", line 305, in execute
return current_graph().execute(outputs, run_option, run_statistic)
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/lib/graph.py", line 177, in execute
result = pybind.execute(self._graph_def, xdl_output_spec, run_option)
RuntimeError: [01:57:37] /usr/local/lib/python2.7/dist-packages/mxnet/cpp-package/include/mxnet-cpp/symbol.hpp:
Exception AttributeError: "'NoneType' object has no attribute 'NNSymbolFree'" in ignored
Exception AttributeError: "'NoneType' object has no attribute 'NNSymbolFree'" in ignored
Exception AttributeError: "'NoneType' object has no attribute 'NNSymbolFree'" in ignored
Exception AttributeError: "'NoneType' object has no attribute 'NNSymbolFree'" in ignored
Exception AttributeError: "'NoneType' object has no attribute 'NNSymbolFree'" in ignored
Exception AttributeError: "'NoneType' object has no attribute 'NNSymbolFree'" in ignored
```
The mock.dat is the file in the repo.
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the failure with `python train.py --run_mode=local --config=config.train.json` using the repository's `mock.dat`. Start at `train.py` lines 274 and 314, then inspect the XDL training-session traceback and the referenced `config.train.json`. Done means the local training run completes without the reported runtime error, with the cause and required change documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100