alibaba / alibaba/x-deeplearning

tdm tdm_ub_att_ubuntu train error

Open
#250 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
PureBasic
Stars
4.3k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

I built the XDL and the *.json files follows the instruction `https://github.com/alibaba/x-deeplearning/wiki/%E6%B7%B1%E5%BA%A6%E6%A0%91%E5%8C%B9%E9%85%8D%E6%A8%A1%E5%9E%8B(TDM)`
But When I run `python train.py --run_mode=local --config=config.train.json`, the error was
```
if mx.symbol.FullyConnected
[WARNING]: no gradient found for var:bn_1_d_moving_mean under scope: , maybe not used?
[WARNING]: no gradient found for var:bn_1_d_moving_var under scope: , maybe not used?
[WARNING]: no gradient found for var:bn_2_d_moving_mean under scope: , maybe not used?
[WARNING]: no gradient found for var:bn_2_d_moving_var under scope: , maybe not used?
[WARNING]: no gradient found for var:bn_3_d_moving_mean under scope: , maybe not used?
[WARNING]: no gradient found for var:bn_3_d_moving_var under scope: , maybe not used?
>>>>>>>>>>>> 0 >>>>>>>>>>>
Traceback (most recent call last):
File "train.py", line 314, in
train(is_training=True)
File "train.py", line 274, in train
result = sess.run([train_op, xdl.get_collection(xdl.UPDATE_OPS)])
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/training/train_session.py", line 268, in run
return self._session.run(v, run_option, run_statistic, feed_dict=feed_dict)
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/training/train_session.py", line 241, in run
return self._session.run(v, run_option, run_statistic, feed_dict=feed_dict)
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/framework/session.py", line 54, in run
results = execute(run_item, run_option, run_statistic)
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/lib/graph.py", line 305, in execute
return current_graph().execute(outputs, run_option, run_statistic)
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/lib/graph.py", line 177, in execute
result = pybind.execute(self._graph_def, xdl_output_spec, run_option)
RuntimeError: [01:57:37] /usr/local/lib/python2.7/dist-packages/mxnet/cpp-package/include/mxnet-cpp/symbol.hpp:
Exception AttributeError: "'NoneType' object has no attribute 'NNSymbolFree'" in ignored
Exception AttributeError: "'NoneType' object has no attribute 'NNSymbolFree'" in ignored
Exception AttributeError: "'NoneType' object has no attribute 'NNSymbolFree'" in ignored
Exception AttributeError: "'NoneType' object has no attribute 'NNSymbolFree'" in ignored
Exception AttributeError: "'NoneType' object has no attribute 'NNSymbolFree'" in ignored
Exception AttributeError: "'NoneType' object has no attribute 'NNSymbolFree'" in ignored
```
The mock.dat is the file in the repo.

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the failure with `python train.py --run_mode=local --config=config.train.json` using the repository's `mock.dat`. Start at `train.py` lines 274 and 314, then inspect the XDL training-session traceback and the referenced `config.train.json`. Done means the local training run completes without the reported runtime error, with the cause and required change documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.