alibaba / alibaba/x-deeplearning

调用Save报错

Open
#56 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
PureBasic
Stars
4.3k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

调用Save保存到Hdfs报错,请问原因是什么,报错内容为:
Traceback (most recent call last):
File "deepctr.py", line 74, in
train()
File "deepctr.py", line 54, in train
saver.save(version = checkpoint_version)
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/training/saver.py", line 79, in save
execute(self.save_op(version))
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/lib/graph.py", line 242, in execute
return current_graph().execute(outputs, run_option, run_statistic)
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/lib/graph.py", line 178, in execute
check_error(result.status)
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/lib/error.py", line 33, in check_error
raise PsError(status.msg)
xdl.python.lib.error.PsError: ErrorCode[4], ErrorMsg[The current op is to save checkpoint `10000`, not assigning new op to save checkpoint `10000`
Check Status [PS2XDL::ConvertStatus(st)] at [/tmp/x-deeplearning/xdl/xdl/core/ops/ps_ops/ps_save_op.cc]operator()@40]
2019-01-09 08:42:02,401:7(0x7fd637b82700):ZOO_INFO@zookeeper_close@3222: Closing zookeeper sessionId=0x2666712ffee11c0 to 172.31.39.58:2181

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at deepctr.py lines 54 and 74, then trace saver.py line 79 into ps_save_op.cc line 40 using the reported checkpoint value 10000. Reproduce the Save call and inspect whether the same checkpoint version is being assigned repeatedly; done means identifying the cause and documenting a confirmed fix or workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.