alibaba / alibaba/x-deeplearning

从checkpoint里恢复后,再次保存checkpoint时报错

Open
#126 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
PureBasic
Stars
4.3k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

`

def test_all(self):

var = xdl.Variable(name="w", dtype=DataType.int32, shape=[4], initializer=xdl.Zeros())
execute(xdl.variable_registers())
execute(xdl.global_initializers())
save_op = xdl.ps_save_op(ckpt_version = np.array(123, dtype=np.int8))
execute(save_op)
add_op = xdl.ps_assign_add_op(var_name="w", var_type="index", delta=np.array([1,2,3,4], dtype=np.int32))
execute(add_op)
ret = execute(var.value)
self.assertTrue((ret == np.array([1,2,3,4])).all())
restore_op = xdl.ps_restore_op(ckpt_version = np.array(123, dtype=np.int8))
execute(restore_op)
#execute(xdl.variable_registers())
#execute(xdl.global_initializers())
ret = execute(var.value)
self.assertTrue((ret == np.array([0,0,0,0])).all())
#pdb.set_trace()
save_op = xdl.ps_save_op(ckpt_version = np.array(1234, dtype=np.int8))
execute(save_op)
`
`

Traceback (most recent call last):
File "test_ps_save_and_restore_op.py", line 43, in test_all
execute(save_op)
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/lib/graph.py", line 242, in execute
return current_graph().execute(outputs, run_option, run_statistic)
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/lib/graph.py", line 178, in execute
check_error(result.status)
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/lib/error.py", line 29, in check_error
raise ArgumentError(status.msg)
ArgumentError: ErrorCode[1], ErrorMsg[Serializer should not accept nullptr
Check Status [PS2XDL::ConvertStatus(st)] at [/home/yue.song/x-deeplearning/xdl/xdl/core/ops/ps_ops/ps_save_op.cc]operator()@40]

`

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the failure with test_ps_save_and_restore_op.py, following the save, restore, and second-save sequence shown in the report. Then inspect xdl/core/ops/ps_ops/ps_save_op.cc at the reported error location; the issue is done when saving a checkpoint after restoring one completes without the Serializer nullptr error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.