alibaba / alibaba/x-deeplearning
从checkpoint里恢复后,再次保存checkpoint时报错
- Dominant language
- PureBasic
- Stars
- 4.3k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
`
def test_all(self):
var = xdl.Variable(name="w", dtype=DataType.int32, shape=[4], initializer=xdl.Zeros())
execute(xdl.variable_registers())
execute(xdl.global_initializers())
save_op = xdl.ps_save_op(ckpt_version = np.array(123, dtype=np.int8))
execute(save_op)
add_op = xdl.ps_assign_add_op(var_name="w", var_type="index", delta=np.array([1,2,3,4], dtype=np.int32))
execute(add_op)
ret = execute(var.value)
self.assertTrue((ret == np.array([1,2,3,4])).all())
restore_op = xdl.ps_restore_op(ckpt_version = np.array(123, dtype=np.int8))
execute(restore_op)
#execute(xdl.variable_registers())
#execute(xdl.global_initializers())
ret = execute(var.value)
self.assertTrue((ret == np.array([0,0,0,0])).all())
#pdb.set_trace()
save_op = xdl.ps_save_op(ckpt_version = np.array(1234, dtype=np.int8))
execute(save_op)
`
`
Traceback (most recent call last):
File "test_ps_save_and_restore_op.py", line 43, in test_all
execute(save_op)
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/lib/graph.py", line 242, in execute
return current_graph().execute(outputs, run_option, run_statistic)
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/lib/graph.py", line 178, in execute
check_error(result.status)
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/lib/error.py", line 29, in check_error
raise ArgumentError(status.msg)
ArgumentError: ErrorCode[1], ErrorMsg[Serializer should not accept nullptr
Check Status [PS2XDL::ConvertStatus(st)] at [/home/yue.song/x-deeplearning/xdl/xdl/core/ops/ps_ops/ps_save_op.cc]operator()@40]
`
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the failure with test_ps_save_and_restore_op.py, following the save, restore, and second-save sequence shown in the report. Then inspect xdl/core/ops/ps_ops/ps_save_op.cc at the reported error location; the issue is done when saving a checkpoint after restoring one completes without the Serializer nullptr error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100