您好,在进行完分布式训练后,导出模型时报出以下错误,请问一下该如何解决啊
- Dominant language
- C++
- Stars
- 2.9k
- Forks
- 553
- PR merge metrics
- No merged PRs in 30d
Description
InvalidArgumentError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:
hdfs://ip-xxx-xx-xx-xx.ec2.internal:9000/euler/BR/single_case_embedding/model.ckpt-52.data-00000-of-00001; Invalid argument
[[node save/RestoreV2 (defined at python/run_loop.py:153) = RestoreV2[dtypes=[DT_INT64, DT_FLOAT], _device="/job:ps/replica:0/task:0/device:CPU:0"](_recv_save/Const_0_S1, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with python/run_loop.py:153 and trace the distributed-training export path around the RestoreV2 failure. Reproduce the export with model.ckpt-52, then compare the current graph and checkpoint details to identify the mismatch. Done means the checkpoint restores and the exported model completes without the reported InvalidArgumentError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, tensorflow
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100