alibaba / alibaba/euler

您好,在进行完分布式训练后,导出模型时报出以下错误,请问一下该如何解决啊

Open
#182 20 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
2.9k
Forks
553
PR merge metrics
No merged PRs in 30d

Description

InvalidArgumentError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

hdfs://ip-xxx-xx-xx-xx.ec2.internal:9000/euler/BR/single_case_embedding/model.ckpt-52.data-00000-of-00001; Invalid argument
[[node save/RestoreV2 (defined at python/run_loop.py:153) = RestoreV2[dtypes=[DT_INT64, DT_FLOAT], _device="/job:ps/replica:0/task:0/device:CPU:0"](_recv_save/Const_0_S1, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with python/run_loop.py:153 and trace the distributed-training export path around the RestoreV2 failure. Reproduce the export with model.ckpt-52, then compare the current graph and checkpoint details to identify the mismatch. Done means the checkpoint restores and the exported model completes without the reported InvalidArgumentError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.