alibaba / alibaba/x-deeplearning

CrossMedia的预测模式

Open
#169 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
PureBasic
Stars
4.3k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

CrossMedia不支持单机模式,在我部署的分布式环境下做predict遇到如下问题,求大神帮忙看看问题:
1、ps报错,提示Not found variable[user_graphfc1/bias] part[5] in variable_infos when load variable.
2、worker报错,提示:
execute fail retry cnt[1]
I
('An error was raised. This may be due to a preemption in a connected worker or parameter server. The current session will be closed and a new session will be created. Error: %s', 'ErrorCode[4], ErrorMsg[RunUdfChain Version Mismatch\nCheck Status [PS2XDL::ConvertStatus(st)] at [/home/yue.song/x-deeplearning/xdl/xdl/core/ops/ps_ops/ps_pull_op.cc]operator()@41\nCheck Status [PS2XDL::ConvertStatus(st)] at [/home/yue.song/x-deeplearning/xdl/xdl/core/ops/ps_ops/model_server/model_server_client_forward.cc]operator()@44]')
I
('An error was raised. This may be due to a preemption in a connected worker or parameter server. The current session will be closed and a new session will be created. Error: %s', 'ErrorCode[4], ErrorMsg[RunUdfChain Version Mismatch\nCheck Status [PS2XDL::ConvertStatus(st)] at [/home/yue.song/x-deeplearning/xdl/xdl/core/ops/ps_ops/ps_pull_op.cc]operator()@41\nCheck Status [PS2XDL::ConvertStatus(st)] at [/home/yue.song/x-deeplearning/xdl/xdl/core/ops/ps_ops/model_server/model_server_client_forward.cc]operator()@44]')
3、schedule报错提示:
I0308 09:41:26.938618 88 scheduler_impl.cc:268] Restore checkpoint :, Get Status: ErrorCode [11]: network error
I0308 09:41:26.938792 88 scheduler_impl.cc:215] All servers are ready
I0308 09:41:26.938827 88 scheduler_impl.cc:221] Try Restoring to latest checkpoint due to error
I0308 09:41:26.938848 88 scheduler_impl.cc:266] Restore checkpoint:
I0308 09:41:26.955829 88 scheduler_impl.cc:442] Real Checkpoints[ckpt-...............10017] with info_size[23]
report error in process, seq=79, serverid=18
I0308 09:41:28.348634 88 scheduler_impl.cc:268] Restore checkpoint :, Get Status: ErrorCode [11]: network error
I0308 09:41:28.348829 88 scheduler_impl.cc:215] All servers are ready

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing CrossMedia prediction in the reported distributed deployment and collect the parameter-server, worker, and scheduler logs shown in the issue. Trace the variable-loading, RunUdfChain version, and checkpoint-restore failures to their relevant xdl components. Done means the root cause is identified and distributed prediction completes with checkpoint restoration.

Written by the indexing model from the issue text.

Assessment

Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.