dmlc / dmlc/parameter_server

System hangs when a server is killed

Open
#32 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
649
Forks
233
PR merge metrics
No merged PRs in 30d

Description

The system hangs (node where scheduler is running stops displaying iteration outcomes on the terminal. Logs are also not generated) whenever a server process is killed (both on local machine or on remote machine). Is it the expected behaviour ? Shouldn't it still continue to run with gradients being updated on the backup/replicated server node as described in the paper ?

Here are the steps that I ran (from "parameter_server/example/linear" dir):

> ../../script/ps.sh start -nw 4 -ns 3 -hostfile hostfile ../../build/linear -app_file ctr/online_l1lr.conf -num_replicas 2 -report_interval 1

Then I killed a server process on one of the nodes. This stops the system. Killing a worker node, still continues the SGD and converges eventually.

Any help in this regard will be highly appreciated.

Thanks,
Danish

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.