alibaba / alibaba/x-deeplearning

ps启动时出core

Open
#188 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
PureBasic
Stars
4.3k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

日志如下:

Segmentation fault (core dumped)

cmd: /usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/utils/../../bin/ps -bc "False" -sp "zfs://172.21.9.152:12181" -si "1" -r "server"
Traceback (most recent call last):
File "../../testMark.py", line 18, in
import xdl
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/__init__.py", line 16, in
from xdl.python.init import *
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/__init__.py", line 16, in
from xdl.python.init import *
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/init.py", line 42, in
from xdl.python.training.train_session import TrainSession, LoggerHook, SyncRunHook, SemiSyncRunHook, BarrierHook
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/training/train_session.py", line 21, in
from xdl.python.training.env import current_env, is_local_mode
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/training/env.py", line 228, in
_ENV = init_env()
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/training/env.py", line 226, in init_env
return DistributedEnv()
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/training/env.py", line 148, in __init__
self._start()
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/training/env.py", line 54, in _start
self._ps_do()
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/training/env.py", line 157, in _ps_do
bind_cores = self._bind_core)
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/utils/ps_utils.py", line 78, in run_ps_server
bc=bind_cores)
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/utils/ps_utils.py", line 67, in run_ps_cmd
raise ValueError("Run cmd Error, cmd=[%s] exit_code=[%s]" % (cmd, ret))
ValueError: Run cmd Error, cmd=[/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/utils/../../bin/ps -bc "False" -sp "zfs://172.21.9.152:12181" -si "1" -r
"server"] exit_code=[35584]

core堆栈:

(gdb) bt

#0 0x0000000000e1d812 in ps::network::SessionContext::SetProcessorId (this=0x0, id=2) at /xdl/xdl/third_party/seastar/service/session_context.hh:37
#1 0x0000000000e204ac in ps::network::ClientNetworkContext::SetResponseProcessorIdOfServer (this=0x7f010ec59948, serverId=0, id=2)
at /xdl/xdl/third_party/seastar/service/client_network_context.hh:227
#2 0x0000000000e6e3b1 in ps::service::seastar::SeastarServerClientLib::Connect (this=0x7f010ec59800, server_id=0, addr="172.17.0.2:43207", async=false,
connect_in_seastar=false) at /xdl/xdl/ps-plus/ps-plus/service/seastar/lib/seastar_server_client_lib.cc:155
#3 0x0000000000ca1e8c in ps::server::ServerService::RegisterServer (this=0x7ffd3de5a8c0) at /xdl/xdl/ps-plus/ps-plus/server/server_service.cc:360
#4 0x0000000000c9ebe9 in ps::server::ServerService::::operator()(void) const (__closure=0x7f0050b64458) at /xdl/xdl/ps-plus/ps-plus/server/server_service.cc:158
#5 0x0000000000ca6f1e in std::_Bind_simple()>::_M_invoke<>(std::_Index_tuple<>) (this=0x7f0050b64458)
at /usr/include/c++/5/functional:1531
#6 0x0000000000ca6e74 in std::_Bind_simple()>::operator()(void) (this=0x7f0050b64458) at /usr/include/c++/5/functional:1520
#7 0x0000000000ca6e04 in std::thread::_Impl()> >::_M_run(void) (this=0x7f0050b64440)
at /usr/include/c++/5/thread:115
#8 0x00007f0113e3ac80 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#9 0x00007f01116756ba in start_thread (arg=0x7eff893ff700) at pthread_create.c:333
#10 0x00007f0110e8041d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb) f 1
#1 0x0000000000e204ac in ps::network::ClientNetworkContext::SetResponseProcessorIdOfServer (this=0x7f010ec59948, serverId=0, id=2)
at /xdl/xdl/third_party/seastar/service/client_network_context.hh:227
227 mSessionsContext[serverId]->SetProcessorId(id);

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with ps-plus/ps-plus/service/seastar/lib/seastar_server_client_lib.cc:155 and third_party/seastar/service/client_network_context.hh:227, then inspect the gdb backtrace through ServerService::RegisterServer. Reproduce the reported ps command and determine why mSessionsContext[serverId] is null; done means ps starts without the segmentation fault and the failure is covered by an appropriate regression check.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.