alibaba / alibaba/x-deeplearning
ps启动时出core
Open
- Dominant language
- PureBasic
- Stars
- 4.3k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
日志如下:
Segmentation fault (core dumped)
cmd: /usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/utils/../../bin/ps -bc "False" -sp "zfs://172.21.9.152:12181" -si "1" -r "server"
Traceback (most recent call last):
File "../../testMark.py", line 18, in
import xdl
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/__init__.py", line 16, in
from xdl.python.init import *
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/__init__.py", line 16, in
from xdl.python.init import *
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/init.py", line 42, in
from xdl.python.training.train_session import TrainSession, LoggerHook, SyncRunHook, SemiSyncRunHook, BarrierHook
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/training/train_session.py", line 21, in
from xdl.python.training.env import current_env, is_local_mode
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/training/env.py", line 228, in
_ENV = init_env()
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/training/env.py", line 226, in init_env
return DistributedEnv()
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/training/env.py", line 148, in __init__
self._start()
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/training/env.py", line 54, in _start
self._ps_do()
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/training/env.py", line 157, in _ps_do
bind_cores = self._bind_core)
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/utils/ps_utils.py", line 78, in run_ps_server
bc=bind_cores)
File "/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/utils/ps_utils.py", line 67, in run_ps_cmd
raise ValueError("Run cmd Error, cmd=[%s] exit_code=[%s]" % (cmd, ret))
ValueError: Run cmd Error, cmd=[/usr/local/lib/python2.7/dist-packages/xdl-1.0-py2.7.egg/xdl/python/utils/../../bin/ps -bc "False" -sp "zfs://172.21.9.152:12181" -si "1" -r
"server"] exit_code=[35584]
core堆栈:
(gdb) bt
#0 0x0000000000e1d812 in ps::network::SessionContext::SetProcessorId (this=0x0, id=2) at /xdl/xdl/third_party/seastar/service/session_context.hh:37
#1 0x0000000000e204ac in ps::network::ClientNetworkContext::SetResponseProcessorIdOfServer (this=0x7f010ec59948, serverId=0, id=2)
at /xdl/xdl/third_party/seastar/service/client_network_context.hh:227
#2 0x0000000000e6e3b1 in ps::service::seastar::SeastarServerClientLib::Connect (this=0x7f010ec59800, server_id=0, addr="172.17.0.2:43207", async=false,
connect_in_seastar=false) at /xdl/xdl/ps-plus/ps-plus/service/seastar/lib/seastar_server_client_lib.cc:155
#3 0x0000000000ca1e8c in ps::server::ServerService::RegisterServer (this=0x7ffd3de5a8c0) at /xdl/xdl/ps-plus/ps-plus/server/server_service.cc:360
#4 0x0000000000c9ebe9 in ps::server::ServerService::::operator()(void) const (__closure=0x7f0050b64458) at /xdl/xdl/ps-plus/ps-plus/server/server_service.cc:158
#5 0x0000000000ca6f1e in std::_Bind_simple()>::_M_invoke<>(std::_Index_tuple<>) (this=0x7f0050b64458)
at /usr/include/c++/5/functional:1531
#6 0x0000000000ca6e74 in std::_Bind_simple()>::operator()(void) (this=0x7f0050b64458) at /usr/include/c++/5/functional:1520
#7 0x0000000000ca6e04 in std::thread::_Impl()> >::_M_run(void) (this=0x7f0050b64440)
at /usr/include/c++/5/thread:115
#8 0x00007f0113e3ac80 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#9 0x00007f01116756ba in start_thread (arg=0x7eff893ff700) at pthread_create.c:333
#10 0x00007f0110e8041d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb) f 1
#1 0x0000000000e204ac in ps::network::ClientNetworkContext::SetResponseProcessorIdOfServer (this=0x7f010ec59948, serverId=0, id=2)
at /xdl/xdl/third_party/seastar/service/client_network_context.hh:227
227 mSessionsContext[serverId]->SetProcessorId(id);
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with ps-plus/ps-plus/service/seastar/lib/seastar_server_client_lib.cc:155 and third_party/seastar/service/client_network_context.hh:227, then inspect the gdb backtrace through ServerService::RegisterServer. Reproduce the reported ps command and determine why mSessionsContext[serverId] is null; done means ps starts without the segmentation fault and the failure is covered by an appropriate regression check.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100