alibaba / alibaba/graph-learn

Error occurs when running gl on ps mode

Open
#281 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
1.3k
Forks
267
PR merge metrics
No merged PRs in 30d

Description

here is the error message from ps node
```
[train-fr-v020-incnqlab-ps-0:00065] *** Process received signal ***

[train-fr-v020-incnqlab-ps-0:00065] Signal: Segmentation fault (11)

[train-fr-v020-incnqlab-ps-0:00065] Signal code: Address not mapped (1)

[train-fr-v020-incnqlab-ps-0:00065] Failing at address: (nil)

[train-fr-v020-incnqlab-ps-0:00065] [ 0] /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x995689)[0x7f6d64e37689]

[train-fr-v020-incnqlab-ps-0:00065] [ 1] /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(JVM_handle_linux_signal+0x28d)[0x7f6d64e3c59d]

[train-fr-v020-incnqlab-ps-0:00065] [ 2] /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so(+0x98c9c8)[0x7f6d64e2e9c8]

[train-fr-v020-incnqlab-ps-0:00065] [ 3] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7f6d9e2ec980]

[train-fr-v020-incnqlab-ps-0:00065] [ 4] /opt/conda/lib/python3.7/site-packages/graphlearn/python/lib/libgraphlearn_shared.so(_ZN4absl12lts_202103245Mutex4LockEv+0x4)[0x7f6c717274c4]

[train-fr-v020-incnqlab-ps-0:00065] [ 5] /opt/conda/lib/python3.7/site-packages/graphlearn/python/lib/libgraphlearn_shared.so(_ZN9grpc_core8channelz16ChannelzRegistry16InternalRegisterEPNS0_8BaseNodeE+0x17)[0x7f6c714dc577]

[train-fr-v020-incnqlab-ps-0:00065] [ 6] /opt/conda/lib/python3.7/site-packages/graphlearn/python/lib/libgraphlearn_shared.so(_ZN9grpc_core8channelz8BaseNodeC2ENS1_10EntityTypeENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x6f)[0x7f6c714d01df]

[train-fr-v020-incnqlab-ps-0:00065] [ 7] /opt/conda/lib/python3.7/site-packages/graphlearn/python/lib/libgraphlearn_shared.so(_ZN9grpc_core8channelz10ServerNodeC1Em+0x35)[0x7f6c714d19b5]

[train-fr-v020-incnqlab-ps-0:00065] [ 8] /opt/conda/lib/python3.7/site-packages/graphlearn/python/lib/libgraphlearn_shared.so(_ZN9grpc_core6ServerC1EPK17grpc_channel_args+0x1dc)[0x7f6c7151a6dc]

[train-fr-v020-incnqlab-ps-0:00065] [ 9] /opt/conda/lib/python3.7/site-packages/graphlearn/python/lib/libgraphlearn_shared.so(grpc_server_create+0xfc)[0x7f6c7151a85c]

[train-fr-v020-incnqlab-ps-0:00065] [10] /opt/conda/lib/python3.7/site-packages/graphlearn/python/lib/libgraphlearn_shared.so(_ZN4grpc6ServerC2EPNS_16ChannelArgumentsESt10shared_ptrISt6vectorISt10unique_ptrINS_21ServerCompletionQueueESt14default_deleteIS6_EESaIS9_EEEiiiS4_IS3_INS_8internal30ExternalConnectionAcceptorImplEESaISF_EEP26grpc_server_config_fetcherP19grpc_resource_quotaS4_IS5_INS_12experimental33ServerInterceptorFactoryInterfaceES7_ISN_EESaISP_EE+0x56e)[0x7f6c71308a1e]

[train-fr-v020-incnqlab-ps-0:00065] [11] /opt/conda/lib/python3.7/site-packages/graphlearn/python/lib/libgraphlearn_shared.so(_ZN4grpc13ServerBuilder13BuildAndStartEv+0x2dc)[0x7f6c7130372c]

[train-fr-v020-incnqlab-ps-0:00065] [12] /opt/conda/lib/python3.7/site-packages/graphlearn/python/lib/libgraphlearn_shared.so(_ZN10graphlearn17DistributeService12StartAndJoinEv+0x99)[0x7f6c712a6e89]

[train-fr-v020-incnqlab-ps-0:00065] [13] /opt/conda/lib/python3.7/site-packages/graphlearn/python/lib/libgraphlearn_shared.so(+0x321223)[0x7f6c712a7223]

[train-fr-v020-incnqlab-ps-0:00065] [14] /opt/conda/lib/python3.7/site-packages/graphlearn/python/lib/libgraphlearn_shared.so(_ZN10graphlearn23DynamicWorkerThreadPool13WorkerRoutineEv+0x7d)[0x7f6c7124b06d]

[train-fr-v020-incnqlab-ps-0:00065] [15] /opt/conda/lib/python3.7/site-packages/graphlearn/python/lib/libgraphlearn_shared.so(+0x2c5263)[0x7f6c7124b263]

[train-fr-v020-incnqlab-ps-0:00065] [16] /opt/conda/lib/python3.7/site-packages/graphlearn/python/lib/libgraphlearn_shared.so(_ZN10graphlearn6Thread7RoutineEv+0xe)[0x7f6c7124cb0e]

[train-fr-v020-incnqlab-ps-0:00065] [17] /opt/conda/lib/python3.7/site-packages/graphlearn/python/lib/libgraphlearn_shared.so(+0x2c6cc3)[0x7f6c7124ccc3]

[train-fr-v020-incnqlab-ps-0:00065] [18] /opt/conda/lib/python3.7/site-packages/graphlearn/python/lib/libgraphlearn_shared.so(+0x2c69ca)[0x7f6c7124c9ca]

[train-fr-v020-incnqlab-ps-0:00065] [19] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f6d9e2e16db]

[train-fr-v020-incnqlab-ps-0:00065] [20] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f6d9e00a71f]

[train-fr-v020-incnqlab-ps-0:00065] *** End of error message ***
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the reported `gl` run in ps mode with the Python 3.7 environment described in the trace. Inspect the C++ distributed-service path around `graphlearn::DistributeService::StartAndJoin`, `grpc_server_create`, and the Channelz stack; done means the ps process no longer segfaults under the reported setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, grpc, python
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
18/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.