alibaba / alibaba/x-deeplearning

训练ps报错

Open
#237 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
PureBasic
Stars
4.3k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

#0 ps::serializer::SerializeHelper::Deserialize (mem_guard=..., len=0x7fcc45bf40c0, data=, buf=0x7fcc407ffffc "")
at /root/x-deeplearning/xdl/ps-plus/ps-plus/common/serialize_helper.h:62
62 *data = *(reinterpret_cast(buf));
[Current thread is 1 (Thread 0x7fcc45bff700 (LWP 1208282))]
(gdb) bt
#0 ps::serializer::SerializeHelper::Deserialize (mem_guard=..., len=0x7fcc45bf40c0, data=, buf=0x7fcc407ffffc "")
at /root/x-deeplearning/xdl/ps-plus/ps-plus/common/serialize_helper.h:62
#1 ps::serializer::SerializeHelper::Deserialize (buf=buf@entry=0x7fcc407fe018 "\004", t=t@entry=0x7fc6b12c5048, len=len@entry=0x7fcc45bf42e0, mem_guard=...)
at /root/x-deeplearning/xdl/ps-plus/ps-plus/common/serialize_helper.h:237
#2 0x0000000000e48598 in ps::serializer::WrapperDataDerializer::Deserialize (mem_guard=..., len=0x7fcc45bf42e0, result=0x7fcc45bf41d0, offset=, buf=, this=)
at /root/x-deeplearning/xdl/ps-plus/ps-plus/common/serialize_helper.h:569
#3 ps::serializer::Deserializer >::DeserializeWrapper (this=, id=, buffer=, offset=, result=0x7fcc45bf42d8,
len=0x7fcc45bf42e0, mem_guard=...) at /root/x-deeplearning/xdl/ps-plus/ps-plus/common/serializer.h:119
#4 0x0000000000f599ec in ps::serializer::DeserializerBase::DeserializeAny (mem_guard=..., len=0x7fcc45bf42e0, result=0x7fcc45bf42d8, offset=4, buffer=0x7fcc45bf42f0, id=12635033306690797231)
at /root/x-deeplearning/xdl/ps-plus/ps-plus/common/serializer.h:101
#5 ps::serializer::DeserializeAny (mem_guard=..., len=0x7fcc45bf42e0, result=0x7fcc45bf42d8, offset=4, buffer=0x7fcc45bf42f0, id=)
at /root/x-deeplearning/xdl/ps-plus/ps-plus/common/serializer.h:134
#6 ps::service::seastar::EventReader::ReadDone (this=this@entry=0x7fcc52df7000) at /root/x-deeplearning/xdl/ps-plus/ps-plus/service/seastar/lib/event_client_lib.cc:155
#7 0x0000000000f59fda in ps::service::seastar::EventReader::ReadEvent (this=0x7fcc52df7000) at /root/x-deeplearning/xdl/ps-plus/ps-plus/service/seastar/lib/event_client_lib.cc:62
#8 0x0000000000f4a1b9 in event_process_active_single_queue.isra ()
#9 0x0000000000f4abdf in event_base_loop ()
#10 0x00007fcc650ccc80 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#11 0x00007fcc560006ba in start_thread (arg=0x7fcc45bff700) at pthread_create.c:333
#12 0x00007fcc55a2d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with ps-plus/common/serialize_helper.h at line 62 and trace the Tensor deserialization path through ps-plus/common/serializer.h and service/seastar/lib/event_client_lib.cc at lines 155 and 62. Reproduce the training PS error, determine why the serialized buffer is invalid, and verify that training no longer fails during event reading.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.