alibaba / alibaba/x-deeplearning
训练ps报错
- Dominant language
- PureBasic
- Stars
- 4.3k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
#0 ps::serializer::SerializeHelper::Deserialize (mem_guard=..., len=0x7fcc45bf40c0, data=, buf=0x7fcc407ffffc "")
at /root/x-deeplearning/xdl/ps-plus/ps-plus/common/serialize_helper.h:62
62 *data = *(reinterpret_cast(buf));
[Current thread is 1 (Thread 0x7fcc45bff700 (LWP 1208282))]
(gdb) bt
#0 ps::serializer::SerializeHelper::Deserialize (mem_guard=..., len=0x7fcc45bf40c0, data=, buf=0x7fcc407ffffc "")
at /root/x-deeplearning/xdl/ps-plus/ps-plus/common/serialize_helper.h:62
#1 ps::serializer::SerializeHelper::Deserialize (buf=buf@entry=0x7fcc407fe018 "\004", t=t@entry=0x7fc6b12c5048, len=len@entry=0x7fcc45bf42e0, mem_guard=...)
at /root/x-deeplearning/xdl/ps-plus/ps-plus/common/serialize_helper.h:237
#2 0x0000000000e48598 in ps::serializer::WrapperDataDerializer::Deserialize (mem_guard=..., len=0x7fcc45bf42e0, result=0x7fcc45bf41d0, offset=, buf=, this=)
at /root/x-deeplearning/xdl/ps-plus/ps-plus/common/serialize_helper.h:569
#3 ps::serializer::Deserializer >::DeserializeWrapper (this=, id=, buffer=, offset=, result=0x7fcc45bf42d8,
len=0x7fcc45bf42e0, mem_guard=...) at /root/x-deeplearning/xdl/ps-plus/ps-plus/common/serializer.h:119
#4 0x0000000000f599ec in ps::serializer::DeserializerBase::DeserializeAny (mem_guard=..., len=0x7fcc45bf42e0, result=0x7fcc45bf42d8, offset=4, buffer=0x7fcc45bf42f0, id=12635033306690797231)
at /root/x-deeplearning/xdl/ps-plus/ps-plus/common/serializer.h:101
#5 ps::serializer::DeserializeAny (mem_guard=..., len=0x7fcc45bf42e0, result=0x7fcc45bf42d8, offset=4, buffer=0x7fcc45bf42f0, id=)
at /root/x-deeplearning/xdl/ps-plus/ps-plus/common/serializer.h:134
#6 ps::service::seastar::EventReader::ReadDone (this=this@entry=0x7fcc52df7000) at /root/x-deeplearning/xdl/ps-plus/ps-plus/service/seastar/lib/event_client_lib.cc:155
#7 0x0000000000f59fda in ps::service::seastar::EventReader::ReadEvent (this=0x7fcc52df7000) at /root/x-deeplearning/xdl/ps-plus/ps-plus/service/seastar/lib/event_client_lib.cc:62
#8 0x0000000000f4a1b9 in event_process_active_single_queue.isra ()
#9 0x0000000000f4abdf in event_base_loop ()
#10 0x00007fcc650ccc80 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#11 0x00007fcc560006ba in start_thread (arg=0x7fcc45bff700) at pthread_create.c:333
#12 0x00007fcc55a2d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with ps-plus/common/serialize_helper.h at line 62 and trace the Tensor deserialization path through ps-plus/common/serializer.h and service/seastar/lib/event_client_lib.cc at lines 155 and 62. Reproduce the training PS error, determine why the serialized buffer is invalid, and verify that training no longer fails during event reading.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100