alibaba / alibaba/PhotonLibOS

crashed in photon::prelocked_thread_interrupt

Open
#1,093 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
1.2k
Forks
179
Avg merge
1d 15h
Merged PRs (30d)
26

Description

photon: 0.9.2

It has occurred twice so far, and the probability of occurrence is very low.

From the coredump analysis, the cvar queue in the thread holds a photon thread of 0xffffffff, which is the same value for both crashes.

```
#0 0x0000000001f584bc in std::__atomic_base::exchange (__m=std::memory_order_acquire, __i=true, this=0x40) at /opt/rh/gcc-toolset-12/root/usr/include/c++/12/bits/atomic_base.h:503
503 exchange(__int_type __i,
[Current thread is 1 (Thread 0x7fae84998640 (LWP 87))]
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.34-168.el9_6.23.x86_64 hwloc-libs-2.4.1-5.el9.x86_64 libaio-0.3.111-13.el9.x86_64 libdwarf-0.3.4-1.el9.1.x86_64 libgcc-11.3.1-2.1.el9.x86_64 libibverbs-54.0-1.el9.x86_64 libnl3-3.11.0-1.el9.x86_64 librdmacm-54.0-1.el9.x86_64 libstdc++-11.3.1-2.1.el9.x86_64 liburing-2.5-1.el9.x86_64 libuuid-2.37.4-21.el9.x86_64 lz4-libs-1.9.3-5.el9.x86_64 numactl-libs-2.0.19-1.el9.x86_64 openssl-libs-3.2.2-6.el9_5.1.x86_64 re2-20211101-20.el9.x86_64 systemd-libs-250-12.el9_1.1.x86_64 zlib-1.2.11-35.el9_1.x86_64
(gdb) bt
#0 0x0000000001f584bc in std::__atomic_base::exchange (__m=std::memory_order_acquire, __i=true, this=0x40) at /opt/rh/gcc-toolset-12/root/usr/include/c++/12/bits/atomic_base.h:503
#1 std::atomic::exchange (__m=std::memory_order_acquire, __i=true, this=0x40) at /opt/rh/gcc-toolset-12/root/usr/include/c++/12/atomic:120
#2 photon::spinlock::xchg (this=0x40) at /home/jenkins/workspace/photon/include/photon/thread/thread.h:240
#3 photon::spinlock::lock (this=0x40) at /home/jenkins/workspace/photon/include/photon/thread/thread.h:220
#4 photon::locker::lock (must_lock=true, this=)
at /home/jenkins/workspace/photon/include/photon/thread/thread.h:340
#5 photon::locker::locker (mutex=0x40, do_lock=2, this=)
at /home/jenkins/workspace/photon/include/photon/thread/thread.h:323
#6 photon::locker::locker (do_lock=2, mutex=0x40, this=)
at /home/jenkins/workspace/photon/include/photon/thread/thread.h:320
#7 photon::locker::locker (do_lock=2, mutex=..., this=)
at /home/jenkins/workspace/photon/include/photon/thread/thread.h:328
#8 photon::vcpu_t::_move_to_standbyq_atomic (th=0xffffffff, this=0x0) at /home/jenkins/workspace/photon/thread/thread.cpp:557
#9 photon::vcpu_t::move_to_standbyq_atomic (x=0xffffffff, this=0x0)
at /home/jenkins/workspace/photon/thread/thread.cpp:538
#10 photon::prelocked_thread_interrupt (th=th@entry=0xffffffff, error_number=)
at /home/jenkins/workspace/photon/thread/thread.cpp:1444
#11 0x0000000001f58f6d in photon::waitq::resume_one (this=this@entry=0x3d923f60, error_number=error_number@entry=-1)
at /home/jenkins/workspace/photon/thread/thread.cpp:1677
#12 0x0000000001f5d053 in photon::condition_variable::notify_one (this=0x3d923f60)
at /home/jenkins/workspace/photon/include/photon/thread/thread.h:427
#13 photon::ThreadPoolBase::thread_create_ex (this=this@entry=0x36062f00, start=start@entry=0x1f56c40 , arg=arg@entry=0x3d923e30, joinable=joinable@entry=false)
at /home/jenkins/workspace/photon/thread/thread-pool.cpp:60
#14 0x0000000001f57692 in photon::ThreadPoolBase::thread_create (arg=0x3d923e30, start=0x1f56c40 , this=0x36062f00)
at /home/jenkins/workspace/photon/include/photon/thread/thread-pool.h:43
#15 photon::WorkPool::impl::main_loop (this=this@entry=0x32a14000) at /home/jenkins/workspace/photon/thread/workerpool.cpp:134
#16 0x0000000001f56bd2 in photon::WorkPool::impl::join_current_vcpu_into_workpool (this=0x32a14000)
at /home/jenkins/workspace/photon/thread/workerpool.cpp:165
#17 photon::WorkPool::join_current_vcpu_into_workpool (this=)
at /home/jenkins/workspace/photon/thread/workerpool.cpp:221
#18 0x000000000103e56e in xbase::photon::WorkerPool::join_current_vcpu_into_workpool (id=4, this=0x2a96dc70)
at /home/jenkins/workspace/r-build_XFinity_4.2.000.0.251225/xmake_globaldir/.xmake/packages/x/xbase/2.4.3/aeb6a6d6a0fc48e68ef63cfef8c8df3d/include/xbase/photon/worker_pool.h:32
#19 bs::shard::XdfsSdkModule::init()::{lambda(unsigned long)#1}::operator()(unsigned long) const::{lambda()#1}::operator()() const (__closure=) at src/bdev/store/xdfs_sdk_adaptor/xdfs_sdk_module.cc:109
#20 photon::FunctorWrapper::operator()() (this=)
at /home/jenkins/workspace/include/photon/thread/thread11.h:54
#21 tuple_assistance::apply_impl, std::tuple<>>(photon::FunctorWrapper&&, std::tuple<>&&, std::integer_sequence) (t=..., f=...)
at /home/jenkins/workspace/include/photon/common/tuple-assistance.h:80
#22 tuple_assistance::apply, std::tuple<> >(photon::FunctorWrapper&&, std::tuple<>&&) (t=..., f=...)
at /home/jenkins/workspace/include/photon/common/tuple-assistance.h:87
#23 photon::__stub11:: >, std::tuple<> > >(void *) ()
at /home/jenkins/workspace/include/photon/thread/thread11.h:31
#24 0x0000000001f5caa7 in _photon_thread_stub ()
#25 0x0000000000000000 in ?? ()
(gdb) f 11
#11 0x0000000001f58f6d in photon::waitq::resume_one (this=this@entry=0x3d923f60, error_number=error_number@entry=-1)
at /home/jenkins/workspace/photon/thread/thread.cpp:1677
1677 /home/jenkins/workspace/photon/thread/thread.cpp: No such file or directory.
(gdb) p *this
$1 = {q = {th = 0xffffffff, lock = {_lock = {_M_base = {static _S_alignment = 1, _M_i = false}}}}}
(gdb) f 12
#12 0x0000000001f5d053 in photon::condition_variable::notify_one (this=0x3d923f60)
at /home/jenkins/workspace/photon/include/photon/thread/thread.h:427
427 /home/jenkins/workspace/photon/include/photon/thread/thread.h: No such file or directory.
(gdb) p *this
$2 = { = {q = {th = 0xffffffff, lock = {_lock = {_M_base = {static _S_alignment = 1, _M_i = false}}}}}, }
(gdb) f 13
#13 photon::ThreadPoolBase::thread_create_ex (this=this@entry=0x36062f00, start=start@entry=0x1f56c40 , arg=arg@entry=0x3d923e30, joinable=joinable@entry=false)
at /home/jenkins/workspace/photon/thread/thread-pool.cpp:60
60 /home/jenkins/workspace/photon/thread/thread-pool.cpp: No such file or directory.
(gdb) p pCtrl
$3 = (photon::TPControl *) 0x3d923f40
(gdb) p pCtrl[0]
$4 = {th = 0x0, pool = 0x36062f00, start = 0x1f56c40 , arg = 0x3d923e30, cvar = { = {q = {th = 0xffffffff, lock = {_lock = {_M_base = {static _S_alignment = 1,
_M_i = false}}}}}, }, joinable = false, joining = false, m_mtx = {_lock = {_M_base = {static _S_alignment = 1, _M_i = true}}}}
(gdb) p this
$5 = (photon::ThreadPoolBase * const) 0x36062f00
(gdb) p pCtrl[0].pool
$6 = (photon::ThreadPoolBase *) 0x36062f00
(gdb) p pCtrl[0].pool[0]
$7 = {> = { = {> = {<__intrusive_list_node> = {__prev_ptr = 0x36062f00, __next_ptr = 0x36062f00}, }, m_capacity = 64,
m_size = 0, m_refcnt = 2820, min_size_in_interval = 0, m_mtx = {_lock = {_M_base = {static _S_alignment = 1, _M_i = false}}}, m_cvar = { = {q = {th = 0x0, lock = {_lock = {_M_base = {static _S_alignment = 1,
_M_i = false}}}}}, }, m_ctor = { = {}, _obj = 0x36062f00, _func = 0x1f5cf40 },
m_dtor = { = {}, _obj = 0x36062f00, _func = 0x1f5d3b0 },
m_reserved = 0x800000 > >::format(spdlog::details::log_msg const&, tm const&, fmt::v9::basic_memory_buffer >&)+128>, autoscale = false, m_items = 0x36062f68}, }, }
(gdb)
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the provided coredump and inspect thread.cpp at lines 1444, 1677, and 557, then trace the condition-variable path through thread.h and thread-pool.cpp:60. Determine how 0xffffffff enters the cvar queue during thread creation and verify that the underlying crash is prevented, ideally with a reproducible regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.