[Bug](PHDR) BE may deadlock during startup when jemalloc profiling unwinds through PHDRCache dl_iterate_phdr interposer
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Version
master
### What's Wrong?
Doris BE may hang during startup before the service becomes ready.
The observed hang happens very early during process initialization. GDB shows only
one thread, blocked in jemalloc initialization. The stack indicates a recursive
jemalloc initialization path:
1. `libjvm.so` static initialization performs allocation.
2. jemalloc enters `malloc_init_hard()` and holds jemalloc init mutex.
3. jemalloc profiling initialization calls `_Unwind_Backtrace()`.
4. libunwind calls `dl_iterate_phdr()`.
5. Doris' PHDRCache interposes the process-wide `dl_iterate_phdr`.
6. Because `use_phdr_cache == false`, the interposer calls
`getOriginalDLIteratePHDR()`.
7. `getOriginalDLIteratePHDR()` calls `dlsym(RTLD_NEXT, "dl_iterate_phdr")`.
8. `dlsym()` internally allocates memory and re-enters jemalloc initialization
while the jemalloc init mutex is already held.
This causes a startup deadlock.
Relevant Doris code:
```cpp
// be/src/common/phdr_cache.cpp
DLIterateFunction getOriginalDLIteratePHDR() {
void* func = dlsym(RTLD_NEXT, "dl_iterate_phdr");
if (!func) {
throw std::runtime_error("Cannot find dl_iterate_phdr function with
dlsym");
}
return reinterpret_cast(func);
}
extern "C" int dl_iterate_phdr(
int (*callback)(dl_phdr_info* info, size_t size, void* data), void* data)
{
if (!use_phdr_cache) {
return getOriginalDLIteratePHDR()(callback, data);
}
return iteratePHDRCache(callback, data, 0);
}
This seems inconsistent with the comment in be/src/common/phdr_cache.h, which says
normal code paths should keep using the original glibc dl_iterate_phdr, while
Doris-patched libunwind should use the narrower doris_unwind_iterate_phdr hook.
```
Full Stack
```
#0 0x00007f2a73ff029c in __lll_lock_wait () from /lib/x86_64-linux-gnu/
libpthread.so.0
#1 0x00007f2a73fe9714 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/
libpthread.so.0
#2 0x000055b05f66f0c5 in malloc_mutex_lock_final (mutex=0x55b064f4bb50
)
at ../include/jemalloc/internal/mutex.h:151
#3 je_malloc_mutex_lock_slow (mutex=0x55b064f4bb50 ) at ../src/
mutex.c:90
#4 0x000055b05f639140 in malloc_mutex_lock (tsdn=0x0, mutex=)
at ../include/jemalloc/internal/mutex.h:217
#5 malloc_init_hard () at ../src/jemalloc.c:2118
#6 0x000055b05f629cd7 in malloc_init () at ../src/jemalloc.c:298
#7 imalloc_init_check (sopts=, dopts=) at ../src/
jemalloc.c:2658
#8 imalloc (sopts=, dopts=) at ../src/
jemalloc.c:2689
#9 jecalloc (num=1, size=32) at ../src/jemalloc.c:2852
#10 0x00007f2a74001a45 in ?? () from /lib/x86_64-linux-gnu/libdl.so.2
#11 0x00007f2a7400140f in dlsym () from /lib/x86_64-linux-gnu/libdl.so.2
#12 0x000055b057940afc in (anonymous namespace)::getOriginalDLIteratePHDR ()
at ./be/build_Release/./be/src/common/phdr_cache.cpp:86
#13 dl_iterate_phdr (callback=0x55b063fe9420 <_Unwind_IteratePhdrCallback>,
data=0x7ffdc7c0cd50)
at ./be/build_Release/./be/src/common/phdr_cache.cpp:238
#14 0x000055b063febebe in _Unwind_Find_FDE ()
#15 0x000055b063fe7ab7 in uw_frame_state_for ()
#16 0x000055b063fe810d in uw_init_context_1 ()
#17 0x000055b063fe920c in _Unwind_Backtrace ()
#18 0x000055b05f673888 in je_prof_boot2 (tsd=tsd@entry=0x7f2a73e087c0,
base=)
at ../src/prof.c:693
#19 0x000055b05f639591 in malloc_init_hard () at ../src/jemalloc.c:2151
#20 0x000055b05f62718f in malloc_init () at ../src/jemalloc.c:298
#21 imalloc_init_check (sopts=, dopts=) at ../src/
jemalloc.c:2658
#22 imalloc (sopts=, dopts=) at ../src/
jemalloc.c:2689
#23 je_malloc_default (size=40) at ../src/jemalloc.c:2722
#24 0x00007f2a74d8e7a5 in NMTPreInitAllocation::do_alloc(unsigned long) ()
from /opt/tiger/jdk/jdk17/lib/server/libjvm.so
#25 0x00007f2a74dba390 in os::malloc(unsigned long, MEMFLAGS, NativeCallStack
const&) ()
from /opt/tiger/jdk/jdk17/lib/server/libjvm.so
#26 0x00007f2a7457409a in AllocateHeap(unsigned long, MEMFLAGS,
AllocFailStrategy::AllocFailEnum) ()
from /opt/tiger/jdk/jdk17/lib/server/libjvm.so
#27 0x00007f2a74c47876 in LogOutputList::add_output(LogOutput*, LogLevel::type) ()
from /opt/tiger/jdk/jdk17/lib/server/libjvm.so
#28 0x00007f2a74400a3f in _GLOBAL__sub_I_abstractDisassembler.cpp ()
from /opt/tiger/jdk/jdk17/lib/server/libjvm.so
#29 0x00007f2a754f838a in ?? () from /lib64/ld-linux-x86-64.so.2
#30 0x00007f2a754f8486 in ?? () from /lib64/ld-linux-x86-64.so.2
#31 0x00007f2a754ea0ca in ?? () from /lib64/ld-linux-x86-64.so.2
#32 0x0000000000000001 in ?? ()
#33 0x00007ffdc7c0e8ea in ?? ()
#34 0x0000000000000000 in ?? ()
```
### What You Expected?
BE should not deadlock during startup.
### How to Reproduce?
jemalloc profiling is enabled
### Anything Else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Read be/src/common/phdr_cache.cpp and be/src/common/phdr_cache.h, especially getOriginalDLIteratePHDR and the dl_iterate_phdr interposer; reproduce BE startup with jemalloc profiling enabled. Done means BE reaches readiness without the recursive initialization deadlock while the normal and Doris-patched libunwind paths retain their intended behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, java
- Domain
- backend, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100