TiFlash can not start occasionally with "prof_active:true"

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

export MALLOC_CONF="prof:true,prof_active:true"
./tiflash version

2. What did you expect to see? (Required)

tiflash run normally, print the version info

3. What did you see instead (Required)

In certain environment, tiflash can not start. With gdb, we can see the backtrack as follow

[root@tc-tiflash-1 tiflash]# gdb ./tiflash
GNU gdb (Rocky Linux) 14.2-3.el9
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./tiflash...
(gdb) r
Starting program: /tiflash/tiflash
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
access_mem (as=<optimized out>, addr=0, val=0xffffffffcc18, write=<optimized out>, arg=<optimized out>) at /workspace/source/tiflash/contrib/libunwind/src/aarch64/Ginit.c:337
warning: 337    /workspace/source/tiflash/contrib/libunwind/src/aarch64/Ginit.c: No such file or directory
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.34-125.el9_5.1.aarch64 libgcc-11.5.0-2.el9.aarch64
(gdb) bt
#0  access_mem (as=<optimized out>, addr=0, val=0xffffffffcc18, write=<optimized out>, arg=<optimized out>) at /workspace/source/tiflash/contrib/libunwind/src/aarch64/Ginit.c:337
#1  0x0000aaaab374a510 in is_plt_entry (c=0xffffffffd0f0) at /workspace/source/tiflash/contrib/libunwind/src/aarch64/Gstep.c:43
#2  0x0000aaaab374a480 in _ULaarch64_step (cursor=0xffffffffd0f0) at /workspace/source/tiflash/contrib/libunwind/src/aarch64/Gstep.c:171
#3  0x0000aaaab374cd38 in trace_init_addr (f=0xfffff0dc0780, cursor=0xffffffffd0f0, cfa=281474976697600, pc=187650131914299, fp=281474976702608, sp=281474976697600) at /workspace/source/tiflash/contrib/libunwind/src/aarch64/Gtrace.c:251
#4  trace_lookup (cursor=0xffffffffd0f0, cache=0xfffff0dfffe0, cfa=281474976697600, pc=187650131914299, fp=281474976702608, sp=281474976697600) at /workspace/source/tiflash/contrib/libunwind/src/aarch64/Gtrace.c:333
#5  _ULaarch64_tdep_trace (cursor=0xffffffffd0f0, buffer=0xffffa00200b8, size=0xffffffffcd0c) at /workspace/source/tiflash/contrib/libunwind/src/aarch64/Gtrace.c:451
#6  0x0000aaaab3746e60 in unw_backtrace (buffer=0xffffa00200b8, size=128) at /workspace/source/tiflash/contrib/libunwind/src/mi/backtrace.c:69
#7  0x0000aaaab372f484 in prof_backtrace_impl (vec=0xaaaab487ca58 <local_addr_space.llvm>, len=0xffffffffe128, max_len=<optimized out>) at /workspace/source/tiflash/contrib/jemalloc/src/prof_sys.c:60
#8  0x0000aaaab372f374 in prof_backtrace (tsd=0xfffff0e10858, bt=<optimized out>) at /workspace/source/tiflash/contrib/jemalloc/src/prof_sys.c:284
#9  0x0000aaaab3718354 in prof_tctx_create (tsd=0xfffff0e10858) at /workspace/source/tiflash/contrib/jemalloc/src/prof.c:195
#10 0x0000aaaab36bd374 in prof_alloc_prep (tsd=0xfffff0e10858, prof_active=<optimized out>, sample_event=<optimized out>) at /workspace/source/tiflash/contrib/jemalloc/include/jemalloc/internal/prof_inlines.h:141
#11 imalloc_body (sopts=<optimized out>, dopts=<optimized out>, tsd=<optimized out>) at /workspace/source/tiflash/contrib/jemalloc/src/jemalloc.c:2550
#12 imalloc (sopts=<optimized out>, dopts=<optimized out>) at /workspace/source/tiflash/contrib/jemalloc/src/jemalloc.c:2694
#13 malloc_default (size=320) at /workspace/source/tiflash/contrib/jemalloc/src/jemalloc.c:2722
#14 0x0000fffff7e1aeb8 in operator new(unsigned long) () from /tiflash/libc++abi.so.1
#15 0x0000fffff728c1e8 in std::__1::__libcpp_operator_new[abi:ue170006]<unsigned long>(unsigned long) (__args=320) at /usr/local/bin/../include/c++/v1/new:268
#16 std::__1::__libcpp_allocate[abi:ue170006](unsigned long, unsigned long) (__size=320, __align=16) at /usr/local/bin/../include/c++/v1/new:294
#17 std::__1::allocator<std::__1::__hash_node<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, rocksdb::OptionTypeInfo>, void*> >::allocate[abi:ue170006](unsigned long) (this=0xfffff7db8f70 <rocksdb::stats_type_info+16>, __n=1) at /usr/local/bin/../include/c++/v1/__memory/allocator.h:114
#18 std::__1::allocator_traits<std::__1::allocator<std::__1::__hash_node<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, rocksdb::OptionTypeInfo>, void*> > >::allocate[abi:ue170006](std::__1::allocator<std::__1::__hash_node<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, rocksdb::OptionTypeInfo>, void*> >&, unsigned long) (__a=..., __n=1) at /usr/local/bin/../include/c++/v1/__memory/allocator_traits.h:268
#19 std::__1::__hash_table<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, rocksdb::OptionTypeInfo>, std::__1::__unordered_map_hasher<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, rocksdb::OptionTypeInfo>, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::__unordered_map_equal<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, rocksdb::OptionTypeInfo>, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::allocator<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, rocksdb::OptionTypeInfo> > >::__construct_node_hash<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, rocksdb::OptionTypeInfo> const&> (this=this@entry=0xfffff7db8f60 <rocksdb::stats_type_info>, __hash=__hash@entry=11229108921762462564, __f=...) at /usr/local/bin/../include/c++/v1/__hash_table:2211
#20 0x0000fffff728bf90 in std::__1::__hash_table<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, rocksdb::OptionTypeInfo>, std::__1::__unordered_map_hasher<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, rocksdb::OptionTypeInfo>, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::__unordered_map_equal<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, rocksdb::OptionTypeInfo>, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::allocator<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, rocksdb::OptionTypeInfo> > >::__emplace_unique_key_args<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, rocksdb::OptionTypeInfo> const&> (this=0xfffff7db8f60 <rocksdb::stats_type_info>, __k=..., __args=...) at /usr/local/bin/../include/c++/v1/__hash_table:1847
#21 0x0000fffff728f86c in std::__1::__hash_table<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, rocksdb::OptionTypeInfo>, std::__1::__unordered_map_hasher<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, rocksdb::OptionTypeInfo>, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::__unordered_map_equal<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, rocksdb::OptionTypeInfo>, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::allocator<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, rocksdb::OptionTypeInfo> > >::__insert_unique[abi:ue170006](std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, rocksdb::OptionTypeInfo> const&) (this=0xaaaab487ca58 <local_addr_space.llvm>, __x=...) at /usr/local/bin/../include/c++/v1/__hash_table:974
#22 std::__1::unordered_map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, rocksdb::OptionTypeInfo, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, rocksdb::OptionTypeInfo> > >::insert<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, rocksdb::OptionTypeInfo> const*> (this=0xaaaab487ca58 <local_addr_space.llvm>,
    __first=0xffffffffe3f0, __last=0xffffffffe520) at /usr/local/bin/../include/c++/v1/unordered_map:1893
#23 std::__1::unordered_map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, rocksdb::OptionTypeInfo, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, rocksdb::OptionTypeInfo> > >::unordered_map (this=0xaaaab487ca58 <local_addr_space.llvm>, __il=...) at /usr/local/bin/../include/c++/v1/unordered_map:1840
#24 __cxx_global_var_init () at rocksdb/monitoring/statistics_impl.h:137
#25 0x0000fffff728f86c in _GLOBAL__sub_I_c.cc () from /tiflash/libtiflash_proxy.so
#26 0x0000fffff7fb4c14 in call_init (env=0xffffffffe888, argv=0xffffffffe878, argc=1, l=<optimized out>) at dl-init.c:70
#27 call_init (l=<optimized out>, argc=1, argv=0xffffffffe878, env=0xffffffffe888) at dl-init.c:26
#28 0x0000fffff7fb4d1c in _dl_init (main_map=0xfffff7ff12b0, argc=1, argv=0xffffffffe878, env=0xffffffffe888) at dl-init.c:117
#29 0x0000fffff7fc7008 in _dl_start_user () from /lib/ld-linux-aarch64.so.1
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

4. What is your TiFlash version? (Required)

v8.5.2

Reproduce the startup failure with MALLOC_CONF="prof:true,prof_active:true" and ./tiflash version. Read the backtrace through contrib/libunwind/src/aarch64/Ginit.c, contrib/jemalloc/src/prof_sys.c, and rocksdb/monitoring/statistics_impl.h to isolate the initialization path. Done means TiFlash starts and prints its version in the affected environment.