[BOLT][x86_64] Application crash with segment fault after i change some shared libraries into static libraries
- Dominant language
- LLVM
- Stars
- 40.5k
- Forks
- 18.7k
- PR merge metrics
- PR metrics pending
Description
I am working on DAOS, a storage system, and 'daos_engine' is the bin we discussed. Recently, i tried to change some shared libraries needed by daos_engine into static libraried, like argobots, spdk, dpdk, libfabric, etc., pmdk(libpmem) is still shared libraries, it worked.
However, after bolt instrument, daos_engine crash when load libpmem.so, and nothing happened on aarch64. LLVM version is v21.1.2 with [patch ](https://github.com/llvm/llvm-project/pull/141918)
instrument outputs:
```
[root@ajx116 ceastor]# llvm-bolt --version
LLVM (http://llvm.org/):
LLVM version 21.1.2
Optimized build with assertions.
BOLT revision
Registered Targets:
x86 - 32-bit X86: Pentium-Pro and above
x86-64 - 64-bit X86: EM64T and AMD64
[root@ajx116 ceastor]# llvm-bolt /usr/bin/daos_engine -o daos_engine.inst -instrument --instrumentation-file=daos_engine.fdata
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: BOLT version:
BOLT-INFO: first alloc address is 0x400000
BOLT-INFO: creating new program header table at address 0x4000000, offset 0x3c00000
BOLT-WARNING: debug info will be stripped from the binary. Use -update-debug-sections to keep it.
BOLT-INFO: enabling relocation mode
BOLT-INFO: forcing -jump-tables=move for instrumentation
BOLT-INFO: enabling lite mode
BOLT-WARNING: split function detected on input : ABT_thread_get_func_name.cold. The support is limited in relocation mode
BOLT-WARNING: Failed to analyze 9774 relocations
BOLT-WARNING: unable to disassemble instruction at offset 0x0 (address 0x53644d) in function gf_vect_dot_prod_sse_slver_00050060(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x0 (address 0x5364e9) in function gf_vect_dot_prod_avx_slver_02050061(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x0 (address 0x536665) in function gf_2vect_dot_prod_sse_slver(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x2 (address 0x536785) in function gf_3vect_dot_prod_sse_slver_00060063(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x5 (address 0x5368ee) in function gf_4vect_dot_prod_sse_slver_00060064(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x7 (address 0x536a86) in function gf_5vect_dot_prod_sse_slver(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x7 (address 0x536c55) in function gf_6vect_dot_prod_sse_slver_00050066(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x7 (address 0x536d37) in function gf_2vect_dot_prod_avx_slver_02050191(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x1 (address 0x536e4c) in function gf_3vect_dot_prod_avx_slver(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x7 (address 0x536fa5) in function gf_4vect_dot_prod_avx_slver_02050193(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x7 (address 0x5373e8) in function gf_2vect_dot_prod_avx2_slver_04050196(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x7 (address 0x537518) in function gf_3vect_dot_prod_avx2_slver(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x7 (address 0x537684) in function gf_4vect_dot_prod_avx2_slver_04050198(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x0 (address 0x5379f6) in function gf_6vect_dot_prod_avx2_slver(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x0 (address 0x537d46) in function gf_3vect_mad_sse_slver(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x5 (address 0x53853c) in function gf_6vect_mad_sse_slver_0001020f(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x0 (address 0x538830) in function gf_3vect_mad_avx_slver(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x5 (address 0x538cc9) in function gf_5vect_mad_avx_slver_0201020d(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x5 (address 0x539039) in function gf_vect_mad_avx2_slver(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x0 (address 0x539792) in function gf_5vect_mad_avx2_slver_0401020e(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x2 (address 0x539db0) in function ec_encode_data_slver(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x2 (address 0x539db4) in function gf_vect_mul_slver_00050134(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x2 (address 0x539db8) in function ec_encode_data_update_slver(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x2 (address 0x539dbc) in function gf_vect_dot_prod_slver_00050138(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x2 (address 0x53b717) in function pq_check_sse_slver_00060033(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x0 (address 0x53ba45) in function xor_gen_avx_slver(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x0 (address 0x53c0ee) in function xor_check_slver_00030127(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x2 (address 0x53c6d0) in function crc16_t10dif_01_slver(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x0 (address 0x53cb0a) in function crc16_t10dif_by4_slver_05020016(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x2 (address 0x53e480) in function crc32_ieee_01_slver_01060011(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x0 (address 0x53ee1a) in function crc32_ieee_by4_slver_05020017(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x24cc (address 0x5417dc) in function crc32_iscsi_01
BOLT-WARNING: unable to disassemble instruction at offset 0x0 (address 0x544f24) in function crc64_iso_norm_slver(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x0 (address 0x544f28) in function crc64_jones_refl_slver_00000027(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0x6 (address 0x548299) in function crc64_jones_norm_by8_slver(*2)
BOLT-WARNING: unable to disassemble instruction at offset 0xb61 (address 0x54dbae) in function isal_deflate_icf_body_hash_hist_04
BOLT-WARNING: skipped 4 functions due to cold fragments
BOLT-INFO: 0 out of 24146 functions in the binary (0.0%) have non-empty execution profile
BOLT-INSTRUMENTER: Number of indirect call site descriptors: 23767
BOLT-INSTRUMENTER: Number of indirect call target descriptors: 26214
BOLT-INSTRUMENTER: Number of function descriptors: 23931
BOLT-INSTRUMENTER: Number of branch counters: 1072659
BOLT-INSTRUMENTER: Number of ST leaf node counters: 549051
BOLT-INSTRUMENTER: Number of direct call counters: 0
BOLT-INSTRUMENTER: Total number of counters: 1621710
BOLT-INSTRUMENTER: Total size of counters: 12973680 bytes (static alloc memory)
BOLT-INSTRUMENTER: Total size of string table emitted: 590227 bytes in file
BOLT-INSTRUMENTER: Total size of descriptors: 96168392 bytes in file
BOLT-INSTRUMENTER: Profile will be saved to file daos_engine.fdata
BOLT-INFO: 910719 instructions were shortened
BOLT-INFO: removed 10147 empty blocks
BOLT-INFO: UCE removed 113770 blocks and 6890023 bytes of code
BOLT-WARNING: failed to patch entries in gf_vect_mul_sse. The function will not be optimized
BOLT-WARNING: failed to patch entries in gf_vect_mul_avx. The function will not be optimized
BOLT-WARNING: failed to patch entries in xor_gen_sse. The function will not be optimized
BOLT-WARNING: failed to patch entries in pq_gen_sse. The function will not be optimized
BOLT-WARNING: failed to patch entries in xor_check_sse. The function will not be optimized
BOLT-WARNING: failed to patch entries in pq_check_sse. The function will not be optimized
BOLT-WARNING: failed to patch entries in pq_gen_avx. The function will not be optimized
BOLT-WARNING: failed to patch entries in pq_gen_avx2. The function will not be optimized
BOLT-WARNING: failed to patch entries in crc32_iscsi_00. The function will not be optimized
BOLT-WARNING: failed to patch entries in isal_deflate_icf_body_hash_hist_01. The function will not be optimized
BOLT-WARNING: failed to patch entries in isal_deflate_icf_body_hash_hist_02. The function will not be optimized
BOLT-WARNING: failed to patch entries in isal_update_histogram_01. The function will not be optimized
BOLT-WARNING: failed to patch entries in isal_update_histogram_04. The function will not be optimized
BOLT-WARNING: failed to patch entries in decode_huffman_code_block_stateless_01. The function will not be optimized
BOLT-WARNING: failed to patch entries in decode_huffman_code_block_stateless_04. The function will not be optimized
BOLT-WARNING: failed to patch entries in pci_uio_remap_resource. The function will not be optimized
BOLT-WARNING: failed to patch entries in pci_vfio_map_resource_primary/1(*2). The function will not be optimized
BOLT-WARNING: failed to patch entries in pci_vfio_map_resource_secondary/1(*2). The function will not be optimized
BOLT-WARNING: failed to patch entries in rte_eth_dev_rx_intr_ctl. The function will not be optimized
BOLT-WARNING: failed to patch entries in eal_dynmem_memseg_lists_init. The function will not be optimized
BOLT-WARNING: failed to patch entries in eal_legacy_hugepage_init/1(*2). The function will not be optimized
BOLT-INFO: padding code to 0x9a00000 to accommodate hot text
BOLT-INFO: output linked against instrumentation runtime library, lib entry point is 0xac1c120
BOLT-INFO: clear procedure is 0xac1ac60
BOLT-INFO: patched build-id (flipped last bit)
BOLT-INFO: setting _end to 0xac7f4cc
BOLT-INFO: setting _end to 0xac7f4cc
BOLT-INFO: setting __bolt_runtime_start to 0xac1c120
BOLT-INFO: setting __bolt_runtime_fini to 0xac1c170
BOLT-INFO: setting __hot_start to 0x4200000
BOLT-INFO: setting __hot_end to 0x98113f8
```
backtrace is:
```
#0 0x00007ffff768b00d in __memset_evex_unaligned_erms () from /usr/lib64/libc.so.6
#1 0x00007ffff77a4ae4 in fs_new () from /usr/prereq/release/pmdk/lib/libpmem.so.1
#2 0x00007ffff77ab5a8 in pmem2_auto_flush () from /usr/prereq/release/pmdk/lib/libpmem.so.1
#3 0x00007ffff77a9d8a in pmem_init () from /usr/prereq/release/pmdk/lib/libpmem.so.1
#4 0x00007ffff7fd12ce in call_init () from /lib64/ld-linux-x86-64.so.2
#5 0x00007ffff7fd13bc in _dl_init () from /lib64/ld-linux-x86-64.so.2
#6 0x00007ffff7fe793a in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#7 0x0000000000000001 in ?? ()
#8 0x00007fffffffe3ce in ?? ()
#9 0x0000000000000000 in ?? ()
```
this is gdb info:
```
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff768b00d in __memset_evex_unaligned_erms () from /usr/lib64/libc.so.6
(gdb) disas $rip-32, $rip+32
Dump of assembler code from 0x7ffff768afed to 0x7ffff768b02d:
0x00007ffff768afed: nopl (%rax)
0x00007ffff768aff0 <__memset_evex_unaligned_erms+0>: endbr64
0x00007ffff768aff4 <__memset_evex_unaligned_erms+4>: mov %rdi,%rax
0x00007ffff768aff7 <__memset_evex_unaligned_erms+7>: vpbroadcastb %esi,%ymm16
0x00007ffff768affd <__memset_evex_unaligned_erms+13>: cmp $0x20,%rdx
0x00007ffff768b001 <__memset_evex_unaligned_erms+17>: jb 0x7ffff768b0d0 <__memset_evex_unaligned_erms+224>
0x00007ffff768b007 <__memset_evex_unaligned_erms+23>: cmp $0x40,%rdx
0x00007ffff768b00b <__memset_evex_unaligned_erms+27>: ja 0x7ffff768b020 <__memset_evex_unaligned_erms+48>
=> 0x00007ffff768b00d <__memset_evex_unaligned_erms+29>: vmovdqu64 %ymm16,-0x20(%rdi,%rdx,1)
0x00007ffff768b015 <__memset_evex_unaligned_erms+37>: vmovdqu64 %ymm16,(%rdi)
0x00007ffff768b01b <__memset_evex_unaligned_erms+43>: ret
0x00007ffff768b01c <__memset_evex_unaligned_erms+44>: nopl 0x0(%rax)
0x00007ffff768b020 <__memset_evex_unaligned_erms+48>: cmp 0x1041d1(%rip),%rdx # 0x7ffff778f1f8 <__x86_rep_stosb_threshold>
0x00007ffff768b027 <__memset_evex_unaligned_erms+55>: ja 0x7ffff768afc4 <__memset_evex_erms+4>
0x00007ffff768b029 <__memset_evex_unaligned_erms+57>: vmovdqu64 %ymm16,(%rdi)
End of assembler dump.
(gdb) info registers
rax 0xffffffff 4294967295
rbx 0x7ffff77e9209 140737345655305
rcx 0x7ffff76d86be 140737344538302
rdx 0x38 56
rsi 0x0 0
rdi 0xffffffff 4294967295
rbp 0x1 0x1
rsp 0x7fffffffdf98 0x7fffffffdf98
r8 0x1 1
r9 0x1 1
r10 0x7ffff75a4d30 140737343278384
r11 0x7ffff768aff0 140737344221168
r12 0x0 0
r13 0x7fffffffe168 140737488347496
r14 0x7ffff77efd98 140737345682840
r15 0x0 0
rip 0x7ffff768b00d 0x7ffff768b00d <__memset_evex_unaligned_erms+29>
eflags 0x10283 [ CF SF IF RF ]
cs 0x33 51
ss 0x2b 43
ds 0x0 0
es 0x0 0
fs 0x0 0
gs 0x0 0
k0 0x2000 8192
k1 0xff 255
k2 0x0 0
k3 0x0 0
k4 0x0 0
k5 0x0 0
k6 0x0 0
k7 0x0 0
(gdb) x/10gx $rsi
0x0: Cannot access memory at address 0x0
(gdb) x/10gx $rdx
0x38: Cannot access memory at address 0x38
```
Contributor guide
Assessment
This issue has not been assessed yet.