Strange segfault observed since 1.59.0
- Dominant language
- C++
- Stars
- 45.3k
- Forks
- 11.4k
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 47
Description
### What version of gRPC and what language are you using?
1.59.0+
php82-pecl-gRPC
### What operating system (Linux, Windows,...) and version?
Multiple FreeBSD versions: 12.3, 13.1, 13.2
Multiple PHP versions, all 8.2.x though
### What runtime / compiler are you using (e.g. python version or version of GCC)
c++ -v
FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2)
Target: x86_64-unknown-freebsd12.3
### What did you do?
Ever since I upgraded from pecl-grpc-1.58.0 to 1.59.1 (and subsequently other versions) my php-fpm child processes started segfault randomly at a high rate (I have the same code running on different physical machines, and they all started to segfault immediately after upgrade from pecl-grpc-1.58.0)
Here's a graph showing this:

The first green annotation is upgraded from 1.58.0 to 1.59.1, the second annotation is upgraded to 1.60.0, and the third is 1.61.0
There is nothing in my logs, except the crash:
```
Feb 28 18:07:52 hub2 kernel: pid 33795 (php-fpm), jid 4, uid 80: exited on signal 11
Feb 28 18:13:02 hub2 kernel: pid 16792 (php-fpm), jid 4, uid 80: exited on signal 11
Feb 28 18:20:40 hub2 kernel: pid 58462 (php-fpm), jid 4, uid 80: exited on signal 11
Feb 28 18:23:00 hub2 kernel: pid 59039 (php-fpm), jid 4, uid 80: exited on signal 11
Feb 28 18:24:11 hub2 kernel: pid 59121 (php-fpm), jid 4, uid 80: exited on signal 11
Feb 28 18:32:15 hub2 kernel: pid 59842 (php-fpm), jid 4, uid 80: exited on signal 11
Feb 28 18:53:21 hub2 kernel: pid 62657 (php-fpm), jid 4, uid 80: exited on signal 11
Feb 28 19:11:56 hub2 kernel: pid 64540 (php-fpm), jid 4, uid 80: exited on signal 11
Feb 28 19:18:09 hub2 kernel: pid 10681 (php-fpm), jid 4, uid 80: exited on signal 11
```
This is the last 24 hours, where the green annotation shows the point I downgraded back to 1.58.0:

Pretty much all the core-dumps looks the same:
```
(lldb) bt all
* thread #1, name = 'php-fpm', stop reason = signal SIGSEGV
* frame #0: 0x0000000803e00288
frame #1: 0x00007fffdf9faef0
thread #2, name = 'php-fpm', stop reason = signal SIGSEGV
frame #0: 0x0000000800a3bcee ld-elf.so.1`__crt_free(cp=0x0000000802b78800) at rtld_malloc.c:247:14
frame #1: 0x0000000800a3e5b2 ld-elf.so.1`obj_free(obj=0x0000000802b7f400) at map_object.c:444:2
frame #2: 0x0000000800a39e2c ld-elf.so.1`unload_object [inlined] release_object(obj=0x0000000802b7f400) at rtld.c:4165:2
frame #3: 0x0000000800a39e0c ld-elf.so.1`unload_object(root=, lockstate=0x00007fffffffe500) at rtld.c:4778
frame #4: 0x0000000800a34490 ld-elf.so.1`dlclose_locked(handle=0x0000000802b7f400, lockstate=0x00007fffffffe500) at rtld.c:3332:2
frame #5: 0x0000000800a34301 ld-elf.so.1`dlclose(handle=) at rtld.c:3302:10
frame #6: 0x000000000083f6aa php-fpm`module_destructor(module=0x0000000804218330) at zend_API.c:3078:3
frame #7: 0x000000000082f450 php-fpm`module_destructor_zval(zv=0x00007fffffffe628) at zend.c:854:2
frame #8: 0x0000000000851d63 php-fpm`_zend_hash_del_el_ex(ht=0x0000000000a43388, idx=21, p=0x00000008017854a0, prev=0x0000000000000000) at zend_hash.c:1425:3
frame #9: 0x0000000000851910 php-fpm`_zend_hash_del_el(ht=0x0000000000a43388, idx=21, p=0x00000008017854a0) at zend_hash.c:1452:2
frame #10: 0x0000000000854253 php-fpm`zend_hash_graceful_reverse_destroy(ht=0x0000000000a43388) at zend_hash.c:1977:4
frame #11: 0x000000000083c7ab php-fpm`zend_destroy_modules at zend_API.c:2373:2
frame #12: 0x000000000082f6ba php-fpm`zend_shutdown at zend.c:1099:2
frame #13: 0x000000000076d4ae php-fpm`php_module_shutdown at main.c:2436:2
frame #14: 0x0000000000a0cf12 php-fpm`main(argc=1, argv=0x00007fffffffeb78) at fpm_main.c:1982:2
frame #15: 0x000000000041b150 php-fpm`_start(ap=, cleanup=) at crt1.c:76:7
```
We cannot look up the addresses for the offending thread-1 in all the dumps, making it hard to pinpoint the offending code.
After over a week of debugging, I discovered that downgrading to 1.58.0 "solves" my issues.
(and yes, the shared module is compiled with debug symbols)
```
file /usr/local/lib/php/20220829-debug/grpc.so
/usr/local/lib/php/20220829-debug/grpc.so: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, with debug_info, not stripped
```
I do not know how to move forward from here, so any pointers or ideas are welcome.
I cannot reliably reproduce, it just happens randomly.
Contributor guide
Assessment
This issue has not been assessed yet.