fullstaq-ruby / fullstaq-ruby/server-edition
Missing debugging symbols
- Dominant language
- Shell
- Stars
- 630
- Forks
- 38
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the bug**
We're trying to diagnose some hung processes. We're able to use signal handling to get ruby backtraces (ie sidekiq's TTIN handler), but the problem seems to be one level deeper, ie C. The processes are multithreaded, so we need to get the state of every
SIGABRT/SIGEV are able to trigger ruby dumping it's own
These signals (and others) do also trigger a kernel core dump. If I try to analyze the `core` with `gdb` after the fact, it doesn't appear there are any debugging symbols present:
```
Reading symbols from /usr/lib/fullstaq-ruby/versions/3.2.2-jemalloc/bin/ruby...
(No debugging symbols found in /usr/lib/fullstaq-ruby/versions/3.2.2-jemalloc/bin/ruby)
```
**To Reproduce**
Linux distribution: Ubuntu 22.04
Package name: `fullstaq-ruby-3.2.2`
Steps to reproduce the behavior:
1. Enable dumping cores: `ulimit -c unlimited`
2. Run command `ruby -e 'puts Process.pid; sleep 10000' &`, then `kill -s ABRT `
3. `gdb $(which ruby) core` (assuming `kernel.core_pattern=core`)
Note: this is running an x86_64 docker image on an M1, so I a fair amount of output is due to that rather than debugging symbols.
~~~
Reading symbols from /usr/lib/fullstaq-ruby/versions/3.2.2-jemalloc/bin/ruby...
(No debugging symbols found in /usr/lib/fullstaq-ruby/versions/3.2.2-jemalloc/bin/ruby)
warning: malformed note - filename area is too big
warning: Can't open file @ which was expanded to @ during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file `@ which was expanded to `@ during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file ( which was expanded to ( during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file `@ which was expanded to `@ during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file �!@ which was expanded to �!@ during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file � which was expanded to � during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file �!@ which was expanded to �!@ during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file "@ which was expanded to "@ during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file "@ which was expanded to "@ during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file @"@ which was expanded to @"@ during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file 0#@ which was expanded to 0#@ during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file `#@ which was expanded to `#@ during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file `#@ which was expanded to `#@ during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file �$@ which was expanded to �$@ during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file �$@ which was expanded to �$@ during file-backed mapping note processing
warning: core file may not match specified executable file.
warning: Selected architecture i386:x86-64 is not compatible with reported target architecture i386
warning: Architecture rejected target-supplied description
warning: Couldn't find general-purpose registers in core file.
warning: Unexpected size of section `.reg2' in core file.
Core was generated by `/usr/bin/qemu-x86_64 /usr/lib/fullstaq-ruby/versions/3.2.2-jemalloc/bin/ruby ru'.
warning: Couldn't find general-purpose registers in core file.
warning: Unexpected size of section `.reg2' in core file.
#0 in ?? ()
(gdb)
~~~
**Expected behavior**
After opening the core, being able to run `bt` and see the backtrace. This is what it looks like on the vanilla `ruby:3.2.2` docker image:
```
GNU gdb (Debian 10.1-1.7) 10.1.90.20210103-git
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/bin/ruby...
[New LWP 370]
[New LWP 371]
[New LWP 373]
[New LWP 372]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
Core was generated by `ruby reproducible-case.rb'.
Program terminated with signal SIGABRT, Aborted.
#0 0x0000fffface5b3c8 in __ppoll (fds=fds@entry=0xfffff4c4d190, nfds=nfds@entry=1, timeout=, timeout@entry=0x0, sigmask=sigmask@entry=0x0)
at ../sysdeps/unix/sysv/linux/ppoll.c:44
44 ../sysdeps/unix/sysv/linux/ppoll.c: No such file or directory.
[Current thread is 1 (Thread 0xffffacd6da00 (LWP 370))]
(gdb) bt
#0 0x0000fffface5b3c8 in __ppoll (fds=fds@entry=0xfffff4c4d190, nfds=nfds@entry=1, timeout=, timeout@entry=0x0, sigmask=sigmask@entry=0x0)
at ../sysdeps/unix/sysv/linux/ppoll.c:44
#1 0x0000ffffad3ee258 in rb_sigwait_sleep (th=th@entry=0xaaaae7b964f0, sigwait_fd=sigwait_fd@entry=3, rel=rel@entry=0x0) at hrtime.h:158
#2 0x0000ffffad3ef6c8 [PAC] in native_sleep (th=th@entry=0xaaaae7b964f0, rel=0x0) at thread_pthread.c:2347
#3 0x0000ffffad3f3090 [PAC] in thread_join_sleep (arg=arg@entry=281474788283432) at thread.c:1056
#4 0x0000ffffad263d10 [PAC] in rb_ensure (b_proc=b_proc@entry=0xffffad3f2e90 , data1=data1@entry=281474788283432,
e_proc=e_proc@entry=0xffffad3e7ee0 , data2=data2@entry=281474788283432) at eval.c:1007
#5 0x0000ffffad3eb0c8 [PAC] in thread_join (limit=0x0, timeout=4, target_th=0xaaaae7fa66f0) at thread.c:1105
#6 thread_join_m (argc=, argv=, self=) at thread.c:1202
#7 0x0000ffffad444770 [PAC] in vm_call0_cfunc_with_frame (argv=0xfffff4c4d650, calling=0xfffff4c4d4c8, ec=0xaaaae7b97680) at vm_eval.c:150
#8 vm_call0_cfunc (argv=0xfffff4c4d650, calling=0xfffff4c4d4c8, ec=0xaaaae7b97680) at vm_eval.c:164
#9 vm_call0_body (ec=ec@entry=0xaaaae7b97680, calling=calling@entry=0xfffff4c4d598, argv=argv@entry=0xfffff4c4d650) at vm_eval.c:210
#10 0x0000ffffad4462c8 [PAC] in vm_call0_cc (kw_splat=0, cc=0xffffa8466ad8, argv=0xfffff4c4d650, argc=0, id=13617, recv=281473504931160, ec=0xaaaae7b97680) at vm_eval.c:87
#11 rb_call0 (ec=0xaaaae7b97680, recv=281473504931160, mid=13617, argc=argc@entry=0, argv=argv@entry=0xfffff4c4d650, call_scope=call_scope@entry=CALL_PUBLIC, self=281473504931360)
at vm_eval.c:551
#12 0x0000ffffad446fcc [PAC] in rb_call (scope=CALL_PUBLIC, argv=0xfffff4c4d650, argc=0, mid=, recv=) at vm_eval.c:877
#13 0x0000ffffad3d9524 in rb_sym_proc_call (mid=, argc=argc@entry=1, argv=argv@entry=0xfffff4c4d648, kw_splat=kw_splat@entry=0, passed_proc=passed_proc@entry=4)
at string.c:11547
#14 0x0000ffffad4439dc in vm_yield_with_symbol (block_handler=0, kw_splat=0, argv=0xfffff4c4d648, argc=1, symbol=, ec=0xaaaae7b97680) at vm_insnhelper.c:4297
#15 invoke_block_from_c_bh (force_blockarg=, is_lambda=, cref=, passed_block_handler=, kw_splat=,
argv=, argc=, block_handler=, ec=) at vm.c:1481
#16 vm_yield_with_cref (is_lambda=0, cref=0x0, kw_splat=0, argv=0xfffff4c4d648, argc=1, ec=0xaaaae7b97680) at vm.c:1509
#17 vm_yield (kw_splat=0, argv=0xfffff4c4d648, argc=1, ec=0xaaaae7b97680) at vm.c:1517
#18 rb_yield_0 (argv=0xfffff4c4d648, argc=1) at vm_eval.c:1348
#19 rb_yield (val=) at vm_eval.c:1364
#20 0x0000ffffad1c6bd4 [PAC] in rb_ary_each (ary=281473504931360) at array.c:2740
#21 0x0000ffffad42a120 [PAC] in vm_call_cfunc_with_frame (ec=0xaaaae7b97680, reg_cfp=0xffffacd16f90, calling=) at vm_insnhelper.c:3268
#22 0x0000ffffad4300f8 [PAC] in vm_sendish (ec=0xaaaae7b97680, reg_cfp=0xffffacd16f90, cd=0xaaaae7c522b0, block_handler=, method_explorer=mexp_search_method)
at vm_callinfo.h:367
#23 0x0000ffffad439970 [PAC] in vm_exec_core (ec=0xaaaae7b97680, initial=1, initial@entry=0) at insns.def:801
#24 0x0000ffffad43f4fc [PAC] in rb_vm_exec (ec=0xaaaae7b97680, jit_enable_p=true) at vm.c:2383
#25 0x0000ffffad25f54c [PAC] in rb_ec_exec_node (ec=ec@entry=0xaaaae7b97680, n=n@entry=0xffffacc00b00) at eval.c:289
#26 0x0000ffffad2659bc [PAC] in ruby_run_node (n=0xffffacc00b00) at eval.c:330
#27 0x0000aaaab8f90b9c [PAC] in rb_main (argv=0xfffff4c4dd68, argc=2) at ./main.c:38
#28 main (argc=, argv=) at ./main.c:57
```
**Additional context**
Interestingly, I am able to get C level backtrace when using `kill -s SEGV`. However, it seems to only include the backtrace of the thread responding, not multiple threads.
Contributor guide
Assessment
This issue has not been assessed yet.