Possibly a double free in ProcSyms::resolve_addr (aarch64, multi-threaded program)
- Dominant language
- C
- Stars
- 22.7k
- Forks
- 4.1k
- Avg merge
- 10d 4h
- Merged PRs (30d)
- 3
Description
Hi folks,
I've come across an issue where my eBPF program occasionally crashes with the following error:
```
double free or corruption (fasttop)
malloc_consolidate(): invalid chunk size
double free or corruption (out)
```
In my program I read from various BPF maps from multiple threads, among those maps is BPF_STACK_TRACE, which may be causing the issue. BCC is built from source (last commit is 7c02284faa2968d86720f416ea51cc74d446ab37) for aarch64.
I tried running it in gdb and observed the following backtrace:
```
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x0000007fbf414ea0 in __GI_abort () at abort.c:79
#2 0x0000007fbf462280 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7fbf526070 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#3 0x0000007fbf4697dc in malloc_printerr (str=str@entry=0x7fbf521ba0 "double free or corruption (fasttop)") at malloc.c:5347
#4 0x0000007fbf46a950 in _int_free (av=, p=0x7fa401d7b0, have_lock=0) at malloc.c:4266
#5 0x0000007fbbc168bc in ProcSyms::resolve_addr(unsigned long, bcc_symbol*, bool) () from /lib/aarch64-linux-gnu/libbcc.so.0
#6 0x0000007fbbc143a8 in bcc_symcache_resolve () from /lib/aarch64-linux-gnu/libbcc.so.0
#7 0x0000007fbf242048 in ?? () from /lib/aarch64-linux-gnu/libffi.so.7
#8 0x0000007fbf241770 in ?? () from /lib/aarch64-linux-gnu/libffi.so.7
#9 0x0000007fbeff3b68 in ?? () from /usr/lib/python3.9/lib-dynload/_ctypes.cpython-39-aarch64-linux-gnu.so
#10 0x0000007fbeff2c6c in ?? () from /usr/lib/python3.9/lib-dynload/_ctypes.cpython-39-aarch64-linux-gnu.so
#11 0x00000000004a5300 in _PyObject_MakeTpCall ()
#12 0x000000000049c900 in _PyEval_EvalFrameDefault ()
*** redacted for brevity ***
```
From this backtrace I conjecture that it might be some race condition when freeing, although I'm not sure where exactly does the freeing take place in `ProcSyms::resolve_addr`, perhaps when it calls [refresh](https://github.com/iovisor/bcc/blob/eab81d91290a6987f87f34366a963e2d6a7fcb40/src/cc/bcc_syms.cc#L171)? (But then that means `refresh` was inlined since it doesn't show in the trace)
It seems to me that `bcc_symcache_resolve` is only called by `BPFStackTable::get_stack_symbol` and in `__init__.py`. The crash happens after bcc has been imported so the only culprit is `BPFStackTable::get_stack_symbol` so I suspect it happens when I'm trying to walk the stack by calling `stack_trace.walk(stack_id)`.
I would greatly appreciate if anybody could help me further debug this issue and fix it (without going single-threaded).
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.