iovisor / iovisor/bcc

Possibly a double free in ProcSyms::resolve_addr (aarch64, multi-threaded program)

Open
#3,985 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
22.7k
Forks
4.1k
Avg merge
10d 4h
Merged PRs (30d)
3

Description

Hi folks,

I've come across an issue where my eBPF program occasionally crashes with the following error:

```
double free or corruption (fasttop)
malloc_consolidate(): invalid chunk size
double free or corruption (out)
```

In my program I read from various BPF maps from multiple threads, among those maps is BPF_STACK_TRACE, which may be causing the issue. BCC is built from source (last commit is 7c02284faa2968d86720f416ea51cc74d446ab37) for aarch64.

I tried running it in gdb and observed the following backtrace:

```
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x0000007fbf414ea0 in __GI_abort () at abort.c:79
#2 0x0000007fbf462280 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7fbf526070 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#3 0x0000007fbf4697dc in malloc_printerr (str=str@entry=0x7fbf521ba0 "double free or corruption (fasttop)") at malloc.c:5347
#4 0x0000007fbf46a950 in _int_free (av=, p=0x7fa401d7b0, have_lock=0) at malloc.c:4266
#5 0x0000007fbbc168bc in ProcSyms::resolve_addr(unsigned long, bcc_symbol*, bool) () from /lib/aarch64-linux-gnu/libbcc.so.0
#6 0x0000007fbbc143a8 in bcc_symcache_resolve () from /lib/aarch64-linux-gnu/libbcc.so.0
#7 0x0000007fbf242048 in ?? () from /lib/aarch64-linux-gnu/libffi.so.7
#8 0x0000007fbf241770 in ?? () from /lib/aarch64-linux-gnu/libffi.so.7
#9 0x0000007fbeff3b68 in ?? () from /usr/lib/python3.9/lib-dynload/_ctypes.cpython-39-aarch64-linux-gnu.so
#10 0x0000007fbeff2c6c in ?? () from /usr/lib/python3.9/lib-dynload/_ctypes.cpython-39-aarch64-linux-gnu.so
#11 0x00000000004a5300 in _PyObject_MakeTpCall ()
#12 0x000000000049c900 in _PyEval_EvalFrameDefault ()
*** redacted for brevity ***
```

From this backtrace I conjecture that it might be some race condition when freeing, although I'm not sure where exactly does the freeing take place in `ProcSyms::resolve_addr`, perhaps when it calls [refresh](https://github.com/iovisor/bcc/blob/eab81d91290a6987f87f34366a963e2d6a7fcb40/src/cc/bcc_syms.cc#L171)? (But then that means `refresh` was inlined since it doesn't show in the trace)

It seems to me that `bcc_symcache_resolve` is only called by `BPFStackTable::get_stack_symbol` and in `__init__.py`. The crash happens after bcc has been imported so the only culprit is `BPFStackTable::get_stack_symbol` so I suspect it happens when I'm trying to walk the stack by calling `stack_trace.walk(stack_id)`.

I would greatly appreciate if anybody could help me further debug this issue and fix it (without going single-threaded).

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.