iovisor / iovisor/bcc

Issues in the memleak tool

Open
#3,341 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
22.7k
Forks
4.1k
Avg merge
10d 4h
Merged PRs (30d)
3

Description

As a side-item to https://github.com/iovisor/bcc/issues/3331 (where I discuss contributing a tool I made related to `memleak`), I was asked by Brendan to list out some of the issues I've encountered with the `memleak` tool.

**Correctness issue**
With `--combined-only` on, the statistics suffer from race conditions. Example use case:
```
#include
#include
#include
#include

extern void* malloc(size_t);

int random_preempt_work() {
static unsigned int rng = 0;
for (int i = 0; i < 1000000; i++) {
rng = 1664525 * rng + 1013904223;
int *data = new int; /*** The only dynamic allocation ***/
if ((rng & ((1U << 7) - 1U)) == 0) {
std::this_thread::yield();
}
if ((rng & ((1U << 14) - 1U)) == 0) {
usleep((rng >> 14) & ((1U << 13) - 1U));
}
delete data; /*** The free event for that allocation ***/
}
}

int main() {
std::cout << "Sleeping at the beginning" << std::endl;
sleep(10);

std::cout << "Starting single threaded work" << std::endl;
random_preempt_work();
std::cout << "Finished single threaded work" << std::endl;
sleep(10);

std::cout << "Starting multi threaded work" << std::endl;
{
std::thread threads[30];
for (int j = 0; j < 30; j++) {
threads[j] = std::thread(random_preempt_work);
}
for (int j = 0; j < 30; j++) {
threads[j].join();
}
}
std::cout << "Finished multi threaded work" << std::endl;
sleep(20);
return 0;
}
```
The output sometimes become:
```
[02:02:06] Top 10 stacks with outstanding allocations:
0 bytes in 0 allocations from stack
operator new(unsigned long)+0x1d [libstdc++.so.6.0.19]
main+0x52 [demo_allocations]
__libc_start_main+0xf5 [libc-2.17.so]
0 bytes in 0 allocations from stack
operator new(unsigned long)+0x1d [libstdc++.so.6.0.19]
__libc_start_main+0xf5 [libc-2.17.so]
8 bytes in 2 allocations from stack
operator new(unsigned long)+0x1d [libstdc++.so.6.0.19]
[unknown] [libstdc++.so.6.0.19]
2304 bytes in 4 allocations from stack
_dl_allocate_tls+0x25 [ld-2.17.so]
```
On my machine, the race conditions usually push toward missed allocations rather than missed frees, so usually the leak report is empty as one might expect. But it's only because the stat updates truncate at zero, if I modify the script so that the aggregated stats are not truncated to zero when they become negative, then the leak report contains many negative numbers.

**Usability issues**
1. The `--combined-only` option has a misleading name. The reported stats are aggregated with or without this option. This option instead controls whether the statistics are aggregated in user space or on the eBPF map level.
2. When the application that is being analyzed terminates, the stacks traces become `[unknown]`. This makes `memleak` very difficult to use for transient processes.
3. The tool silently fails if the application being analyzed makes allocations from enough places to saturate the `stack_traces` map. Similar problems happen if there are enough live allocations to saturate the `allocs` map. The output stats make an application to appear to have no leaks, or fewer / smaller leaks than in reality, and there is no way to know from the output that this kind of problem has happened.

**Miscellaneous**
1. This is just my opinion, but I found it hard to use the output stack traces effectively since they report binary offsets into functions rather than line numbers. I end up manually feeding many entries into `addr2line`.
2. The source file has pep8 violations, mostly related to indentation.
3. Also just my opinion, but I found it weird that there is a default command line option that allocations need to be at least 500ms old.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.