iovisor / iovisor/bcc

bcc hangs when probing malloc

Open
#2,046 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
22.7k
Forks
4.1k
Avg merge
10d 4h
Merged PRs (30d)
3

Description

I'm trying to get some better information about how much time my programs are spending in malloc, how frequently they call malloc and free, and how big their allocations are.

I have a small bcc script that installs BPF uprobes on malloc and free, to collect and summarize this data, and write it to disk as frequently as possible.

When I run `stress -m 200 --vm-bytes 1024` (or the program I'm trying to measure), my bcc script hangs while trying to read or write the BPF tables:

```
^C
Traceback (most recent call last):
File "./bpf_malloc.py", line 84, in main
alloc_stats['malloc'][tid.value] = malloc_calls[tid]
KeyboardInterrupt
```

My full script is:

```
#!/usr/bin/env python

from __future__ import print_function

import bcc
import json
import time
import traceback

BPF_PROG="""
struct malloc_info {
u64 total_duration;
u64 total_size;
u64 count;
};

BPF_HASH(malloc_start, pid_t, u64);
BPF_HASH(malloc_calls, pid_t, struct malloc_info);
BPF_HASH(free_calls, pid_t, u64);

int probe_malloc(struct pt_regs *ctx) {
u64 time = bpf_ktime_get_ns();
size_t size = PT_REGS_PARM1(ctx);
u32 tgid = (bpf_get_current_pid_tgid() >> 32);
struct malloc_info malloc_zero = {
.total_duration = 0,
.total_size = 0,
.count = 0
};

malloc_start.update(&tgid, &time);
struct malloc_info *malloc = malloc_calls.lookup_or_init(&tgid, &malloc_zero);
malloc->total_size += size;
malloc->count++;
return 0;
}

int probe_malloc_ret(struct pt_regs *ctx) {
u64 time = bpf_ktime_get_ns();
u32 tgid = (bpf_get_current_pid_tgid() >> 32);
u64 *start = malloc_start.lookup(&tgid);
if(start) {
struct malloc_info malloc_zero = {
.total_duration = 0,
.total_size = 0,
.count = 0
};
struct malloc_info *malloc = malloc_calls.lookup_or_init(&tgid, &malloc_zero);
malloc->total_duration += (time - *start);
}
return 0;
}

int probe_free(struct pt_regs *ctx) {
u32 tgid = (bpf_get_current_pid_tgid() >> 32);
free_calls.increment(tgid);
return 0;
}
"""

def main():
b = bcc.BPF(text=BPF_PROG)

# Attach probes for malloc
b.attach_uprobe(name="c", sym="malloc", fn_name='probe_malloc')
b.attach_uretprobe(name="c", sym="malloc", fn_name='probe_malloc_ret')
b.attach_uprobe(name="c", sym="free", fn_name='probe_free')

alloc_file = open('allocs.json', 'w')
alloc_file.write('[\n')

malloc_calls = b.get_table('malloc_calls')
free_calls = b.get_table('free_calls')

start = time.time()
i = 0;
try:
while True:
alloc_stats = {'t': time.time(), 'malloc': {}, 'free': {}}
for tid in malloc_calls:
alloc_stats['malloc'][tid.value] = malloc_calls[tid]
# Zero out the counter the same way that `malloc_calls.zero()` would
# this attempts to avoid a race condition with the BPF program
malloc_calls[tid] = malloc_calls.Leaf()

for tid in free_calls:
alloc_stats['free'][tid.value] = free_calls[tid].value
free_calls[tid] = free_calls.Leaf()

for tid in alloc_stats['malloc']:
alloc_stats['malloc'][tid] = { field: getattr(alloc_stats['malloc'][tid], field) for field, _ in alloc_stats['malloc'][tid]._fields_ }

if i != 0:
alloc_file.write(',\n')
alloc_file.write(json.dumps(alloc_stats))

# Sleep for remaining time in interval.
start += 0.1
remaining_time = start - time.time()
if remaining_time > 0:
time.sleep(remaining_time)
print("Slack time {:.3f} seconds".format(remaining_time))

i += 1
except KeyboardInterrupt:
traceback.print_exc()

alloc_file.write(']\n')
alloc_file.close()

if __name__ == '__main__':
main()
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the full bpf_malloc.py script and reproduce the hang using `stress -m 200 --vm-bytes 1024`. Read the malloc/free uprobes and the loops reading and resetting the `malloc_calls` and `free_calls` BPF tables, then determine why table access does not complete under load. Done means the script continues reading and writing its JSON output while probing the workload.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, python
Domain
devtools, observability, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.