iovisor / iovisor/bcc

0.19.0: offwaketime: user stacks of hot codepaths is missed (-17)

Open
#3,389 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
22.7k
Forks
4.1k
Avg merge
10d 4h
Merged PRs (30d)
3

Description

```
git clone git://github.com/asheplyakov/profilingmulticore.git
git checkout 20210429
make bin/thunderingherd
sudo /bin/true && ./bin/thunderingherd & pid=$!; echo "tracing whole system"; sudo env PYTHONPATH=/opt/bcc/lib/python3/dist-packages LD_LIBRARY_PATH=/opt/bcc/lib /opt/bcc/share/bcc/tools/offwaketime -f --stack-storage-size=$((1024*1024)) 8 > stacks; echo "finished tracing; waiting for $pid"; wait $pid
```

```
[1] 638766
tracing whole system
6 worker threads
producer: message period 10 usec
workers: service time 1 usec
ETA: 10 sec
WARNING: 6 stack traces lost and could not be displayed.
finished tracing; waiting for 638766
Actual time: 14404 msec
[1]+ Done sudo /bin/true && ./bin/thunderingherd
```

Those 6 missing stacks are hot code paths (with heavy lock contention), and capturing them is the whole point of the exercise

```
tworker_4;[Missed User Stack] -17;entry_SYSCALL_64_after_hwframe;do_syscall_64;__x64_sys_futex;do_futex;futex_wait;futex_wait_queue_me;schedule;--;futex_wake;do_futex;__x64_sys_futex;do_sysc
all_64;entry_SYSCALL_64_after_hwframe;tproducer 559361
tworker_1;[Missed User Stack] -17;entry_SYSCALL_64_after_hwframe;do_syscall_64;__x64_sys_futex;do_futex;futex_wait;futex_wait_queue_me;schedule;--;futex_wake;do_futex;__x64_sys_futex;do_sysc
all_64;entry_SYSCALL_64_after_hwframe;tproducer 561615
tworker_3;[Missed User Stack] -17;entry_SYSCALL_64_after_hwframe;do_syscall_64;__x64_sys_futex;do_futex;futex_wait;futex_wait_queue_me;schedule;--;futex_wake;do_futex;__x64_sys_futex;do_sysc
all_64;entry_SYSCALL_64_after_hwframe;tproducer 562282
tworker_0;[Missed User Stack] -17;entry_SYSCALL_64_after_hwframe;do_syscall_64;__x64_sys_futex;do_futex;futex_wait;futex_wait_queue_me;schedule;--;futex_wake;do_futex;__x64_sys_futex;do_sysc
all_64;entry_SYSCALL_64_after_hwframe;tproducer 563466
tworker_5;[Missed User Stack] -17;entry_SYSCALL_64_after_hwframe;do_syscall_64;__x64_sys_futex;do_futex;futex_wait;futex_wait_queue_me;schedule;--;futex_wake;do_futex;__x64_sys_futex;do_sysc
all_64;entry_SYSCALL_64_after_hwframe;tproducer 564472
tworker_2;[Missed User Stack] -17;entry_SYSCALL_64_after_hwframe;do_syscall_64;__x64_sys_futex;do_futex;futex_wait;futex_wait_queue_me;schedule;--;futex_wake;do_futex;__x64_sys_futex;do_sysc
all_64;entry_SYSCALL_64_after_hwframe;tproducer 570661
```

Ubuntu 20.10, kernel 5.8, llvm-8-dev, libclang-8-dev

Note:

- I've set a very big stack storage size (1024x the default one), yet some the "interesting" stacks are missed anyway
- `-17` is `-EEXISTS`, `bpf_get_stackid` returns `-EEXISTS` if (and only if) [two different stacks end up in the same hash map bucket](https://github.com/torvalds/linux/blob/v5.8/kernel/bpf/stackmap.c#L417). I've tried adding `BPF_F_REUSE_STACKID` ([over here](https://github.com/iovisor/bcc/blob/v0.19.0/tools/offwaketime.py#L273)). This way there are no missed stacks, instead they become `[unknown]` (not any better).

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.