iovisor / iovisor/bcc

offwaketime does not show wakers of Java threads

Open
#2,487 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
22.7k
Forks
4.1k
Avg merge
10d 4h
Merged PRs (30d)
3

Description

When I profile a Java workload with `offwaketime`, wakers of Java threads become empty.

Component | Version
------------ | -------------
BCC tools | 0.10.0-1 (Upstream Ubuntu packages from repo.iovisor.org)
OS | Ubuntu 18.04.2 (Linux 4.15.0-43-generic)
Java | openjdk 12.0.1 2019-04-16
OpenJDK Runtime Environment AdoptOpenJDK (build 12.0.1+12)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 12.0.1+12, mixed mode, sharing)
perf-map-agent | Latest version (d9843a0)
https://github.com/jvm-profiling-tools/perf-map-agent

Below is the example pair of target and waker. The target contains frames of JIT-compiled code, but the waker is empty. Even frames in kernel are not included in the waker.

```
waker: 10904
-- --
finish_task_switch
schedule
exit_to_usermode_loop
prepare_exit_to_usermode
swapgs_restore_regs_and_return_to_usermode
ObjectSynchronizer::inflate(Thread*, oopDesc*, ObjectSynchronizer::InflateCause)
SharedRuntime::complete_monitor_locking_C(oopDesc*, BasicLock*, JavaThread*)
_complete_monitor_locking_Java
Lcom/ibm/websphere/jaxrs/server/IBMRestServlet;::doPost
Lcom/ibm/ws/webcontainer/servlet/ServletWrapper;::service
Lcom/ibm/ws/webcontainer/servlet/ServletWrapper;::handleRequest
Lcom/ibm/ws/webcontainer/filter/WebAppFilterChain;::invokeTarget
Lcom/ibm/ws/webcontainer/filter/WebAppFilterChain;::doFilter
Lcom/ibm/ws/webcontainer/filter/FilterInstanceWrapper;::doFilter
Lcom/ibm/ws/webcontainer/filter/WebAppFilterChain;::doFilter
Lcom/ibm/ws/webcontainer/filter/WebAppFilterManager;::doFilter
Lcom/ibm/ws/webcontainer/filter/WebAppFilterManager;::invokeFilters
Lcom/ibm/ws/webcontainer/servlet/CacheServletWrapper;::handleRequest
Lcom/ibm/ws/webcontainer/WebContainer;::handleRequest
Lcom/ibm/ws/webcontainer/osgi/DynamicVirtualHost$2;::run
Lcom/ibm/ws/http/dispatcher/internal/channel/HttpDispatcherLink;::ready
Lcom/ibm/ws/http/channel/internal/inbound/HttpInboundLink;::handleDiscrimination
Lcom/ibm/ws/http/channel/internal/inbound/HttpICLReadCallback;::complete
Lcom/ibm/ws/tcpchannel/internal/WorkQueueManager;::attemptIO
Lcom/ibm/ws/tcpchannel/internal/WorkQueueManager;::workerRun
Ljava/util/concurrent/ThreadPoolExecutor;::runWorker
Ljava/util/concurrent/ThreadPoolExecutor$Worker;::run
Ljava/lang/Thread;::run
call_stub
JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)
JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, Thread*)
thread_entry(JavaThread*, Thread*)
JavaThread::run()
Thread::call_run()
thread_native_entry(Thread*)
start_thread
target: Default Executo 0
20
```

Below is another pair of waker and target in the same profile result. In this case, both waker and target frames are recorded, but the waker is not a Java thread. It looks like target only contains kernel frames.

```
waker: VM Thread 11644
start_thread
thread_native_entry(Thread*)
Thread::call_run()
VMThread::run()
VMThread::loop()
SafepointSynchronize::begin()
SafepointSynchronize::do_cleanup_tasks()
WorkGang::run_task(AbstractGangTask*)
SemaphoreGangTaskDispatcher::coordinator_execute_on_workers(AbstractGangTask*, unsigned int)
entry_SYSCALL_64_after_hwframe
do_syscall_64
sys_futex
do_futex
futex_wake
-- --
finish_task_switch
schedule
futex_wait_queue_me
futex_wait
do_futex
sys_futex
do_syscall_64
entry_SYSCALL_64_after_hwframe
do_futex_wait.constprop.1
target: GC Thread#6 11619
2618955
```

I am not sure this issue is specific to Java or not. Could anyone please advise how to diagnose this issue?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the offwaketime entry point and the perf-map-agent setup described in the report. Reproduce the Java profile and compare stack collection for Java wakers, kernel frames, and the VM Thread example; done means the cause is identified and waker stacks are recorded consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, linux
Domain
observability-sre, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.