offwaketime does not show wakers of Java threads
- Dominant language
- C
- Stars
- 22.7k
- Forks
- 4.1k
- Avg merge
- 10d 4h
- Merged PRs (30d)
- 3
Description
When I profile a Java workload with `offwaketime`, wakers of Java threads become empty.
Component | Version
------------ | -------------
BCC tools | 0.10.0-1 (Upstream Ubuntu packages from repo.iovisor.org)
OS | Ubuntu 18.04.2 (Linux 4.15.0-43-generic)
Java | openjdk 12.0.1 2019-04-16
OpenJDK Runtime Environment AdoptOpenJDK (build 12.0.1+12)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 12.0.1+12, mixed mode, sharing)
perf-map-agent | Latest version (d9843a0)
https://github.com/jvm-profiling-tools/perf-map-agent
Below is the example pair of target and waker. The target contains frames of JIT-compiled code, but the waker is empty. Even frames in kernel are not included in the waker.
```
waker: 10904
-- --
finish_task_switch
schedule
exit_to_usermode_loop
prepare_exit_to_usermode
swapgs_restore_regs_and_return_to_usermode
ObjectSynchronizer::inflate(Thread*, oopDesc*, ObjectSynchronizer::InflateCause)
SharedRuntime::complete_monitor_locking_C(oopDesc*, BasicLock*, JavaThread*)
_complete_monitor_locking_Java
Lcom/ibm/websphere/jaxrs/server/IBMRestServlet;::doPost
Lcom/ibm/ws/webcontainer/servlet/ServletWrapper;::service
Lcom/ibm/ws/webcontainer/servlet/ServletWrapper;::handleRequest
Lcom/ibm/ws/webcontainer/filter/WebAppFilterChain;::invokeTarget
Lcom/ibm/ws/webcontainer/filter/WebAppFilterChain;::doFilter
Lcom/ibm/ws/webcontainer/filter/FilterInstanceWrapper;::doFilter
Lcom/ibm/ws/webcontainer/filter/WebAppFilterChain;::doFilter
Lcom/ibm/ws/webcontainer/filter/WebAppFilterManager;::doFilter
Lcom/ibm/ws/webcontainer/filter/WebAppFilterManager;::invokeFilters
Lcom/ibm/ws/webcontainer/servlet/CacheServletWrapper;::handleRequest
Lcom/ibm/ws/webcontainer/WebContainer;::handleRequest
Lcom/ibm/ws/webcontainer/osgi/DynamicVirtualHost$2;::run
Lcom/ibm/ws/http/dispatcher/internal/channel/HttpDispatcherLink;::ready
Lcom/ibm/ws/http/channel/internal/inbound/HttpInboundLink;::handleDiscrimination
Lcom/ibm/ws/http/channel/internal/inbound/HttpICLReadCallback;::complete
Lcom/ibm/ws/tcpchannel/internal/WorkQueueManager;::attemptIO
Lcom/ibm/ws/tcpchannel/internal/WorkQueueManager;::workerRun
Ljava/util/concurrent/ThreadPoolExecutor;::runWorker
Ljava/util/concurrent/ThreadPoolExecutor$Worker;::run
Ljava/lang/Thread;::run
call_stub
JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)
JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, Thread*)
thread_entry(JavaThread*, Thread*)
JavaThread::run()
Thread::call_run()
thread_native_entry(Thread*)
start_thread
target: Default Executo 0
20
```
Below is another pair of waker and target in the same profile result. In this case, both waker and target frames are recorded, but the waker is not a Java thread. It looks like target only contains kernel frames.
```
waker: VM Thread 11644
start_thread
thread_native_entry(Thread*)
Thread::call_run()
VMThread::run()
VMThread::loop()
SafepointSynchronize::begin()
SafepointSynchronize::do_cleanup_tasks()
WorkGang::run_task(AbstractGangTask*)
SemaphoreGangTaskDispatcher::coordinator_execute_on_workers(AbstractGangTask*, unsigned int)
entry_SYSCALL_64_after_hwframe
do_syscall_64
sys_futex
do_futex
futex_wake
-- --
finish_task_switch
schedule
futex_wait_queue_me
futex_wait
do_futex
sys_futex
do_syscall_64
entry_SYSCALL_64_after_hwframe
do_futex_wait.constprop.1
target: GC Thread#6 11619
2618955
```
I am not sure this issue is specific to Java or not. Could anyone please advise how to diagnose this issue?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the offwaketime entry point and the perf-map-agent setup described in the report. Reproduce the Java profile and compare stack collection for Java wakers, kernel frames, and the VM Thread example; done means the cause is identified and waker stacks are recorded consistently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, linux
- Domain
- observability-sre, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100