[SUPPORT] hudi sql task hang java.lang.System.exit block
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Describe the problem you faced**
The sql task is over, bug the drive can not exit some times .
If the same task is run many times, there is a small chance that it will exit abnormally.
Tens of thousands of tasks are executed every day, and this problem has never occurred for non-hudi spark tasks.
Hudi task have occasionally appeared several times before.



**To Reproduce**
Steps to reproduce the behavior:
1. /usr/lib/spark/bin/spark-sql --name 63130__VOLCANO_JOB_1699949768615_004319 -f /tmp/VOLCANO_JOB_1699949768615_004319.sql --master yarn --queue hadoop --driver-memory 8g --executor-memory 4G --executor-cores 2 --num-executors 8 --packages org.apache.hudi:hudi-spark3.2-bundle_2.12:0.14.0 --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension --conf spark.sql.autoBroadcastJoinThreshold=2G --conf spark.sql.broadcastTimeout=60000 --conf spark.memory.storageFraction=0.7 --conf spark.yarn.priority=5 --conf spark.sql.adaptive.enabled=true
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version :0.14.0
* Spark version :3.2.1
* Hive version :3.1.3
* Hadoop version :3.2.2
* Storage (HDFS/S3/GCS..) :s3
* Running on Docker? (yes/no) :no
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```
Attaching to process ID 8854, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.382-b05
Deadlock Detection:
No deadlocks found.
Thread 23860: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)
Thread 23859: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)
Thread 23858: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)
Thread 23857: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)
Thread 23856: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)
Thread 23855: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)
Thread 23854: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)
Thread 23853: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)
Thread 23852: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
- java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=160, line=460 (Compiled frame)
- java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, boolean, long) @bci=102, line=362 (Compiled frame)
- java.util.concurrent.SynchronousQueue.poll(long, java.util.concurrent.TimeUnit) @bci=11, line=941 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1073 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1134 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=624 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)
Thread 23850: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)
Thread 23849: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)
Thread 23848: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)
Thread 23847: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)
Thread 23846: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)
Thread 23845: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)
Thread 23844: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)
Thread 23843: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)
Thread 23842: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)
Thread 23841: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)
Thread 23840: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)
Thread 23839: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)
Thread 23838: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)
Thread 23837: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)
Thread 23836: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)
Thread 23835: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)
Thread 23834: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.apache.hadoop.ipc.Client$Connection.waitForWork() @bci=59, line=1034 (Compiled frame)
- org.apache.hadoop.ipc.Client$Connection.run() @bci=58, line=1078 (Interpreted frame)
Thread 20557: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)
Thread 20556: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)
Thread 20555: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)
Thread 20554: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)
Thread 20553: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)
Thread 20550: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)
Thread 18356: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)
Thread 18354: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)
Thread 18353: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)
Thread 18351: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)
Thread 18350: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)
Thread 18349: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)
Thread 18348: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)
Thread 18346: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)
Thread 18345: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)
Thread 12596: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)
Thread 12595: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)
Thread 12585: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)
Thread 12580: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)
Thread 12578: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)
Thread 10594: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)
Thread 10593: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)
Thread 10423: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)
Thread 10036: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)
Thread 10035: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)
Thread 10034: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)
Thread 10033: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)
Thread 10032: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)
Thread 10029: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1074 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1134 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=624 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)
Thread 9868: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)
Thread 9866: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)
Thread 9862: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)
Thread 9833: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=78, line=2078 (Compiled frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=124, line=1093 (Compiled frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=809 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1074 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1134 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=624 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)
Thread 9832: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.TimerThread.mainLoop() @bci=201, line=552 (Interpreted frame)
- java.util.TimerThread.run() @bci=1, line=505 (Interpreted frame)
Thread 9736: (state = BLOCKED)
Thread 9346: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)
Thread 9345: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)
Thread 9344: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)
Thread 9343: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)
Thread 9342: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)
Thread 9341: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)
Thread 9340: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)
Thread 9299: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.IdleConnectionReaper.run() @bci=90, line=188 (Interpreted frame)
Thread 9271: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)
Thread 9269: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)
Thread 9056: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(int) @bci=493, line=411 (Interpreted frame)
- org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(org.apache.hadoop.hdfs.client.impl.LeaseRenewer, int) @bci=2, line=76 (Interpreted frame)
- org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run() @bci=69, line=307 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)
Thread 9051: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
- java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=160, line=460 (Compiled frame)
- java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, boolean, long) @bci=102, line=362 (Compiled frame)
- java.util.concurrent.SynchronousQueue.poll(long, java.util.concurrent.TimeUnit) @bci=11, line=941 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1073 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1134 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=624 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)
Thread 9050: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.apache.hadoop.ipc.Client$Connection.waitForWork() @bci=59, line=1034 (Compiled frame)
- org.apache.hadoop.ipc.Client$Connection.run() @bci=58, line=1078 (Interpreted frame)
Thread 8955: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)
Thread 8953: (state = IN_NATIVE_TRANS)
- org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(int, org.apache.hadoop.net.unix.DomainSocketWatcher$FdSet) @bci=0 (Interpreted frame)
- org.apache.hadoop.net.unix.DomainSocketWatcher.access$900(int, org.apache.hadoop.net.unix.DomainSocketWatcher$FdSet) @bci=2, line=52 (Interpreted frame)
- org.apache.hadoop.net.unix.DomainSocketWatcher$2.run() @bci=763, line=503 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)
Thread 8951: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- java.lang.ref.ReferenceQueue.remove(long) @bci=59, line=144 (Compiled frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=165 (Compiled frame)
- org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run() @bci=9, line=3830 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)
Thread 8929: (state = BLOCKED)
Thread 8928: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- java.lang.ref.ReferenceQueue.remove(long) @bci=59, line=144 (Compiled frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=165 (Compiled frame)
- java.lang.ref.Finalizer$FinalizerThread.run() @bci=36, line=188 (Compiled frame)
Thread 8927: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- java.lang.Object.wait() @bci=2, line=502 (Compiled frame)
- java.lang.ref.Reference.tryHandlePending(boolean) @bci=54, line=191 (Compiled frame)
- java.lang.ref.Reference$ReferenceHandler.run() @bci=1, line=153 (Compiled frame)
Thread 8902: (state = BLOCKED)
- java.lang.Shutdown.halt0(int) @bci=0 (Interpreted frame)
- java.lang.Shutdown.halt(int) @bci=7, line=156 (Interpreted frame)
- java.lang.Shutdown.exit(int) @bci=42, line=179 (Interpreted frame)
- java.lang.Runtime.exit(int) @bci=14, line=103 (Interpreted frame)
- java.lang.System.exit(int) @bci=4, line=987 (Interpreted frame)
- org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(java.lang.String[]) @bci=722, line=213 (Interpreted frame)
- org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(java.lang.String[]) @bci=4 (Interpreted frame)
- sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]) @bci=0 (Interpreted frame)
- sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=100, line=62 (Interpreted frame)
- sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=43 (Interpreted frame)
- java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=56, line=498 (Interpreted frame)
- org.apache.spark.deploy.JavaMainApplication.start(java.lang.String[], org.apache.spark.SparkConf) @bci=97, line=52 (Interpreted frame)
- org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(org.apache.spark.deploy.SparkSubmitArguments, boolean) @bci=449, line=1000 (Interpreted frame)
- org.apache.spark.deploy.SparkSubmit.doRunMain$1(org.apache.spark.deploy.SparkSubmitArguments, boolean) @bci=111, line=180 (Interpreted frame)
- org.apache.spark.deploy.SparkSubmit.submit(org.apache.spark.deploy.SparkSubmitArguments, boolean) @bci=65, line=203 (Interpreted frame)
- org.apache.spark.deploy.SparkSubmit.doSubmit(java.lang.String[]) @bci=78, line=90 (Interpreted frame)
- org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(java.lang.String[]) @bci=2, line=1089 (Interpreted frame)
- org.apache.spark.deploy.SparkSubmit$.main(java.lang.String[]) @bci=10, line=1098 (Interpreted frame)
- org.apache.spark.deploy.SparkSubmit.main(java.lang.String[]) @bci=4 (Interpreted frame)
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the provided spark-sql reproduction command and the Java/Netty thread dump, using the listed Hudi 0.14.0 and Spark 3.2.1 environment. Investigate why repeated Hudi SQL tasks sometimes fail to terminate; done means the task exits normally and reliably without the reported hang.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100