apache / apache/hudi

[SUPPORT] hudi sql task hang java.lang.System.exit block

Open
#10,112 4 comments 0 reactions 0 assignees View on GitHub
area:writer engine:spark
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**
The sql task is over, bug the drive can not exit some times .

If the same task is run many times, there is a small chance that it will exit abnormally.

Tens of thousands of tasks are executed every day, and this problem has never occurred for non-hudi spark tasks.

Hudi task have occasionally appeared several times before.

![企业微信截图_28aec49f-d1c0-45d0-b9e0-dc1f31b21ee0](https://github.com/apache/hudi/assets/15028279/16c6762f-afde-47ee-ac12-bb2d2c590f45)

![企业微信截图_4dcf7b0c-1c6b-44b7-99e7-d8d5134434a8](https://github.com/apache/hudi/assets/15028279/6b156a0f-43c4-4fa4-adaa-b75458f16a3f)

![企业微信截图_b8884f5e-ff16-4115-a826-f5a50b281df9](https://github.com/apache/hudi/assets/15028279/2b79c34c-694b-49ca-839b-65c7cd2c4769)

**To Reproduce**

Steps to reproduce the behavior:

1. /usr/lib/spark/bin/spark-sql --name 63130__VOLCANO_JOB_1699949768615_004319 -f /tmp/VOLCANO_JOB_1699949768615_004319.sql --master yarn --queue hadoop --driver-memory 8g --executor-memory 4G --executor-cores 2 --num-executors 8 --packages org.apache.hudi:hudi-spark3.2-bundle_2.12:0.14.0 --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension --conf spark.sql.autoBroadcastJoinThreshold=2G --conf spark.sql.broadcastTimeout=60000 --conf spark.memory.storageFraction=0.7 --conf spark.yarn.priority=5 --conf spark.sql.adaptive.enabled=true

**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment Description**

* Hudi version :0.14.0

* Spark version :3.2.1

* Hive version :3.1.3

* Hadoop version :3.2.2

* Storage (HDFS/S3/GCS..) :s3

* Running on Docker? (yes/no) :no

**Additional context**

Add any other context about the problem here.

**Stacktrace**

```
Attaching to process ID 8854, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.382-b05
Deadlock Detection:

No deadlocks found.

Thread 23860: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 23859: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 23858: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 23857: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 23856: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 23855: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 23854: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 23853: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 23852: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
- java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=160, line=460 (Compiled frame)
- java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, boolean, long) @bci=102, line=362 (Compiled frame)
- java.util.concurrent.SynchronousQueue.poll(long, java.util.concurrent.TimeUnit) @bci=11, line=941 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1073 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1134 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=624 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 23850: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 23849: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 23848: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 23847: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 23846: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 23845: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 23844: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 23843: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 23842: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 23841: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 23840: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 23839: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 23838: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 23837: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 23836: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 23835: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 23834: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.apache.hadoop.ipc.Client$Connection.waitForWork() @bci=59, line=1034 (Compiled frame)
- org.apache.hadoop.ipc.Client$Connection.run() @bci=58, line=1078 (Interpreted frame)

Thread 20557: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 20556: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 20555: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 20554: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 20553: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 20550: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 18356: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 18354: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 18353: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 18351: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 18350: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 18349: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 18348: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 18346: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 18345: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 12596: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 12595: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 12585: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 12580: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 12578: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 10594: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 10593: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 10423: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 10036: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 10035: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 10034: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 10033: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 10032: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 10029: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1074 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1134 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=624 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 9868: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 9866: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 9862: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 9833: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=78, line=2078 (Compiled frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=124, line=1093 (Compiled frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=809 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1074 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1134 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=624 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 9832: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.TimerThread.mainLoop() @bci=201, line=552 (Interpreted frame)
- java.util.TimerThread.run() @bci=1, line=505 (Interpreted frame)

Thread 9736: (state = BLOCKED)

Thread 9346: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 9345: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 9344: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 9343: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 9342: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 9341: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 9340: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 9299: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.IdleConnectionReaper.run() @bci=90, line=188 (Interpreted frame)

Thread 9271: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 9269: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 9056: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(int) @bci=493, line=411 (Interpreted frame)
- org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(org.apache.hadoop.hdfs.client.impl.LeaseRenewer, int) @bci=2, line=76 (Interpreted frame)
- org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run() @bci=69, line=307 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 9051: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
- java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=160, line=460 (Compiled frame)
- java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, boolean, long) @bci=102, line=362 (Compiled frame)
- java.util.concurrent.SynchronousQueue.poll(long, java.util.concurrent.TimeUnit) @bci=11, line=941 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1073 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1134 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=624 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 9050: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.apache.hadoop.ipc.Client$Connection.waitForWork() @bci=59, line=1034 (Compiled frame)
- org.apache.hadoop.ipc.Client$Connection.run() @bci=58, line=1078 (Interpreted frame)

Thread 8955: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 8953: (state = IN_NATIVE_TRANS)
- org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(int, org.apache.hadoop.net.unix.DomainSocketWatcher$FdSet) @bci=0 (Interpreted frame)
- org.apache.hadoop.net.unix.DomainSocketWatcher.access$900(int, org.apache.hadoop.net.unix.DomainSocketWatcher$FdSet) @bci=2, line=52 (Interpreted frame)
- org.apache.hadoop.net.unix.DomainSocketWatcher$2.run() @bci=763, line=503 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 8951: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- java.lang.ref.ReferenceQueue.remove(long) @bci=59, line=144 (Compiled frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=165 (Compiled frame)
- org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run() @bci=9, line=3830 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 8929: (state = BLOCKED)

Thread 8928: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- java.lang.ref.ReferenceQueue.remove(long) @bci=59, line=144 (Compiled frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=165 (Compiled frame)
- java.lang.ref.Finalizer$FinalizerThread.run() @bci=36, line=188 (Compiled frame)

Thread 8927: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- java.lang.Object.wait() @bci=2, line=502 (Compiled frame)
- java.lang.ref.Reference.tryHandlePending(boolean) @bci=54, line=191 (Compiled frame)
- java.lang.ref.Reference$ReferenceHandler.run() @bci=1, line=153 (Compiled frame)

Thread 8902: (state = BLOCKED)
- java.lang.Shutdown.halt0(int) @bci=0 (Interpreted frame)
- java.lang.Shutdown.halt(int) @bci=7, line=156 (Interpreted frame)
- java.lang.Shutdown.exit(int) @bci=42, line=179 (Interpreted frame)
- java.lang.Runtime.exit(int) @bci=14, line=103 (Interpreted frame)
- java.lang.System.exit(int) @bci=4, line=987 (Interpreted frame)
- org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(java.lang.String[]) @bci=722, line=213 (Interpreted frame)
- org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(java.lang.String[]) @bci=4 (Interpreted frame)
- sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]) @bci=0 (Interpreted frame)
- sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=100, line=62 (Interpreted frame)
- sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=43 (Interpreted frame)
- java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=56, line=498 (Interpreted frame)
- org.apache.spark.deploy.JavaMainApplication.start(java.lang.String[], org.apache.spark.SparkConf) @bci=97, line=52 (Interpreted frame)
- org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(org.apache.spark.deploy.SparkSubmitArguments, boolean) @bci=449, line=1000 (Interpreted frame)
- org.apache.spark.deploy.SparkSubmit.doRunMain$1(org.apache.spark.deploy.SparkSubmitArguments, boolean) @bci=111, line=180 (Interpreted frame)
- org.apache.spark.deploy.SparkSubmit.submit(org.apache.spark.deploy.SparkSubmitArguments, boolean) @bci=65, line=203 (Interpreted frame)
- org.apache.spark.deploy.SparkSubmit.doSubmit(java.lang.String[]) @bci=78, line=90 (Interpreted frame)
- org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(java.lang.String[]) @bci=2, line=1089 (Interpreted frame)
- org.apache.spark.deploy.SparkSubmit$.main(java.lang.String[]) @bci=10, line=1098 (Interpreted frame)
- org.apache.spark.deploy.SparkSubmit.main(java.lang.String[]) @bci=4 (Interpreted frame)

```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the provided spark-sql reproduction command and the Java/Netty thread dump, using the listed Hudi 0.14.0 and Spark 3.2.1 environment. Investigate why repeated Hudi SQL tasks sometimes fail to terminate; done means the task exits normally and reliably without the reported hang.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.