apache / apache/hudi

[SUPPORT] hudi 0.14.1-RC1 task be stuck sometime.

Open
#10,359 4 comments 0 reactions 0 assignees View on GitHub
area:performance area:upgrade
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

After upgrading the community to version 0.14.1, there is still a probability that the task will be stuck. The yarn task has been completed and the task cannot be viewed through yarn app -list. It requires manual kill and re-run to recover. I personally suspect that there is something wrong with the following.

```
Thread 11806: (state = IN_NATIVE_TRANS)
- org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(int, org.apache.hadoop.net.unix.DomainSocketWatcher$FdSet) @bci=0 (Interpreted frame)
- org.apache.hadoop.net.unix.DomainSocketWatcher.access$900(int, org.apache.hadoop.net.unix.DomainSocketWatcher$FdSet) @bci=2, line=52 (Interpreted frame)
- org.apache.hadoop.net.unix.DomainSocketWatcher$2.run() @bci=763, line=503 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)
```

**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment Description**

* Hudi version :0.14.1-RC

* Spark version :3.2.2

* Hive version :3.1.3

* Hadoop version :3.2.2

* Storage (HDFS/S3/GCS..) :s3

* Running on Docker? (yes/no) :no

**Additional context**
jstack -l 10779 stuck, nothing returned.

```
Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding
```

jstack -F 10779 > /tmp/1.log
```
Attaching to process ID 10779, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.382-b05
Deadlock Detection:

No deadlocks found.

Thread 5893: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 5892: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 5891: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 5890: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 5889: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 5888: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 5887: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 5886: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 5885: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
- java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=160, line=460 (Compiled frame)
- java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, boolean, long) @bci=102, line=362 (Compiled frame)
- java.util.concurrent.SynchronousQueue.poll(long, java.util.concurrent.TimeUnit) @bci=11, line=941 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1073 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1134 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=624 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 5883: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 5882: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 5881: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 5880: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 5879: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 5878: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 5877: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 5876: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 5875: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 5874: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 5873: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 5872: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 5871: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 5870: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 5869: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 5868: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Compiled frame)

Thread 5867: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.apache.hadoop.ipc.Client$Connection.waitForWork() @bci=59, line=1034 (Compiled frame)
- org.apache.hadoop.ipc.Client$Connection.run() @bci=58, line=1078 (Interpreted frame)

Thread 31832: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 31830: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 31821: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 28535: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 28533: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 28531: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 28525: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 28524: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 28520: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 18129: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 18122: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 18119: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 18117: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 18108: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=350, line=1824 (Interpreted frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44, line=1693 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=175 (Interpreted frame)

Thread 14000: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 12858: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 12324: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 11177: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 11153: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 32699: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 29477: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 29058: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 27572: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1074 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1134 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=624 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 25667: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=78, line=2078 (Compiled frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=124, line=1093 (Compiled frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=809 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1074 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1134 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=624 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 25666: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.TimerThread.mainLoop() @bci=201, line=552 (Interpreted frame)
- java.util.TimerThread.run() @bci=1, line=505 (Interpreted frame)

Thread 25168: (state = BLOCKED)

Thread 21059: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 21058: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 21057: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 21055: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 21054: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 21018: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 21017: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 20989: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.IdleConnectionReaper.run() @bci=90, line=188 (Interpreted frame)

Thread 20331: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 20051: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 16045: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(int) @bci=493, line=411 (Interpreted frame)
- org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(org.apache.hadoop.hdfs.client.impl.LeaseRenewer, int) @bci=2, line=76 (Interpreted frame)
- org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run() @bci=69, line=307 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 15492: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
- java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=160, line=460 (Compiled frame)
- java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, boolean, long) @bci=102, line=362 (Compiled frame)
- java.util.concurrent.SynchronousQueue.poll(long, java.util.concurrent.TimeUnit) @bci=11, line=941 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1073 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1134 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=624 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 15491: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- org.apache.hadoop.ipc.Client$Connection.waitForWork() @bci=59, line=1034 (Compiled frame)
- org.apache.hadoop.ipc.Client$Connection.run() @bci=58, line=1078 (Interpreted frame)

Thread 11953: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown() @bci=153, line=787 (Interpreted frame)
- io.netty.channel.nio.NioEventLoop.run() @bci=406, line=530 (Interpreted frame)
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run() @bci=44, line=986 (Interpreted frame)
- io.netty.util.internal.ThreadExecutorMap$2.run() @bci=11, line=74 (Interpreted frame)
- io.netty.util.concurrent.FastThreadLocalRunnable.run() @bci=4, line=30 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 11806: (state = IN_NATIVE_TRANS)
- org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(int, org.apache.hadoop.net.unix.DomainSocketWatcher$FdSet) @bci=0 (Interpreted frame)
- org.apache.hadoop.net.unix.DomainSocketWatcher.access$900(int, org.apache.hadoop.net.unix.DomainSocketWatcher$FdSet) @bci=2, line=52 (Interpreted frame)
- org.apache.hadoop.net.unix.DomainSocketWatcher$2.run() @bci=763, line=503 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 11494: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- java.lang.ref.ReferenceQueue.remove(long) @bci=59, line=144 (Compiled frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=165 (Compiled frame)
- org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run() @bci=9, line=3830 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=750 (Interpreted frame)

Thread 10899: (state = BLOCKED)

Thread 10898: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- java.lang.ref.ReferenceQueue.remove(long) @bci=59, line=144 (Compiled frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=165 (Compiled frame)
- java.lang.ref.Finalizer$FinalizerThread.run() @bci=36, line=188 (Compiled frame)

Thread 10897: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- java.lang.Object.wait() @bci=2, line=502 (Compiled frame)
- java.lang.ref.Reference.tryHandlePending(boolean) @bci=54, line=191 (Compiled frame)
- java.lang.ref.Reference$ReferenceHandler.run() @bci=1, line=153 (Compiled frame)

Thread 10872: (state = BLOCKED)
- java.lang.Shutdown.halt0(int) @bci=0 (Interpreted frame)
- java.lang.Shutdown.halt(int) @bci=7, line=156 (Interpreted frame)
- java.lang.Shutdown.exit(int) @bci=42, line=179 (Interpreted frame)
- java.lang.Runtime.exit(int) @bci=14, line=103 (Interpreted frame)
- java.lang.System.exit(int) @bci=4, line=987 (Interpreted frame)
- org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(java.lang.String[]) @bci=722, line=213 (Interpreted frame)
- org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(java.lang.String[]) @bci=4 (Interpreted frame)
- sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]) @bci=0 (Interpreted frame)
- sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=100, line=62 (Interpreted frame)
- sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=43 (Interpreted frame)
- java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=56, line=498 (Interpreted frame)
- org.apache.spark.deploy.JavaMainApplication.start(java.lang.String[], org.apache.spark.SparkConf) @bci=97, line=52 (Interpreted frame)
- org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(org.apache.spark.deploy.SparkSubmitArguments, boolean) @bci=449, line=1000 (Interpreted frame)
- org.apache.spark.deploy.SparkSubmit.doRunMain$1(org.apache.spark.deploy.SparkSubmitArguments, boolean) @bci=111, line=180 (Interpreted frame)
- org.apache.spark.deploy.SparkSubmit.submit(org.apache.spark.deploy.SparkSubmitArguments, boolean) @bci=65, line=203 (Interpreted frame)
- org.apache.spark.deploy.SparkSubmit.doSubmit(java.lang.String[]) @bci=78, line=90 (Interpreted frame)
- org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(java.lang.String[]) @bci=2, line=1089 (Interpreted frame)
- org.apache.spark.deploy.SparkSubmit$.main(java.lang.String[]) @bci=10, line=1098 (Interpreted frame)
- org.apache.spark.deploy.SparkSubmit.main(java.lang.String[]) @bci=4 (Interpreted frame)

```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the reported DomainSocketWatcher and Netty stack traces and reproducing the stuck task with Hudi 0.14.1-RC, Spark 3.2.2, Hadoop 3.2.2, Hive 3.1.3, and S3. Done means identifying why completed YARN tasks remain unavailable and documenting or validating a fix that prevents manual kill-and-rerun recovery.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.