Dashboard large memory footprint (AtomicReference), cause full GC and surge of CPU usage

Open

#1,520 opened on Jun 4, 2020 2 comments 0 reactions 0 assignees View on GitHub

area/dashboard area/performance

Dominant language: Java
Stars: 23.1k
Forks: 8.1k
PR merge metrics: No merged PRs in 30d

Description

### 背景：
sentinel 版本1.7.0

### 描述：
当前应用上线3个月以上，于昨天下午19：00出现老年代不释放导致频繁FULL GC CPU 飙升

![image](https://user-images.githubusercontent.com/10337065/83734796-12d30500-a682-11ea-8d95-17b639caaa35.png)

### 导出的dump文件内发现 AtomicReference对象占用达到90%
![image](https://user-images.githubusercontent.com/10337065/83735077-75c49c00-a682-11ea-8b2f-73d974bddcf1.png)

### 线程信息如下
```java

"java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#15
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#24
at java.lang.Thread.run(Thread.java:745)

"sentinel-command-center-executor-thread-1" prio=5 tid=14 RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
Local Variable: java.net.SocksSocketImpl#1
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
Local Variable: java.net.SocksSocketImpl#2
at java.net.ServerSocket.accept(ServerSocket.java:513)
Local Variable: java.net.Socket#1
Local Variable: java.net.ServerSocket#1
at com.alibaba.csp.sentinel.transport.command.SimpleHttpCommandCenter$ServerThread.run(SimpleHttpCommandCenter.java:188)
Local Variable: com.alibaba.csp.sentinel.transport.command.SimpleHttpCommandCenter$ServerThread#1
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
Local Variable: java.util.concurrent.Executors$RunnableAdapter#30
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
Local Variable: java.util.concurrent.ThreadPoolExecutor#1
Local Variable: java.util.concurrent.FutureTask#29
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#1
at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-26" daemon prio=5 tid=313 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#23
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#51
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#21
at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 12" prio=5 tid=34 RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
Local Variable: sun.nio.ch.EPollArrayWrapper#9
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
Local Variable: sun.nio.ch.EPollSelectorImpl#9
Local Variable: sun.nio.ch.Util$2#9
Local Variable: java.util.Collections$UnmodifiableSet#216
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
Local Variable: org.apache.http.impl.nio.reactor.BaseIOReactor#5
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
Local Variable: org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker#5
at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-10" daemon prio=5 tid=50 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#48
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#4
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#2
at java.lang.Thread.run(Thread.java:745)

"sentinel-dashboard-metrics-fetchService-thread-6" prio=5 tid=71 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#64
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#25
at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 3" prio=5 tid=24 RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
Local Variable: sun.nio.ch.EPollArrayWrapper#18
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
Local Variable: sun.nio.ch.EPollSelectorImpl#18
Local Variable: sun.nio.ch.Util$2#18
Local Variable: java.util.Collections$UnmodifiableSet#236
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
Local Variable: org.apache.http.impl.nio.reactor.BaseIOReactor#14
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
Local Variable: org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker#14
at java.lang.Thread.run(Thread.java:745)

"logback-2" daemon prio=5 tid=139 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#71
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#34
at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 11" prio=5 tid=33 RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
Local Variable: sun.nio.ch.EPollArrayWrapper#10
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
Local Variable: sun.nio.ch.EPollSelectorImpl#10
Local Variable: java.util.Collections$UnmodifiableSet#217
Local Variable: sun.nio.ch.Util$2#10
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
Local Variable: org.apache.http.impl.nio.reactor.BaseIOReactor#6
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
Local Variable: org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker#6
at java.lang.Thread.run(Thread.java:745)

"ContainerBackgroundProcessor[StandardEngine[Tomcat]]" daemon prio=5 tid=19 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1356)
Local Variable: org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor#1
Local Variable: java.lang.String#88830
at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-ClientPoller-0" daemon prio=5 tid=51 RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
Local Variable: sun.nio.ch.EPollArrayWrapper#2
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
Local Variable: sun.nio.ch.EPollSelectorImpl#2
Local Variable: sun.nio.ch.Util$2#2
Local Variable: java.util.Collections$UnmodifiableSet#209
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:825)
Local Variable: org.apache.tomcat.util.net.NioEndpoint$Poller#2
at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-4" daemon prio=5 tid=44 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#19
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#5
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#3
at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-21" daemon prio=5 tid=308 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#44
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#48
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#18
at java.lang.Thread.run(Thread.java:745)

"sentinel-system-status-record-task-thread-1" daemon prio=5 tid=59 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#8
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#93
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue#4
Local Variable: java.util.concurrent.locks.ReentrantLock#1570
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor#4
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#16
at java.lang.Thread.run(Thread.java:745)

"sentinel-dashboard-metrics-fetchService-thread-5" prio=5 tid=70 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#63
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#26
at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 13" prio=5 tid=35 RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
Local Variable: sun.nio.ch.EPollArrayWrapper#7
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
Local Variable: sun.nio.ch.EPollSelectorImpl#7
Local Variable: java.util.Collections$UnmodifiableSet#214
Local Variable: sun.nio.ch.Util$2#7
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
Local Variable: org.apache.http.impl.nio.reactor.BaseIOReactor#4
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
Local Variable: org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker#4
at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-15" daemon prio=5 tid=302 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#49
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#42
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#12
at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-12" daemon prio=5 tid=62 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#34
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#14
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#10
at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-5" daemon prio=5 tid=45 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#28
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#6
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#4
at java.lang.Thread.run(Thread.java:745)

"sentinel-dashboard-metrics-fetchService-thread-4" prio=5 tid=69 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#61
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#27
at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-24" daemon prio=5 tid=311 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#24
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#53
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#23
at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-23" daemon prio=5 tid=310 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#50
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#50
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#20
at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 6" prio=5 tid=27 RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
Local Variable: sun.nio.ch.EPollArrayWrapper#15
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
Local Variable: sun.nio.ch.EPollSelectorImpl#15
Local Variable: sun.nio.ch.Util$2#15
Local Variable: java.util.Collections$UnmodifiableSet#225
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
Local Variable: org.apache.http.impl.nio.reactor.BaseIOReactor#11
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
Local Variable: org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker#11
at java.lang.Thread.run(Thread.java:745)

"sentinel-heartbeat-send-task-thread-1" daemon prio=5 tid=12 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#59
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#5
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
Local Variable: java.util.concurrent.locks.ReentrantLock#1318
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue#1
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor#1
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#2
at java.lang.Thread.run(Thread.java:745)

"DestroyJavaVM" prio=5 tid=56 RUNNABLE

"sentinel-dashboard-metrics-fetchWorker-thread-3" prio=5 tid=74 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
Local Variable: java.util.concurrent.CountDownLatch$Sync#1
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#56
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
at com.alibaba.csp.sentinel.dashboard.metric.MetricFetcher.fetchOnce(MetricFetcher.java:250)
Local Variable: java.util.concurrent.ConcurrentHashMap#2
Local Variable: com.alibaba.csp.sentinel.dashboard.metric.MetricFetcher#1
at com.alibaba.csp.sentinel.dashboard.metric.MetricFetcher.lambda$doFetchAppMetric$3(MetricFetcher.java:282)
Local Variable: java.lang.String#105171
at com.alibaba.csp.sentinel.dashboard.metric.MetricFetcher$$Lambda$378.run()
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
Local Variable: java.util.concurrent.Executors$RunnableAdapter#1
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
Local Variable: java.util.concurrent.FutureTask#1
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#22
at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-AsyncTimeout" daemon prio=5 tid=54 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at org.apache.coyote.AbstractProtocol$AsyncTimeout.run(AbstractProtocol.java:1149)
Local Variable: org.apache.coyote.AbstractProtocol$AsyncTimeout#1
at java.lang.Thread.run(Thread.java:745)

"container-0" prio=5 tid=20 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.core.StandardServer.await(StandardServer.java:427)
Local Variable: org.apache.catalina.core.StandardServer#1
at org.springframework.boot.web.embedded.tomcat.TomcatWebServer$1.run(TomcatWebServer.java:182)

"Reference Handler" daemon prio=10 tid=2 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)

"http-nio-18088-ClientPoller-1" daemon prio=5 tid=52 RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
Local Variable: sun.nio.ch.EPollArrayWrapper#1
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
Local Variable: sun.nio.ch.EPollSelectorImpl#1
Local Variable: sun.nio.ch.Util$2#1
Local Variable: java.util.Collections$UnmodifiableSet#208
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:825)
Local Variable: org.apache.tomcat.util.net.NioEndpoint$Poller#1
at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 4" prio=5 tid=25 RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
Local Variable: sun.nio.ch.EPollArrayWrapper#17
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
Local Variable: sun.nio.ch.EPollSelectorImpl#17
Local Variable: sun.nio.ch.Util$2#17
Local Variable: java.util.Collections$UnmodifiableSet#235
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
Local Variable: org.apache.http.impl.nio.reactor.BaseIOReactor#13
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
Local Variable: org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker#13
at java.lang.Thread.run(Thread.java:745)

"logback-6" daemon prio=5 tid=183 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#70
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#38
at java.lang.Thread.run(Thread.java:745)

"Finalizer" daemon prio=8 tid=3 WAITING
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
Local Variable: java.lang.ref.ReferenceQueue#18
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
Local Variable: java.lang.System$2#1

"sentinel-dashboard-metrics-fetchService-thread-8" prio=5 tid=73 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#62
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#23
at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-1" daemon prio=5 tid=41 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#80
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#25
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
Local Variable: java.util.concurrent.locks.ReentrantLock#1527
Local Variable: java.util.concurrent.atomic.AtomicInteger#42
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
Local Variable: org.apache.tomcat.util.threads.TaskQueue#1
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
Local Variable: org.apache.tomcat.util.threads.ThreadPoolExecutor#1
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#3
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#1
at java.lang.Thread.run(Thread.java:745)

"pool-1-thread-1" prio=5 tid=21 RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
Local Variable: sun.nio.ch.EPollArrayWrapper#21
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
Local Variable: sun.nio.ch.EPollSelectorImpl#21
Local Variable: sun.nio.ch.Util$2#21
Local Variable: java.util.Collections$UnmodifiableSet#283
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:340)
Local Variable: org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor#2
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:194)
Local Variable: org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager#2
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
Local Variable: org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1#2
Local Variable: org.apache.http.impl.nio.client.InternalIODispatch#2
at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 14" prio=5 tid=36 RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
Local Variable: sun.nio.ch.EPollArrayWrapper#6
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
Local Variable: sun.nio.ch.EPollSelectorImpl#6
Local Variable: sun.nio.ch.Util$2#6
Local Variable: java.util.Collections$UnmodifiableSet#213
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
Local Variable: org.apache.http.impl.nio.reactor.BaseIOReactor#3
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
Local Variable: org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker#3
at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-14" daemon prio=5 tid=301 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#20
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#41
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#11
at java.lang.Thread.run(Thread.java:745)

"Signal Dispatcher" daemon prio=9 tid=4 RUNNABLE

"http-nio-18088-exec-22" daemon prio=5 tid=309 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#37
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#49
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#19
at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 5" prio=5 tid=26 RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
Local Variable: sun.nio.ch.EPollArrayWrapper#16
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
Local Variable: sun.nio.ch.EPollSelectorImpl#16
Local Variable: sun.nio.ch.Util$2#16
Local Variable: java.util.Collections$UnmodifiableSet#234
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
Local Variable: org.apache.http.impl.nio.reactor.BaseIOReactor#12
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
Local Variable: org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker#12
at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 10" prio=5 tid=32 RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
Local Variable: sun.nio.ch.EPollArrayWrapper#11
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
Local Variable: sun.nio.ch.Util$2#11
Local Variable: sun.nio.ch.EPollSelectorImpl#11
Local Variable: java.util.Collections$UnmodifiableSet#218
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
Local Variable: org.apache.http.impl.nio.reactor.BaseIOReactor#7
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
Local Variable: org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker#7
at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 1" prio=5 tid=22 RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
Local Variable: sun.nio.ch.EPollArrayWrapper#20
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
Local Variable: sun.nio.ch.EPollSelectorImpl#20
Local Variable: sun.nio.ch.Util$2#20
Local Variable: java.util.Collections$UnmodifiableSet#238
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
Local Variable: org.apache.http.impl.nio.reactor.BaseIOReactor#16
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
Local Variable: org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker#16
at java.lang.Thread.run(Thread.java:745)

"sentinel-dashboard-metrics-fetchWorker-thread-6" prio=5 tid=77 RUNNABLE
at org.apache.http.message.AbstractHttpMessage.addHeader(AbstractHttpMessage.java:104)
Local Variable: java.lang.String#91058
Local Variable: java.lang.String#92315
at org.apache.http.protocol.RequestUserAgent.process(RequestUserAgent.java:76)
at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:133)
Local Variable: org.apache.http.client.methods.HttpRequestWrapper#6
Local Variable: org.apache.http.client.protocol.HttpClientContext#6
Local Variable: org.apache.http.HttpRequestInterceptor[]#1
at org.apache.http.impl.nio.client.MainClientExec.prepareRequest(MainClientExec.java:520)
at org.apache.http.impl.nio.client.MainClientExec.prepare(MainClientExec.java:146)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:128)
at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141)
Local Variable: org.apache.http.concurrent.BasicFuture#6
Local Variable: org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl#6
at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:75)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:108)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:92)
at com.alibaba.csp.sentinel.dashboard.metric.MetricFetcher.fetchOnce(MetricFetcher.java:214)
Local Variable: java.util.HashMap$KeyIterator#15
Local Variable: java.util.concurrent.atomic.AtomicLong#11679
Local Variable: java.util.concurrent.atomic.AtomicLong#11678
Local Variable: java.util.concurrent.CountDownLatch#4
Local Variable: java.util.concurrent.atomic.AtomicLong#11680
Local Variable: java.util.concurrent.ConcurrentHashMap#13
at com.alibaba.csp.sentinel.dashboard.metric.MetricFetcher.lambda$doFetchAppMetric$3(MetricFetcher.java:282)
Local Variable: java.lang.String#304549
at com.alibaba.csp.sentinel.dashboard.metric.MetricFetcher$$Lambda$378.run()
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
Local Variable: java.util.concurrent.Executors$RunnableAdapter#24
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
Local Variable: java.util.concurrent.FutureTask#24
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#19
at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-9" daemon prio=5 tid=49 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#46
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#10
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#8
at java.lang.Thread.run(Thread.java:745)

"sentinel-dashboard-metrics-fetchWorker-thread-1" prio=5 tid=65 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#86
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#42
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
Local Variable: java.util.concurrent.locks.ReentrantLock#1534
Local Variable: java.util.concurrent.ArrayBlockingQueue#3
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
Local Variable: java.util.concurrent.ThreadPoolExecutor#9
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#31
at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 7" prio=5 tid=28 RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
Local Variable: sun.nio.ch.EPollArrayWrapper#14
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
Local Variable: sun.nio.ch.EPollSelectorImpl#14
Local Variable: java.util.Collections$UnmodifiableSet#224
Local Variable: sun.nio.ch.Util$2#14
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
Local Variable: org.apache.http.impl.nio.reactor.BaseIOReactor#10
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
Local Variable: org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker#10
at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-17" daemon prio=5 tid=304 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#26
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#43
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#13
at java.lang.Thread.run(Thread.java:745)

"logback-1" daemon prio=5 tid=97 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#95
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#78
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue#5
Local Variable: java.util.concurrent.locks.ReentrantLock#2011
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor#5
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#33
at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 16" prio=5 tid=38 RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
Local Variable: sun.nio.ch.EPollArrayWrapper#4
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
Local Variable: sun.nio.ch.EPollSelectorImpl#4
Local Variable: java.util.Collections$UnmodifiableSet#211
Local Variable: sun.nio.ch.Util$2#4
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
Local Variable: org.apache.http.impl.nio.reactor.BaseIOReactor#1
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
Local Variable: org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker#1
at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 9" prio=5 tid=31 RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
Local Variable: sun.nio.ch.EPollArrayWrapper#12
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
Local Variable: sun.nio.ch.EPollSelectorImpl#12
Local Variable: java.util.Collections$UnmodifiableSet#219
Local Variable: sun.nio.ch.Util$2#12
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
Local Variable: org.apache.http.impl.nio.reactor.BaseIOReactor#8
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
Local Variable: org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker#8
at java.lang.Thread.run(Thread.java:745)

"logback-7" daemon prio=5 tid=212 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#72
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#39
at java.lang.Thread.run(Thread.java:745)

"sentinel-dashboard-metrics-fetchWorker-thread-5" prio=5 tid=76 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#33
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#20
at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-16" daemon prio=5 tid=303 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#18
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#44
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#14
at java.lang.Thread.run(Thread.java:745)

"logback-4" daemon prio=5 tid=156 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#73
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#36
at java.lang.Thread.run(Thread.java:745)

"sentinel-metrics-record-task-thread-1" daemon prio=5 tid=60 RUNNABLE
at com.alibaba.csp.sentinel.slots.statistic.metric.ArrayMetric.fromBucket(ArrayMetric.java:191)
Local Variable: com.alibaba.csp.sentinel.slots.statistic.base.WindowWrap#6539
at com.alibaba.csp.sentinel.slots.statistic.metric.ArrayMetric.details(ArrayMetric.java:165)
Local Variable: java.util.ArrayList#78
Local Variable: com.alibaba.csp.sentinel.slots.statistic.metric.ArrayMetric#320
at com.alibaba.csp.sentinel.node.StatisticNode.metrics(StatisticNode.java:121)
Local Variable: java.util.concurrent.ConcurrentHashMap#30
at com.alibaba.csp.sentinel.node.metric.MetricTimerListener.run(MetricTimerListener.java:44)
Local Variable: java.util.TreeMap#1
Local Variable: com.alibaba.csp.sentinel.node.metric.MetricTimerListener#1
Local Variable: com.alibaba.csp.sentinel.node.ClusterNode#80
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
Local Variable: java.util.concurrent.Executors$RunnableAdapter#33
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask#5
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor#3
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#15
at java.lang.Thread.run(Thread.java:745)

"sentinel-dashboard-metrics-fetchService-thread-1" prio=5 tid=64 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#59
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#83
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
Local Variable: java.util.concurrent.ArrayBlockingQueue#2
Local Variable: java.util.concurrent.locks.ReentrantLock#1532
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
Local Variable: java.util.concurrent.ThreadPoolExecutor#8
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#32
at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-Acceptor-0" daemon prio=5 tid=53 RUNNABLE
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
Local Variable: sun.nio.ch.ServerSocketChannelImpl#1
Local Variable: java.lang.Object#14269
Local Variable: java.net.InetSocketAddress[]#5
Local Variable: java.io.FileDescriptor#34
at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:482)
Local Variable: org.apache.tomcat.util.net.NioEndpoint$Acceptor#1
at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-20" daemon prio=5 tid=307 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#22
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#47
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#17
at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-13" daemon prio=5 tid=63 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#35
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#13
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#9
at java.lang.Thread.run(Thread.java:745)

"sentinel-heartbeat-send-task-thread-2" daemon prio=5 tid=57 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#6
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#11
at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 15" prio=5 tid=37 RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
Local Variable: sun.nio.ch.EPollArrayWrapper#5
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
Local Variable: sun.nio.ch.EPollSelectorImpl#5
Local Variable: java.util.Collections$UnmodifiableSet#212
Local Variable: sun.nio.ch.Util$2#5
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
Local Variable: org.apache.http.impl.nio.reactor.BaseIOReactor#2
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
Local Variable: org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker#2
at java.lang.Thread.run(Thread.java:745)

"sentinel-dashboard-metrics-fetchWorker-thread-4" prio=5 tid=75 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#54
Local Variable: java.util.concurrent.CountDownLatch$Sync#2
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
at com.alibaba.csp.sentinel.dashboard.metric.MetricFetcher.fetchOnce(MetricFetcher.java:250)
Local Variable: java.util.concurrent.ConcurrentHashMap#5
at com.alibaba.csp.sentinel.dashboard.metric.MetricFetcher.lambda$doFetchAppMetric$3(MetricFetcher.java:282)
Local Variable: java.lang.String#129459
at com.alibaba.csp.sentinel.dashboard.metric.MetricFetcher$$Lambda$378.run()
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
Local Variable: java.util.concurrent.Executors$RunnableAdapter#2
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
Local Variable: java.util.concurrent.FutureTask#2
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#21
at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-25" daemon prio=5 tid=312 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#51
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#52
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#22
at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-8" daemon prio=5 tid=48 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#36
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#7
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#5
at java.lang.Thread.run(Thread.java:745)

"sentinel-dashboard-metrics-fetchWorker-thread-8" prio=5 tid=79 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#43
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#17
at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 2" prio=5 tid=23 RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
Local Variable: sun.nio.ch.EPollArrayWrapper#19
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
Local Variable: sun.nio.ch.EPollSelectorImpl#19
Local Variable: sun.nio.ch.Util$2#19
Local Variable: java.util.Collections$UnmodifiableSet#237
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
Local Variable: org.apache.http.impl.nio.reactor.BaseIOReactor#15
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
Local Variable: org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker#15
at java.lang.Thread.run(Thread.java:745)

"sentinel-dashboard-metrics-fetchService-thread-3" prio=5 tid=68 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#60
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#28
at java.lang.Thread.run(Thread.java:745)

"I/O dispatcher 8" prio=5 tid=29 RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
Local Variable: sun.nio.ch.EPollArrayWrapper#13
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
Local Variable: sun.nio.ch.EPollSelectorImpl#13
Local Variable: sun.nio.ch.Util$2#13
Local Variable: java.util.Collections$UnmodifiableSet#223
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
Local Variable: org.apache.http.impl.nio.reactor.BaseIOReactor#9
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
Local Variable: org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker#9
at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-19" daemon prio=5 tid=306 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#21
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#46
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#16
at java.lang.Thread.run(Thread.java:745)

"logback-8" daemon prio=5 tid=213 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#77
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#40
at java.lang.Thread.run(Thread.java:745)

"NioBlockingSelector.BlockPoller-1" daemon prio=5 tid=40 RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
Local Variable: sun.nio.ch.EPollArrayWrapper#3
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
Local Variable: sun.nio.ch.EPollSelectorImpl#3
Local Variable: sun.nio.ch.Util$2#3
Local Variable: java.util.Collections$UnmodifiableSet#210
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:298)

"http-nio-18088-exec-18" daemon prio=5 tid=305 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#47
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#45
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#15
at java.lang.Thread.run(Thread.java:745)

"sentinel-dashboard-metrics-fetchWorker-thread-2" prio=5 tid=67 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#41
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#29
at java.lang.Thread.run(Thread.java:745)

"pool-2-thread-1" prio=5 tid=30 RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
Local Variable: sun.nio.ch.EPollArrayWrapper#8
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
Local Variable: sun.nio.ch.EPollSelectorImpl#8
Local Variable: java.util.Collections$UnmodifiableSet#215
Local Variable: sun.nio.ch.Util$2#8
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:340)
Local Variable: org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor#1
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:194)
Local Variable: org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager#1
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
Local Variable: org.apache.http.impl.nio.client.InternalIODispatch#1
Local Variable: org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1#1
at java.lang.Thread.run(Thread.java:745)

"logback-3" daemon prio=5 tid=151 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#1
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#35
at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-6" daemon prio=5 tid=46 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#45
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#8
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#6
at java.lang.Thread.run(Thread.java:745)

"sentinel-time-tick-thread" daemon prio=5 tid=58 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at java.lang.Thread.sleep(Thread.java:340)
at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386)
at com.alibaba.csp.sentinel.util.TimeUtil$1.run(TimeUtil.java:37)
at java.lang.Thread.run(Thread.java:745)

"sentinel-dashboard-metrics-fetch-task-thread-1" prio=5 tid=39 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#89
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#58
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
Local Variable: java.util.concurrent.locks.ReentrantLock#1536
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue#2
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
Local Variable: java.util.concurrent.ScheduledThreadPoolExecutor#2
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#12
at java.lang.Thread.run(Thread.java:745)

"sentinel-dashboard-metrics-fetchWorker-thread-7" prio=5 tid=78 RUNNABLE
at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317)
at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:193)
Local Variable: java.lang.Object#14295
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.connect(DefaultConnectingIOReactor.java:231)
Local Variable: org.apache.http.impl.nio.reactor.SessionRequestImpl#1
at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractNIOConnPool.java:445)
Local Variable: org.apache.http.nio.pool.AbstractNIOConnPool$2#23
at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:277)
Local Variable: org.apache.http.nio.pool.LeaseRequest#1
Local Variable: org.apache.http.concurrent.BasicFuture#8
Local Variable: org.apache.http.impl.nio.conn.CPool#1
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.java:268)
Local Variable: org.apache.http.concurrent.BasicFuture#7
at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:377)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:129)
at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141)
Local Variable: org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl#5
Local Variable: org.apache.http.concurrent.BasicFuture#5
at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:75)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:108)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:92)
at com.alibaba.csp.sentinel.dashboard.metric.MetricFetcher.fetchOnce(MetricFetcher.java:214)
Local Variable: java.util.HashMap$KeyIterator#14
Local Variable: java.util.concurrent.atomic.AtomicLong#11677
Local Variable: java.util.concurrent.atomic.AtomicLong#11676
Local Variable: java.util.concurrent.atomic.AtomicLong#11675
Local Variable: java.util.concurrent.CountDownLatch#3
Local Variable: java.util.concurrent.ConcurrentHashMap#12
at com.alibaba.csp.sentinel.dashboard.metric.MetricFetcher.lambda$doFetchAppMetric$3(MetricFetcher.java:282)
Local Variable: java.lang.String#105189
at com.alibaba.csp.sentinel.dashboard.metric.MetricFetcher$$Lambda$378.run()
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
Local Variable: java.util.concurrent.Executors$RunnableAdapter#23
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
Local Variable: java.util.concurrent.FutureTask#23
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#18
at java.lang.Thread.run(Thread.java:745)

"http-nio-18088-exec-2" daemon prio=5 tid=42 TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#39
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#9
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
Local Variable: org.apache.tomcat.util.threads.TaskThread$WrappingRunnable#7
at java.lang.Thread.run(Thread.java:745)

"Attach Listener" daemon prio=9 tid=329 RUNNABLE

"logback-5" daemon prio=5 tid=164 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#69
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#37
at java.lang.Thread.run(Thread.java:745)

"sentinel-dashboard-metrics-fetchService-thread-2" prio=5 tid=66 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
Local Variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#17
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Local Variable: java.util.concurrent.ThreadPoolExecutor$Worker#30
at java.lang.Thread.run(Thread.java:745)`

```

Contributor guide

Open the contributing guide

Research direction

Start with the Sentinel 1.7.0 dashboard deployment described in the report and inspect the heap dump and thread traces, especially SimpleHttpCommandCenter.java:188 and the dashboard metrics-fetch thread. Trace the AtomicReference retainers to determine what keeps old-generation objects alive; done means the cause of the full-GC and CPU surge is reproduced and a verified fix or regression test is identified.

Written by the indexing model from the issue text.

Assessment

Tech stack: java
Domain: backend, observability, performance
Issue type: Bug
Difficulty: 4/5
Estimated time: 3-5 days
Activity status: Stale
Clarity: Needs clarification
Newbie friendliness: 25/100

Dashboard large memory footprint (AtomicReference), cause full GC and surge of CPU usage

Description

Contributor guide

Research direction

Assessment

Get new issues in your inbox