High CPU utilization in Spring Cloud Gateway application

Hi,

Sample Spring Cloud gateway app used for PST is available at - https://github.com/jacob2221/basic-gateway

• PST was done using Apache Jmeter running on same machine as the spring cloud gateway app.
• There is only one route used in the test case.
• Downstream app is running on a diff box
• PST run for 10 minutes with 500 parallel threads
• Gateway app is deployed on RHEL box with below specs

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 4

JRE version used - Java HotSpot(TM) 64-Bit Server VM (build 25.261-b12, mixed mode)

RESULTS

• It is observed that the cpu utilization for first few minutes of PST shoots up to around ~ 120%
• Thereafter it is in the range of 50-70% for most of the run
• During the latter half of the run it comes to around 40%

Below command was executed while PST was running

top -n 1 -H -p

Using the hex value of the thread PIDs from above command on search the output of "jstack ", it shows 4 threads each having name like "reactor-http-epoll-*" and stack as below -

"reactor-http-epoll-2" #21 daemon prio=5 os_prio=0 tid=0x00007fcd20005000 nid=0x1b086 runnable [0x00007fcd40a6e000]
java.lang.Thread.State: RUNNABLE
at io.netty.channel.epoll.Native.epollWait(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:148)
at io.netty.channel.epoll.Native.epollWait(Native.java:141)
at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

JVM arguments used for the test

MEM_ARGS="-Xms2048m -Xmx2048m -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=60 -XX:+UseCMSInitiatingOccupanc
yOnly -XX:+DisableExplicitGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=102
40K -Xloggc:$FILENAME -verbose:gc -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$LOGDIR -XX:+PrintStringTableStatistics -Djava.security.egd=file:///dev/./urandom -Dr
eactor.netty.http.server.accessLogEnabled=true -Dio.netty.leakDetection.level=advanced"

Could you help advise why these threads are consuming high CPU? Are there any further settings we can do to optimize this?

Begin with the linked basic-gateway sample and reproduce the 10-minute, 500-thread Apache JMeter run on the stated RHEL and Java setup. Compare CPU usage and the reactor-http-epoll-* stacks with the reported jstack output and JVM arguments; done means documenting a confirmed cause and a validated optimization, or explaining the expected utilization.