spring-cloud / spring-cloud/spring-cloud-gateway

no increase in client requests,the server's CPU usage increases day by day,

Open
#3,753 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

waiting-for-triage
Dominant language
Java
Stars
4.9k
Forks
3.5k
Avg merge
20h 57m
Merged PRs (30d)
8

Description

version
spring bood 3.2.7
spring cloud 2023.0.1
jdk 21
-server -Xmx8192m -Xms8192m -XX:MaxDirectMemorySize=6g -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=45 -XX:ParallelGCThreads=7 -XX:ConcGCThreads=7 -Dreactor.netty.pool.leasingStrategy=lifo

Describe the bug
QPS from 8K-11K every day
no increase in client requests, the CPU usage of the gateway server gets higher day by day. However, when I restart the gateway service, the CPU usage immediately drops. I only added a filter to retrieve the client's IP address and pass it through to the backend. On 4.8, I restart the gateway server.

Image

I checked the GC(G1) and it appears to be normal.
I check the direct memory, and it appears to be normal.
I print the Netty client pool usage every 5 minutes, and it appears to be normal.
I also check the java stack, and do not see block thread.

2025-04-08 08:50:00.000  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1847835500,active connections:0.0 
2025-04-08 08:50:00.000  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1846933028,active connections:1.0 
2025-04-08 08:50:00.000  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1847834508,active connections:1.0 
2025-04-08 08:50:00.000  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1846902927,active connections:0.0 
2025-04-08 08:50:00.000  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1846936996,active connections:0.0 
2025-04-08 08:50:00.000  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1847833516,active connections:0.0 
2025-04-08 08:50:00.000  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1846961145,active connections:0.0 
2025-04-08 08:50:00.000  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1847832524,active connections:0.0 
2025-04-08 08:50:00.000  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1847833516,idle connections:29.0 
2025-04-08 08:50:00.000  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1847835500,idle connections:26.0 
2025-04-08 08:50:00.000  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1847834508,idle connections:28.0 
2025-04-08 08:50:00.000  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1847832524,idle connections:24.0 
2025-04-08 08:50:00.000  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1846961145,idle connections:23.0 
2025-04-08 08:50:00.000  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1846902927,idle connections:26.0 
2025-04-08 08:50:00.000  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1846936996,idle connections:32.0 
2025-04-08 08:50:00.000  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1846933028,idle connections:25.0 
2025-04-08 08:50:00.000  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1846961145,max connections:32.0 
2025-04-08 08:50:00.000  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1846936996,max connections:32.0 
2025-04-08 08:50:00.000  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1846902927,max connections:32.0 
2025-04-08 08:50:00.000  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1847834508,max connections:32.0 
2025-04-08 08:50:00.000  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1847833516,max connections:32.0 
2025-04-08 08:50:00.000  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1846933028,max connections:32.0 
2025-04-08 08:50:00.000  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1847835500,max connections:32.0 
2025-04-08 08:50:00.000  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1847832524,max connections:32.0 
2025-04-08 08:50:00.000  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1846902927,total connections:26.0 
2025-04-08 08:50:00.000  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1846936996,total connections:32.0 
2025-04-08 08:50:00.000  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1846933028,total connections:26.0 
2025-04-08 08:50:00.000  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1847834508,total connections:29.0 
2025-04-08 08:50:00.000  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1847833516,total connections:29.0 
2025-04-08 08:50:00.000  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1846961145,total connections:23.0 
2025-04-08 08:50:00.000  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1847835500,total connections:26.0 
2025-04-08 08:50:00.000  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1847832524,total connections:24.0 
2025-04-08 08:50:00.000  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1847835500,max.pending connections:2.147483647E9 
2025-04-08 08:50:00.001  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1847832524,max.pending connections:2.147483647E9 
2025-04-08 08:50:00.001  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1846961145,max.pending connections:2.147483647E9 
2025-04-08 08:50:00.001  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1846933028,max.pending connections:2.147483647E9 
2025-04-08 08:50:00.001  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1847834508,max.pending connections:2.147483647E9 
2025-04-08 08:50:00.001  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1847833516,max.pending connections:2.147483647E9 
2025-04-08 08:50:00.001  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1846936996,max.pending connections:2.147483647E9 
2025-04-08 08:50:00.001  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1846902927,max.pending connections:2.147483647E9 
2025-04-08 08:50:00.001  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1847835500,pending connections:0.0 
2025-04-08 08:50:00.001  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1847833516,pending connections:0.0 
2025-04-08 08:50:00.001  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1846933028,pending connections:0.0 
2025-04-08 08:50:00.001  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1847834508,pending connections:0.0 
2025-04-08 08:50:00.001  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1846961145,pending connections:0.0 
2025-04-08 08:50:00.001  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1846902927,pending connections:0.0 
2025-04-08 08:50:00.001  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1847832524,pending connections:0.0 
2025-04-08 08:50:00.001  INFO {[]} [            scheduling-1]       i.r.g.s.s.AppInfoSchedular: netty pool id:1846936996,pending connections:0.0

My code: ClientHeaderFilter

//
// Source code recreated from a .class file by IntelliJ IDEA
// (powered by FernFlower decompiler)
//

package igg.report.gateway.shell.filter;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.cloud.gateway.filter.GatewayFilterChain;
import org.springframework.cloud.gateway.filter.GlobalFilter;
import org.springframework.core.Ordered;
import org.springframework.http.server.reactive.ServerHttpRequest;
import org.springframework.util.StringUtils;
import org.springframework.web.server.ServerWebExchange;
import reactor.core.publisher.Mono;

public class ClientHeaderFilter implements GlobalFilter, Ordered {
    private static final Logger log = LoggerFactory.getLogger(ClientHeaderFilter.class);

    public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) {
        ServerHttpRequest request = exchange.getRequest();
        String clientIp = this.getIpAddr(request);
        ServerHttpRequest modifiedRequest = request.mutate().header("IGG-PROXY-IP", new String[]{clientIp}).build();
        return chain.filter(exchange.mutate().request(modifiedRequest).build());
    }

    private String getIpAddr(ServerHttpRequest request) {
        String ip = request.getHeaders().getFirst("x-forwarded-for");
        if (StringUtils.isEmpty(ip)) {
            ip = request.getHeaders().getFirst("Proxy-Client-IP");
        }

        if (StringUtils.isEmpty(ip)) {
            ip = request.getHeaders().getFirst("WL-Proxy-Client-IP");
        }

        if (StringUtils.isEmpty(ip)) {
            ip = request.getHeaders().getFirst("HTTP_CLIENT_IP");
        }

        if (StringUtils.isEmpty(ip)) {
            ip = request.getHeaders().getFirst("HTTP_X_FORWARDED_FOR");
        }

        if (StringUtils.isEmpty(ip)) {
            ip = request.getRemoteAddress().getAddress().getHostAddress();
        }

        if (ip != null && ip.indexOf(",") != -1) {
            String[] ipWithMultiProxy = ip.split(",");

            for(int i = 0; i < ipWithMultiProxy.length; ++i) {
                String eachIpSegement = ipWithMultiProxy[i];
                if (!"unknown".equalsIgnoreCase(eachIpSegement)) {
                    ip = eachIpSegement;
                    break;
                }
            }
        }

        return ip;
    }

    public int getOrder() {
        return RouteToRequestUrlFilter.ROUTE_TO_URL_FILTER_ORDER + 2;
    }
}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the provided ClientHeaderFilter and its position in the Spring Cloud Gateway filter chain, then reproduce the CPU increase under the stated QPS while comparing Java stack, GC, direct-memory, and Netty pool observations over time. Done means identifying a reproducible cause for the rising CPU usage and verifying the resulting fix without relying on a gateway restart.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spring, spring-boot
Domain
api, backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.