spring-cloud / spring-cloud/spring-cloud-gateway

Expose `disposeInactivePoolsInBackground` configuration for HttpClient ConnectionProvider

Open
#4,165 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

waiting-for-triage
Dominant language
Java
Stars
4.9k
Forks
3.5k
Avg merge
20h 57m
Merged PRs (30d)
8

Description

Problem Description

Spring Cloud Gateway (SCG) exposes HttpClientProperties to configure the underlying Reactor Netty connection pool, but it does not expose the disposeInactivePoolsInBackground(interval, inactivity) parameter from ConnectionProvider.Builder.

This causes a critical memory leak in environments with dynamic downstream instances (e.g., Kubernetes), where Pod IPs change frequently due to rolling deployments or auto-scaling.

Root Cause Analysis

Reactor Netty's PooledConnectionProvider maintains a ConcurrentHashMap<PoolKey, InstrumentedPool> named channelPools. A PoolKey is composed of:

  • remoteAddress (downstream IP:Port)
  • pipelineKey (channelHash, derived from HttpClientConfig)

Every time a downstream Pod is restarted or replaced, a new IP is assigned → a new PoolKey is created → a new SimpleDequePool is added to channelPools.

The critical problem: channelPools is never cleaned up automatically. The existing configuration options only clean up connections within a pool, not the PoolKey entries (pools themselves):

Config property Effect
httpclient.pool.max-idle-time Evicts idle connections inside a pool ✅
httpclient.pool.max-life-time Evicts long-lived connections inside a pool ✅
httpclient.pool.eviction-interval Triggers periodic connection eviction ✅
disposeInactivePoolsInBackground Removes PoolKey entries for inactive pools ❌ Not exposed

The disposeInactivePoolsInBackground method exists in PooledConnectionProvider (reactor-netty v1.1.x):

final void disposeInactivePoolsInBackground() {
    toDispose = channelPools.entrySet()
        .stream()
        .filter(p -> p.getValue().metrics().isInactiveForMoreThan(poolInactivity))
        .collect(Collectors.toList());

    toDispose.forEach(e -> {
        if (channelPools.remove(e.getKey(), e.getValue())) { // Actually removes the PoolKey!
            e.getValue().dispose();
        }
    });
    scheduleInactivePoolsDisposal();
}

This is triggered only when inactivePoolDisposeInterval is non-zero, which is set via ConnectionProvider.Builder.disposeInactivePoolsInBackground(interval, inactivity). SCG never sets this value, so it defaults to Duration.ZERO and the cleanup never runs.

Observed Impact

In our Kubernetes-based Spring Cloud Gateway deployment, we observed:

  • 541,106 instances of reactor.netty.internal.shaded.reactor.pool.SimpleDequePool
  • Occupying ~14.3 GB (94.34% of heap)
  • GC Root path: HttpResources → DefaultPooledConnectionProvider → channelPools (ConcurrentHashMap) → SimpleDequePool
  • Each PoolKey.fqdn corresponds to a distinct downstream Pod IP that no longer exists

Heap dump analysis confirmed channelPools.size() equals the number of accumulated SimpleDequePool instances, directly corresponding to the number of distinct downstream IPs seen over the application's lifetime.

Current Workaround

Users must bypass HttpClientProperties entirely and register a custom ConnectionProvider bean:

@Bean
@Primary
public ConnectionProvider gatewayConnectionProvider() {
    return ConnectionProvider.builder("gateway")
        .maxConnections(500)
        .maxIdleTime(Duration.ofSeconds(4))
        .maxLifeTime(Duration.ofSeconds(60))
        .evictInBackground(Duration.ofSeconds(10))
        // Not available via application.properties!
        .disposeInactivePoolsInBackground(
            Duration.ofMinutes(5),  // scan interval
            Duration.ofMinutes(5)   // inactivity threshold
        )
        .build();
}

This workaround works but requires users to know the internal reactor-netty API and loses the convenience of application.properties configuration.

Feature Request

Please expose disposeInactivePoolsInBackground in HttpClientProperties (or GatewayProperties) to allow configuration via:

spring.cloud.gateway.httpclient.pool.inactive-pool-dispose-interval=5m
spring.cloud.gateway.httpclient.pool.pool-inactivity=5m

And wire it into HttpClientAutoConfiguration when building the ConnectionProvider:

// In HttpClientAutoConfiguration or similar
if (pool.getInactivePoolDisposeInterval() != null) {
    builder.disposeInactivePoolsInBackground(
        pool.getInactivePoolDisposeInterval(),
        pool.getPoolInactivity()
    );
}
Environment
  • Spring Cloud Gateway: 3.x (based on Spring Boot 3 / SCG 4.x series)
  • Reactor Netty: 1.1.5
  • Deployment: Kubernetes with frequent Pod rolling updates
References

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read HttpClientProperties and HttpClientAutoConfiguration to trace how pool settings reach the ConnectionProvider.Builder. Check the existing configuration-binding and auto-configuration tests, then add the two inactive-pool settings and verify that configured intervals are passed to the builder and exposed through application properties.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spring-boot
Domain
api, backend, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.