spring-cloud / spring-cloud/spring-cloud-gateway

Gateway Retry Filter Causes Gateway to Leak Memory

Open
#988 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

in progress
Dominant language
Java
Stars
4.9k
Forks
3.5k
Avg merge
20h 57m
Merged PRs (30d)
8

Description

Currently under real world test scenario we are finding that the gateway will begin to leak after approximately 20 minutes while under load.

From my testing it seems like the leak will only appear if the retry filter is being used. No retry filter, no leaks. The config we are using for the retry filter is:

.retry(retryConfig -> retryConfig.setRetries(DEFAULT_RETRIES)
	.setExceptions(RetryableHttpClientErrorException.class, IOException.class, TimeoutException.class)
	.setStatuses(HttpStatus.UNAUTHORIZED)
	.setMethods(HttpMethod.GET, HttpMethod.POST))

There have been other comments regarding gateway memory leaks and those users have reported that they are only seeing the leaks when the retry filter is enabled.

Stack traces are the same as what has been seen here:

https://github.com/spring-cloud/spring-cloud-gateway/issues/810

I have tried to make a reproducible example but am currently blocked by the errors coming up here:

https://github.com/spring-cloud/spring-cloud-gateway/issues/982

My latest test was done using the following.

Spring Boot 2.1.3 Release
Greenwich.SR1
Netty 4.1.34.Final
Californium-SR6
Spring Framework 5.1.6.BUILD-SNAPSHOT

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the stack traces and related reports in issue 810, then investigate the retry filter using the configuration and dependency versions listed here. Reproduce the leak under load, accounting for the blocked reproduction described in issue 982; done means the retry-enabled gateway no longer shows the reported memory growth.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spring, spring-boot
Domain
api, backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.