spring-cloud / spring-cloud/spring-cloud-gateway
Gateway Retry Filter Causes Gateway to Leak Memory
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 4.9k
- Forks
- 3.5k
- Avg merge
- 20h 57m
- Merged PRs (30d)
- 8
Description
Currently under real world test scenario we are finding that the gateway will begin to leak after approximately 20 minutes while under load.
From my testing it seems like the leak will only appear if the retry filter is being used. No retry filter, no leaks. The config we are using for the retry filter is:
.retry(retryConfig -> retryConfig.setRetries(DEFAULT_RETRIES)
.setExceptions(RetryableHttpClientErrorException.class, IOException.class, TimeoutException.class)
.setStatuses(HttpStatus.UNAUTHORIZED)
.setMethods(HttpMethod.GET, HttpMethod.POST))
There have been other comments regarding gateway memory leaks and those users have reported that they are only seeing the leaks when the retry filter is enabled.
Stack traces are the same as what has been seen here:
https://github.com/spring-cloud/spring-cloud-gateway/issues/810
I have tried to make a reproducible example but am currently blocked by the errors coming up here:
https://github.com/spring-cloud/spring-cloud-gateway/issues/982
My latest test was done using the following.
Spring Boot 2.1.3 Release
Greenwich.SR1
Netty 4.1.34.Final
Californium-SR6
Spring Framework 5.1.6.BUILD-SNAPSHOT
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the stack traces and related reports in issue 810, then investigate the retry filter using the configuration and dependency versions listed here. Reproduce the leak under load, accounting for the blocked reproduction described in issue 982; done means the retry-enabled gateway no longer shows the reported memory growth.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spring, spring-boot
- Domain
- api, backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100