spring-cloud / spring-cloud/spring-cloud-gateway
RetryFilter uses memory multiple times the size of incoming requests
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 4.9k
- Forks
- 3.5k
- Avg merge
- 20h 57m
- Merged PRs (30d)
- 8
Description
Spring Cloud Gateway 4.2.0
Spring Boot 3.4.3
We have a gateway configured with a retry filter. The filter is configured to only retry on GET method calls, similar to the default config.
We have noticed that for POST calls, when the retry filter is present, lots of buffer memory is used.
If we remove the retry filter completely, then the memory usage drops significantly.
The buffer memory that gets allocated is significantly more than the size of the incoming request.
In our tests, when the incoming post was 500MB, then the memory used was around 2GB.
The problem is in three parts:
- Too much buffer memory is allocated. Why does it need 4 times the file size allocated as a buffer?
- and not all of it is released. The graph below shows the memory trending upwards after each call. Each request seems to allocate 2GB and then only releases 1.5GB. (This one I am not so sure about, perhaps GC would release some more memory eventually)
- The memory is all allocated even if the filter is configured not to retry e.g. even if the method does not match the predicates.
Memory usage for POST requests when retry filter is active only for GET requests
Memory usage for POST request when retry filter is disabled
Retry Filter Config
FilterDefinition retryFilterDef = new FilterDefinition();
retryFilterDef.setName("Retry");
retryFilterDef.addArg("retries", "3");
//The default config for the methods is GET only. Added just to double check.
retryFilterDef.addArg("methods", "GET");
Attached is a test case to show the memory usage. It logs some basic memory stats, but it's easier to just look in JVisualVM.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the attached repro.zip and its test case, using the shown Retry filter configuration with GET-only predicates. Trace the buffer allocation and release path for large POST requests, including requests that do not match the retry method, and compare memory usage with the filter disabled. Done means unnecessary buffering is avoided and allocated buffers are released without changing retry behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spring-boot
- Domain
- api, backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100