spring-cloud / spring-cloud/spring-cloud-gateway

RetryFilter uses memory multiple times the size of incoming requests

Open
#3,741 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

waiting-for-triage
Dominant language
Java
Stars
4.9k
Forks
3.5k
Avg merge
20h 57m
Merged PRs (30d)
8

Description

Spring Cloud Gateway 4.2.0
Spring Boot 3.4.3

We have a gateway configured with a retry filter. The filter is configured to only retry on GET method calls, similar to the default config.
We have noticed that for POST calls, when the retry filter is present, lots of buffer memory is used.
If we remove the retry filter completely, then the memory usage drops significantly.
The buffer memory that gets allocated is significantly more than the size of the incoming request.

In our tests, when the incoming post was 500MB, then the memory used was around 2GB.

The problem is in three parts:

  1. Too much buffer memory is allocated. Why does it need 4 times the file size allocated as a buffer?
  2. and not all of it is released. The graph below shows the memory trending upwards after each call. Each request seems to allocate 2GB and then only releases 1.5GB. (This one I am not so sure about, perhaps GC would release some more memory eventually)
  3. The memory is all allocated even if the filter is configured not to retry e.g. even if the method does not match the predicates.
Memory usage for POST requests when retry filter is active only for GET requests

Image

Memory usage for POST request when retry filter is disabled

Image

Retry Filter Config
        FilterDefinition retryFilterDef = new FilterDefinition();
        retryFilterDef.setName("Retry");
        retryFilterDef.addArg("retries", "3");
        //The default config for the methods is GET only. Added just to double check.
        retryFilterDef.addArg("methods", "GET");

Attached is a test case to show the memory usage. It logs some basic memory stats, but it's easier to just look in JVisualVM.

repro.zip

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the attached repro.zip and its test case, using the shown Retry filter configuration with GET-only predicates. Trace the buffer allocation and release path for large POST requests, including requests that do not match the retry method, and compare memory usage with the filter disabled. Done means unnecessary buffering is avoided and allocated buffers are released without changing retry behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spring-boot
Domain
api, backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.