spring-cloud / spring-cloud/spring-cloud-gateway
Lots of http 500 errors with PrematureCloseException and AbortedException in spring cloud gateway 4.1.5
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 4.9k
- Forks
- 3.5k
- Avg merge
- 20h 57m
- Merged PRs (30d)
- 8
Description
Describe the bug
In Spring cloud gateway we are seeing lots of reactor.netty.http.client.PrematureCloseException and reactor.netty.channel.AbortedException: Connection has been closed BEFORE send operation leading to 500 errors . Many downstream api are being called from gateway and we could see these errors for most of the apis. Currently we are relying on default timeouts of gateway (not sure if changing it would help ?) . Did lots of reading about these issues and found it could happen because of multiple reasons.
Below are the lib versions we are using :
Spring cloud starter gateway : 4.1.5
Spring boot starter : 3.2.12
reactor-netty-http : 1.1.24
reactor-netty-core : 1.1.24
jdk : jdk 17
Logs for both types of exceptions
500 Server Error for HTTP GET '/healthcheck'reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response##<>## Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: ##<>##Error has been observed at the following site(s):##<>## *__checkpoint ⇢ org.springframework.cloud.gateway.filter.Weig"
500 Server Error for HTTP GET '/healthcheck'reactor.netty.channel.AbortedException: Connection has been closed BEFORE send operation##<>##\tat reactor.netty.channel.AbortedException.beforeSend(AbortedException.java:59)##<>##\tSuppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: ##<>##Error has been observed at the following site(s):##<>##\t*__checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
In current setup , Gateway calls backend services via internal AWS application load balancer which has idle timeout of 60 seconds.
What we tried
We were earlier on Spring cloud starter gateway : 3.1.8 , but later moved to latest version thinking it might help us to resolve the issue. But issue is present in both the versions. Any help is appreciated to debug and fix this issue.
We have raised this concern earlier also https://github.com/spring-cloud/spring-cloud-gateway/issues/3461 , but we were asked to upgrade the lib versions
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the Spring Cloud Gateway 4.1.5 setup, the reported PrematureCloseException and AbortedException logs, and the earlier issue #3461. Compare the gateway's default timeout behavior with the internal AWS application load balancer's 60-second idle timeout, then document a reproducible case and identify which component closes the connection.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spring, spring-boot
- Domain
- api, backend, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100