spring-cloud / spring-cloud/spring-cloud-gateway

Lots of http 500 errors with PrematureCloseException and AbortedException in spring cloud gateway 4.1.5

Open
#3,758 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

waiting-for-triage
Dominant language
Java
Stars
4.9k
Forks
3.5k
Avg merge
20h 57m
Merged PRs (30d)
8

Description

Describe the bug
In Spring cloud gateway we are seeing lots of reactor.netty.http.client.PrematureCloseException and reactor.netty.channel.AbortedException: Connection has been closed BEFORE send operation leading to 500 errors . Many downstream api are being called from gateway and we could see these errors for most of the apis. Currently we are relying on default timeouts of gateway (not sure if changing it would help ?) . Did lots of reading about these issues and found it could happen because of multiple reasons.

Below are the lib versions we are using :

Spring cloud starter gateway : 4.1.5
Spring boot starter : 3.2.12
reactor-netty-http : 1.1.24
reactor-netty-core : 1.1.24
jdk : jdk 17

Logs for both types of exceptions

500 Server Error for HTTP GET '/healthcheck'reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response##<>## Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: ##<>##Error has been observed at the following site(s):##<>## *__checkpoint ⇢ org.springframework.cloud.gateway.filter.Weig"

500 Server Error for HTTP GET '/healthcheck'reactor.netty.channel.AbortedException: Connection has been closed BEFORE send operation##<>##\tat reactor.netty.channel.AbortedException.beforeSend(AbortedException.java:59)##<>##\tSuppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: ##<>##Error has been observed at the following site(s):##<>##\t*__checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]

In current setup , Gateway calls backend services via internal AWS application load balancer which has idle timeout of 60 seconds.

What we tried
We were earlier on Spring cloud starter gateway : 3.1.8 , but later moved to latest version thinking it might help us to resolve the issue. But issue is present in both the versions. Any help is appreciated to debug and fix this issue.

We have raised this concern earlier also https://github.com/spring-cloud/spring-cloud-gateway/issues/3461 , but we were asked to upgrade the lib versions

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review the Spring Cloud Gateway 4.1.5 setup, the reported PrematureCloseException and AbortedException logs, and the earlier issue #3461. Compare the gateway's default timeout behavior with the internal AWS application load balancer's 60-second idle timeout, then document a reproducible case and identify which component closes the connection.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spring, spring-boot
Domain
api, backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.