dotnet / dotnet/aspnetcore

ConnectionResetException flood

Open
#53,591 6 comments 0 reactions 0 assignees View on GitHub
area-networking
Dominant language
C#
Stars
38.4k
Forks
10.9k
Avg merge
2d 10h
Merged PRs (30d)
281

Description

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Describe the bug

Our ASP.NET Core 6 services, run behind HAProxy, are flooded with ConnectionResetException.

In this board you can see a very high number of exceptions.
![image](https://github.com/dotnet/aspnetcore/assets/9092290/1f6aee71-e3e4-4e90-b9a7-954db4321b65)
By logging the first chance exceptions it seems to come from
```
Microsoft.AspNetCore.Connections.ConnectionResetException: Connection reset by peer
---> System.Net.Sockets.SocketException (104): Connection reset by peer
at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection.DoReceive()
--- End of inner exception stack trace ---
at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
at System.IO.Pipelines.Pipe.GetReadAsyncResult()
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequestsAsync[TContext](IHttpApplication`1 application)
```
By collecting the exceptions from the event pipe we have similar results:
![image](https://github.com/dotnet/aspnetcore/assets/9092290/3c8b02a7-845e-4fd1-915a-b0439e3da60d)

We believe that issue appeared after this HAProxy change: https://github.com/haproxy/haproxy/commit/4d1ff11f05691aa6820a985c31e72811cf9ef95d which introduced a new way to kill idle connections by sending a RST instead of a FIN ACK.

This create two problems on our side:
1. We can't monitor the rate of exceptions anymore
2. These ConnectionResetException also seem to induce lock contention which impact the performance on of service
![image](https://github.com/dotnet/aspnetcore/assets/9092290/8b5c9dda-b523-4a0e-b0a8-067d99e97866)

### Expected Behavior

Clients should not have a way to flood a server with exceptions.

### Steps To Reproduce

```cs
using System.Net;
using System.Net.Sockets;
using System.Text;

var builder = WebApplication.CreateBuilder(args);

var app = builder.Build();

AppDomain.CurrentDomain.FirstChanceException += (sender, eventArgs) =>
{
Console.WriteLine(eventArgs.Exception.ToString());
};

Task.Run(async () =>
{
await Task.Delay(2000);
Console.WriteLine("Opening");
using Socket socket = new(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
await socket.ConnectAsync(new DnsEndPoint("localhost", 5233));
await Task.Delay(2000);
string reqLine = "POST / HTTP/1.1\r\nHost: example.org\r\nContent-Length: 10000000\r\n\r\n";
socket.Send(Encoding.ASCII.GetBytes(reqLine));

socket.Send(new byte[100_000]);
socket.Send(new byte[100_000]);
socket.Send(new byte[100_000]);

await Task.Delay(2000);

Console.WriteLine("Closing");
socket.Close();
});

app.Run();
```

With this code I'm not getting the exact same exception but something similar. It is reproducible on both .NET 6 and 8.

```
Microsoft.AspNetCore.Connections.ConnectionResetException: An existing connection was forcibly closed by the remote host.
---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
--- End of inner exception stack trace ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
```

### Exceptions (if any)

_No response_

### .NET Version

6.0.25

### Anything else?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.