dotnet / dotnet/aspnetcore

Conectivity issue when upgrading from .NET 7.0.1 to .NET 7.0.4 on Windows server

Open
#48,965 9 comments 0 reactions 0 assignees View on GitHub
area-networking Needs: Attention :wave:
Dominant language
C#
Stars
38.4k
Forks
10.9k
Avg merge
2d 10h
Merged PRs (30d)
281

Description

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Describe the bug

Our product is a distributed system consisting of multiple ASP servers.
There are TCP connections open between the servers.
And the servers also communicate with HTTP (version 1.1).

## The issue:
The server doesn’t accept HTTP requests between the servers.
Even not from the server itself.
No errors on Microsoft logs (the configuration below) on the target server.
In the source server, we got:
```
Raven.Client.Exceptions.RavenException: An exception occurred while contacting ***URL***.
System.Net.Http.HttpRequestException: No connection could be made because the target machine actively refused it. (***URL***:443)
---> System.Net.Sockets.SocketException (10061): No connection could be made because the target machine actively refused it.
```
(full stack trace below)
The issue starts between 1 hour to a couple of hours after a restart.
Azure firewall and OS firewall were checked.

* Is there a known issue with .Net versions greater than .Net7.0.1?
* Are there additional configurations that were added or maybe an old one that wasn't respected and was fixed?
* What additional information can we collect to identify the issue?

## Some details:
One of our customers had an issue in a production environment when upgrading our product.
This customer tested the upgrade in a test environment but no issue there.
We upgraded the servers again to collect more information.
We collected Microsoft logs (the configuration below) and also `tcpdump`.
In the Microsoft logs, we saw no error.
On `tcpdump` that was collected (for 3 minutes) for the target server, we see all HTTPS packages to the port the server is listening to has no response packages `[Conversation completeness: Incomplete, SYN_SENT (1)]`.
We didn’t have any changes regarding the server communication handling between our product versions besides upgrading `.Net` from `.Net6` to `.Net7` so we created the same build with one difference, we used `.Net6` instead and the issue was solved.
Since this is a production we deployed the `.Net7` version twice after the first issue to collect the logs and the issue happened again on both times.
We also saw the issue again in another customer system, this time the .Net upgrade was between .NET 7.0.1 to .NET 7.0.4 - again using `.Net6` build solved the issue.

## Both customers’ servers were on:
* Windows Server 2019 Datacenter.
* WindowsVersion:10.0 BuildVersion:17763 (1809)
* We tried to reproduce it on Azure with no success.
* We don't see it in our Linux customers.

## Addition finding:
* It seems the TCP connections between the nodes are working fine.
* In Microsoft logs, we don’t see any errors or warnings.
* In Microsoft logs `Microsoft.AspNetCore.Hosting.Diagnostics`, we see only HTTP/2 (this is external communication typically when you access the server from a browser).
* In Microsoft logs, we don't see any sign of other requests even those that come from the server itself.
* From `tcpdump` there are no packages that go out from the HTTPS port even though there are packages that go in.
* Requests from outside of the cluster seem to work (We did see at the first time of issue occurrence that after one hour the requests from outside started to fail but we don’t have logs/`tcpdump` from this time).

`tcpdump` was collected for 3 minutes while the issue happened

## Microsoft log configuration:
```
{
"Microsoft.AspNetCore.Server.Kestrel": "Debug",
"Microsoft.AspNetCore.Server.Kestrel.BadRequests": "Debug"
"Microsoft.AspNetCore.Server.Kestrel.Connections": "Debug",
"Microsoft.AspNetCore.Server.Kestrel.Http2": "Debug",
"Microsoft.AspNetCore.Server.Kestrel.Http3": "Debug",

"Microsoft.AspNetCore.Server.Kestrel.Transport.Quic": "Debug",
"Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets": "Debug",

"Microsoft.AspNetCore.Hosting.WebHost": "Debug",
"Microsoft.AspNetCore.Hosting.Diagnostics": "Information",
}
```

### Expected Behavior

_No response_

### Steps To Reproduce

We couldn't reproduce

### Exceptions (if any)

## Exception Stack Trace:
```
Raven.Client.Exceptions.RavenException: An exception occurred while contacting ***URL***.
System.Net.Http.HttpRequestException: No connection could be made because the target machine actively refused it. (***URL***:443)
---> System.Net.Sockets.SocketException (10061): No connection could be made because the target machine actively refused it.
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Sockets.Socket.g__WaitForConnectWithCancellation|281_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)
at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.HttpConnectionWaiter`1.WaitForConnectionAsync(Boolean async, CancellationToken requestCancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at Raven.Client.Http.RequestExecutor.SendAsync[TResult](ServerNode chosenNode, RavenCommand`1 command, SessionInfo sessionInfo, HttpRequestMessage request, CancellationToken token) in C:\Builds\RavenDB-Stable-5.4\54038\src\Raven.Client\Http\RequestExecutor.cs:line 1098
at Raven.Client.Http.RequestExecutor.SendRequestToServer[TResult](ServerNode chosenNode, Nullable`1 nodeIndex, JsonOperationContext context, RavenCommand`1 command, Boolean shouldRetry, SessionInfo sessionInfo, HttpRequestMessage request, String url, CancellationToken token) in C:\Builds\RavenDB-Stable-5.4\54038\src\Raven.Client\Http\RequestExecutor.cs:line 1050.
The server at ***URL*** responded with status code: ServiceUnavailable.
---> System.Net.Http.HttpRequestException: No connection could be made because the target machine actively refused it. (***URL***:443)
---> System.Net.Sockets.SocketException (10061): No connection could be made because the target machine actively refused it.
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Sockets.Socket.g__WaitForConnectWithCancellation|281_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)
at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.HttpConnectionWaiter`1.WaitForConnectionAsync(Boolean async, CancellationToken requestCancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at Raven.Client.Http.RequestExecutor.SendAsync[TResult](ServerNode chosenNode, RavenCommand`1 command, SessionInfo sessionInfo, HttpRequestMessage request, CancellationToken token) in C:\Builds\RavenDB-Stable-5.4\54038\src\Raven.Client\Http\RequestExecutor.cs:line 1098
at Raven.Client.Http.RequestExecutor.SendRequestToServer[TResult](ServerNode chosenNode, Nullable`1 nodeIndex, JsonOperationContext context, RavenCommand`1 command, Boolean shouldRetry, SessionInfo sessionInfo, HttpRequestMessage request, String url, CancellationToken token) in C:\Builds\RavenDB-Stable-5.4\54038\src\Raven.Client\Http\RequestExecutor.cs:line 1050
--- End of inner exception stack trace ---
at Raven.Client.Http.RequestExecutor.ThrowFailedToContactAllNodes[TResult](RavenCommand`1 command, HttpRequestMessage request) in C:\Builds\RavenDB-Stable-5.4\54038\src\Raven.Client\Http\RequestExecutor.cs:line 1177
at Raven.Client.Http.RequestExecutor.SendRequestToServer[TResult](ServerNode chosenNode, Nullable`1 nodeIndex, JsonOperationContext context, RavenCommand`1 command, Boolean shouldRetry, SessionInfo sessionInfo, HttpRequestMessage request, String url, CancellationToken token) in C:\Builds\RavenDB-Stable-5.4\54038\src\Raven.Client\Http\RequestExecutor.cs:line 1050
at Raven.Client.Http.RequestExecutor.ExecuteAsync[TResult](ServerNode chosenNode, Nullable`1 nodeIndex, JsonOperationContext context, RavenCommand`1 command, Boolean shouldRetry, SessionInfo sessionInfo, CancellationToken token) in C:\Builds\RavenDB-Stable-5.4\54038\src\Raven.Client\Http\RequestExecutor.cs:line 919
at Raven.Client.Http.RequestExecutor.HandleServerDown[TResult](String url, ServerNode chosenNode, Nullable`1 nodeIndex, JsonOperationContext context, RavenCommand`1 command, HttpRequestMessage request, HttpResponseMessage response, Exception e, SessionInfo sessionInfo, Boolean shouldRetry, RequestContext requestContext, CancellationToken token) in C:\Builds\RavenDB-Stable-5.4\54038\src\Raven.Client\Http\RequestExecutor.cs:line 1560
at Raven.Client.Http.RequestExecutor.SendRequestToServer[TResult](ServerNode chosenNode, Nullable`1 nodeIndex, JsonOperationContext context, RavenCommand`1 command, Boolean shouldRetry, SessionInfo sessionInfo, HttpRequestMessage request, String url, CancellationToken token) in C:\Builds\RavenDB-Stable-5.4\54038\src\Raven.Client\Http\RequestExecutor.cs:line 1050
at Raven.Client.Http.RequestExecutor.ExecuteAsync[TResult](ServerNode chosenNode, Nullable`1 nodeIndex, JsonOperationContext context, RavenCommand`1 command, Boolean shouldRetry, SessionInfo sessionInfo, CancellationToken token) in C:\Builds\RavenDB-Stable-5.4\54038\src\Raven.Client\Http\RequestExecutor.cs:line 919
at Raven.Server.Utils.ReplicationUtils.GetTcpInfoAsync(String url, String databaseName, String databaseId, Int64 etag, String tag, X509Certificate2 certificate, CancellationToken token) in C:\Builds\RavenDB-Stable-5.4\54038\src\Raven.Server\Utils\ReplicationUtils.cs:line 51
at Raven.Server.Utils.ReplicationUtils.GetTcpInfoAsync(String url, String databaseName, String tag, X509Certificate2 certificate, CancellationToken token) in C:\Builds\RavenDB-Stable-5.4\54038\src\Raven.Server\Utils\ReplicationUtils.cs:line 36
at Raven.Client.Util.AsyncHelpers.RunSync[T](Func`1 task) in C:\Builds\RavenDB-Stable-5.4\54038\src\Raven.Client\Util\AsyncHelpers.cs:line 135
at Raven.Server.Utils.ReplicationUtils.GetTcpInfo(String url, String databaseName, String tag, X509Certificate2 certificate, CancellationToken token) in C:\Builds\RavenDB-Stable-5.4\54038\src\Raven.Server\Utils\ReplicationUtils.cs:line 30
at Raven.Server.ServerWide.Maintenance.ClusterMaintenanceSupervisor.ClusterNode.ListenToMaintenanceWorker() in C:\Builds\RavenDB-Stable-5.4\54038\src\Raven.Server\ServerWide\Maintenance\ClusterMaintenanceSupervisor.cs:line 274
```

### .NET Version

.NET 7.0.4

### Anything else?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.