Intermittent SqlException A connection was successfully established with the server, but then an error occurred during the pre-login handshake
- Dominant language
- C#
- Stars
- 989
- Forks
- 340
- Avg merge
- 4d 19h
- Merged PRs (30d)
- 72
Description
Our application:
- Dotnet Core 6 Web API
- Hosted on two separate Azure App Services; one scaled to 30 instances and one to 10.
- SQL Server Enterprise 2019 (15.0.4280.7) running on Windows Server 2019 Datacenter (10.0)
- EF Core version 6.0.2.0
- Microsoft.Data.SqlClient version 5.1.1 (we were seeing this same behavior with v 4 as well)
We are running into an intermittent issue (about once every other week) where we receive a SqlException with the message "A connection was successfully established with the server, but then an error occurred during the pre-login handshake. (provider: SSL Provider, error: 0 - The wait operation timed out.)" and the following two call stacks:
` Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(Microsoft.Data.SqlClient.TdsParserStateObject, Boolean, Boolean)
Microsoft.Data.SqlClient.TdsParser.ConsumePreLoginHandshake(Boolean, Boolean, Boolean, Boolean ByRef, Boolean ByRef)
Microsoft.Data.SqlClient.TdsParser.Connect(Microsoft.Data.SqlClient.ServerInfo, Microsoft.Data.SqlClient.SqlInternalConnectionTds, Boolean, Int64, Boolean, Boolean, Boolean, Boolean, Microsoft.Data.SqlClient.SqlAuthenticationMethod)
Microsoft.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(Microsoft.Data.SqlClient.ServerInfo, System.String, System.Security.SecureString, Boolean, Microsoft.Data.ProviderBase.TimeoutTimer, Boolean)
Microsoft.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(Microsoft.Data.SqlClient.ServerInfo, System.String, System.Security.SecureString, Boolean, Microsoft.Data.SqlClient.SqlConnectionString, Microsoft.Data.SqlClient.SqlCredential, Microsoft.Data.ProviderBase.TimeoutTimer)
Microsoft.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(Microsoft.Data.ProviderBase.TimeoutTimer, Microsoft.Data.SqlClient.SqlConnectionString, Microsoft.Data.SqlClient.SqlCredential, System.String, System.Security.SecureString, Boolean)
Microsoft.Data.SqlClient.SqlInternalConnectionTds..ctor(Microsoft.Data.ProviderBase.DbConnectionPoolIdentity, Microsoft.Data.SqlClient.SqlConnectionString, Microsoft.Data.SqlClient.SqlCredential, System.Object, System.String, System.Security.SecureString, Boolean, Microsoft.Data.SqlClient.SqlConnectionString, Microsoft.Data.SqlClient.SessionData, Boolean, System.String, Microsoft.Data.ProviderBase.DbConnectionPool)
Microsoft.Data.SqlClient.SqlConnectionFactory.CreateConnection(Microsoft.Data.Common.DbConnectionOptions, Microsoft.Data.Common.DbConnectionPoolKey, System.Object, Microsoft.Data.ProviderBase.DbConnectionPool, System.Data.Common.DbConnection, Microsoft.Data.Common.DbConnectionOptions)
Microsoft.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(Microsoft.Data.ProviderBase.DbConnectionPool, System.Data.Common.DbConnection, Microsoft.Data.Common.DbConnectionOptions, Microsoft.Data.Common.DbConnectionPoolKey, Microsoft.Data.Common.DbConnectionOptions)
Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(System.Data.Common.DbConnection, Microsoft.Data.Common.DbConnectionOptions, Microsoft.Data.ProviderBase.DbConnectionInternal)
Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(System.Data.Common.DbConnection, Microsoft.Data.Common.DbConnectionOptions, Microsoft.Data.ProviderBase.DbConnectionInternal)
Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(System.Data.Common.DbConnection, UInt32, Boolean, Boolean, Microsoft.Data.Common.DbConnectionOptions, Microsoft.Data.ProviderBase.DbConnectionInternal ByRef)
Microsoft.Data.ProviderBase.DbConnectionPool.WaitForPendingOpen()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
HealthChecks.SqlServer.SqlServerHealthCheck+d__4.MoveNext() `
`Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(System.Data.Common.DbConnection, UInt32, Boolean, Boolean, Microsoft.Data.Common.DbConnectionOptions, Microsoft.Data.ProviderBase.DbConnectionInternal ByRef)
Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(System.Data.Common.DbConnection, System.Threading.Tasks.TaskCompletionSource`1, Microsoft.Data.Common.DbConnectionOptions, Microsoft.Data.ProviderBase.DbConnectionInternal ByRef)
Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(System.Data.Common.DbConnection, System.Threading.Tasks.TaskCompletionSource`1, Microsoft.Data.Common.DbConnectionOptions, Microsoft.Data.ProviderBase.DbConnectionInternal, Microsoft.Data.ProviderBase.DbConnectionInternal ByRef)
Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(System.Data.Common.DbConnection, Microsoft.Data.ProviderBase.DbConnectionFactory, System.Threading.Tasks.TaskCompletionSource`1, Microsoft.Data.Common.DbConnectionOptions)
Microsoft.Data.SqlClient.SqlConnection.TryOpen(System.Threading.Tasks.TaskCompletionSource`1, Microsoft.Data.SqlClient.SqlConnectionOverrides)
Microsoft.Data.SqlClient.SqlConnection.InternalOpenAsync(System.Threading.CancellationToken)
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
HealthChecks.SqlServer.SqlServerHealthCheck+d__4.MoveNext()`
When this issue occurs, the affected instance (it's usually just a single instance, but occasionally two) cannot open any database connections via SqlConnection.Open until it is restarted. However, all other instances in the app service are able to connect to the same database servers without any issue during this time.
Additional information:
- We have multiple different connections strings in the application pointing to different database servers and/or databases. When this issue occurs, all of these are affected and none of the connections can be used.
- We are using the default connection timeout of 15 seconds.
- We have quarantined the containing app service (no active requests) for days while trying to troubleshoot and the issue persists until the instance is restarted.
- We are using SQL Authentication
- Our connection string has Encrypt=true and TrustServerCertificate=true (we use a self-signed cert). We have changed one of the connection strings to use Encrypt=false and TrustServerCertificate=false to test if this is SSL related the next time the issue occurs.
- We can provide dmp and nettrace files collected while the issue was occurring.
- We currently restart our app services daily and it seems to prevent this issue from happening as often
- We have not yet been able to replicate this outside of our production environment
- The dump files have shown blocking threads waiting on SNI operations such as this:
Blocking thread stack:
ntdll!NtDelayExecution+14
KERNELBASE!SleepEx+a7
Microsoft_Data_SqlClient_SNI!SNIClose+9a
Microsoft_Data_SqlClient_SNI!SNICloseWrapper+15
Microsoft.Data.SqlClient.SNIHandle.ReleaseHandle()+72
System.Runtime.InteropServices.SafeHandle.InternalRelease(Boolean)+ba
Microsoft.Data.SqlClient.TdsParserStateObjectNative.Dispose()+91
Microsoft.Data.SqlClient.TdsParser.Disconnect()+81
Microsoft.Data.SqlClient.SqlInternalConnectionTds.Dispose()+5b
Microsoft.Data.ProviderBase.DbConnectionPool.DestroyObject(Microsoft.Data.ProviderBase.DbConnectionInternal)+152
Microsoft.Data.ProviderBase.DbConnectionPool.DeactivateObject(Microsoft.Data.ProviderBase.DbConnectionInternal)+16a
Microsoft.Data.ProviderBase.DbConnectionPool.PutObject(Microsoft.Data.ProviderBase.DbConnectionInternal, System.Object)+8a
Microsoft.Data.ProviderBase.DbConnectionInternal.CloseConnection(System.Data.Common.DbConnection, Microsoft.Data.ProviderBase.DbConnectionFactory)+de
Microsoft.Data.SqlClient.SqlConnection.Close()+22e
Microsoft.Data.SqlClient.SqlConnection.OnError(Microsoft.Data.SqlClient.SqlException, Boolean, System.Action`1)+cc
Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(Microsoft.Data.SqlClient.TdsParserStateObject, Boolean, Boolean)+33b
Microsoft.Data.SqlClient.TdsParserStateObject.SNIWritePacket(Microsoft.Data.SqlClient.PacketHandle, UInt32 ByRef, Boolean, Boolean)+2b9
Microsoft.Data.SqlClient.TdsParserStateObject.SendAttention(Boolean)+19c
Microsoft.Data.SqlClient.TdsParserStateObject.OnTimeoutCore(Int32, Int32)+220
Microsoft.Data.SqlClient.TdsParserStateObject.ReadAsyncCallback(IntPtr, Microsoft.Data.SqlClient.PacketHandle, UInt32)+142
Microsoft.Data.SqlClient.SNILoadHandle.ReadDispatcher(IntPtr, IntPtr, UInt32)+a5
ILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, UInt32)+50
Microsoft_Data_SqlClient_SNI!UnmanagedReadCallback+27
Microsoft_Data_SqlClient_SNI!SNI_Conn::SafeReadComp+18
Microsoft_Data_SqlClient_SNI!SNIReadDone+134
Microsoft_Data_SqlClient_SNI!SNIAsyncWait+10a
kernel32!BaseThreadInitThunk+14
ntdll!RtlUserThreadStart+21
Blocked thread stack appears to be waiting on lock(this) in TdsParserStateObject.ResetCancelAndProcessAttention:
ntdll!NtWaitForMultipleObjects+14
KERNELBASE!WaitForMultipleObjectsEx+ef
coreclr!Thread::DoAppropriateWaitWorker+14f [D:\a\_work\1\s\src\coreclr\vm\threads.cpp @ 3526 + 41] D:\a\_work\1\s\src\coreclr\vm\threads.cpp @ 3526 + 41
coreclr!Thread::DoAppropriateWait+89 [D:\a\_work\1\s\src\coreclr\vm\threads.cpp @ 3243 + 15] D:\a\_work\1\s\src\coreclr\vm\threads.cpp @ 3243 + 15
coreclr!CLREventBase::Wait+66 [D:\a\_work\1\s\src\coreclr\vm\synch.cpp @ 413] D:\a\_work\1\s\src\coreclr\vm\synch.cpp @ 413
coreclr!AwareLock::EnterEpilogHelper+139 [D:\a\_work\1\s\src\coreclr\vm\syncblk.cpp @ 2609] D:\a\_work\1\s\src\coreclr\vm\syncblk.cpp @ 2609
coreclr!AwareLock::EnterEpilog+42 [D:\a\_work\1\s\src\coreclr\vm\syncblk.cpp @ 2487 + 5] D:\a\_work\1\s\src\coreclr\vm\syncblk.cpp @ 2487 + 5
coreclr!AwareLock::Enter+ae [D:\a\_work\1\s\src\coreclr\vm\syncblk.cpp @ 2389 + f] D:\a\_work\1\s\src\coreclr\vm\syncblk.cpp @ 2389 + f
coreclr!JIT_MonEnter_Helper+103 [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp @ 3670] D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp @ 3670
coreclr!JIT_MonReliableEnter_Portable+d7 [D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp @ 3718 + 12] D:\a\_work\1\s\src\coreclr\vm\jithelpers.cpp @ 3718 + 12
Microsoft.Data.SqlClient.TdsParserStateObject.ResetCancelAndProcessAttention()+27
Microsoft.Data.SqlClient.TdsParserStateObject.CloseSession()+10
Microsoft.Data.SqlClient.SqlCommand.EndExecuteReaderAsync(System.IAsyncResult)+21b
System.Threading.Tasks.TaskFactory`1[[System.__Canon, System.Private.CoreLib]].FromAsyncCoreLogic(System.IAsyncResult, System.Func`2, System.Action`1, System.Threading.Tasks.Task`1, Boolean)+2b
System.Threading.Tasks.TaskFactory`1+<>c__DisplayClass35_0[[System.__Canon, System.Private.CoreLib]].b__0(System.IAsyncResult)+7d
System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread, System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)+4b
System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)+b6
System.Threading.ThreadPoolWorkQueue.Dispatch()+1fa
System_Private_CoreLib!System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()+37a6e0be
System_Private_CoreLib!System.Threading.Thread.StartCallback()+417ef0df
coreclr!CallDescrWorkerInternal+83
coreclr!DispatchCallSimple+80 [D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp @ 220 + 5] D:\a\_work\1\s\src\coreclr\vm\callhelpers.cpp @ 220 + 5
coreclr!ThreadNative::KickOffThread_Worker+63 [D:\a\_work\1\s\src\coreclr\vm\comsynchronizable.cpp @ 158] D:\a\_work\1\s\src\coreclr\vm\comsynchronizable.cpp @ 158
coreclr!ManagedThreadBase_DispatchMiddle+85 [D:\a\_work\1\s\src\coreclr\vm\threads.cpp @ 7365 + d] D:\a\_work\1\s\src\coreclr\vm\threads.cpp @ 7365 + d
coreclr!ManagedThreadBase_DispatchOuter+ae [D:\a\_work\1\s\src\coreclr\vm\threads.cpp @ 7524] D:\a\_work\1\s\src\coreclr\vm\threads.cpp @ 7524
coreclr!ThreadNative::KickOffThread+79 [D:\a\_work\1\s\src\coreclr\vm\comsynchronizable.cpp @ 230] D:\a\_work\1\s\src\coreclr\vm\comsynchronizable.cpp @ 230
kernel32!BaseThreadInitThunk+14
Contributor guide
Assessment
This issue has not been assessed yet.