SqlClient fails to connect to SGX Secure Enclave backed databases after database events
- Dominant language
- C#
- Stars
- 989
- Forks
- 340
- Avg merge
- 4d 18h
- Merged PRs (30d)
- 69
Description
### Describe the bug
Connectivity is permanently lost when accessing an Azure SQL database on DC-series hardware with SGX enclaves with SqlClient after an "event" on the Azure SQL database. This results in buried exceptions in SqlClient and enclave errors from the database. No exceptions are surfaced in the calling application and the only application-side manifestation (beyond failure) are the below Kestrel errors.
To restore connectivity, all impacted applications using SqlClient must be restarted.
```
Exception message (Application Side):
[05:34:47 FTL] Microsoft.AspNetCore.Server.Kestrel Connection id '0HMVNE3J76L7U' application never completed.
[05:34:47 FTL] Microsoft.AspNetCore.Server.Kestrel Connection id '0HMVNE3J76L7V' application never completed.
[05:34:47 FTL] Microsoft.AspNetCore.Server.Kestrel Connection id '0HMVNE3J76L87' application never completed.
Stack trace (Application Side):
same as above, no exceptions.
Exception message (Database Side):
Internal enclave error. Enclave was provided with an invalid session handle. For more information, contact Customer Support Services.
The service has encountered an error processing your request. Please try again. Error code 33195.
Stack trace (Database Side):
n/a
```
### To reproduce
This example is based on the assumption that the database being connected to is an Azure SQL DB on DC-series hardware with SGX Secure Enclaves. This works under nominal conditions, and consistently fails when one of the below pre-requisite repro steps are taken.
Pre-requisite reproduction steps (one of the following actions must be taken, there may be other triggering events, but these have been observed to cause it to date):
- Scale an elastic pool containing the database from x vCores to y vCores
- Have an Azure automated maintenance event occur. This should incur minimal downtime as described, but SqlClient _never_ recovers from this.
```c#
await using var conn = new SqlConnection("Connection string to AzureSQL with Enclave Attestation");
var command = new SqlCommand("select top 10 * from MyTable", conn);
await using var reader = await command.ExecuteReaderAsync();
while (await reader.ReadAsync())
{
// logic
}
```
### Expected behavior
- SqlClient to successfully make a request and not require a full restart of the owning application.
- Surfacing of any exceptions from SqlClient
### Further technical details
Microsoft.Data.SqlClient version: 5.1.x
.NET target: .NET 6, 8
SQL Server version: Azure SQL Database, DC-series hardware, SGX secure enclaves, elastic pool.
Operating system: (e.g. Windows 2019, Ubuntu 18.04, macOS 10.13, Docker container)
**Additional context**
There are likely other triggering events from the Azure SQL side to reproduce this issue, but the two we've noted thus far are:
- Scaling an elastic pool
- Automated Azure SQL maintenance
When either occur, all applications accessing these DBs with SqlClient must be restarted. All applications are hosted in AKS clusters.
Contributor guide
Assessment
This issue has not been assessed yet.