dotnet / dotnet/SqlClient

Huge performance problem with async

Open
#1,562 14 comments 2 reactions 0 assignees View on GitHub
Area\Async Performance :chart_with_upwards_trend: Repro Available :heavy_check_mark:
Dominant language
C#
Stars
989
Forks
340
Avg merge
4d 19h
Merged PRs (30d)
72

Description

We are trying to scale an application that uses ASP.NET Identity (which is all async) and we have huge performance issues with all async work against SqlClient (we believe it has boiled down to this). We have investigated and tried many different things for a couple of weeks, and we now have very little isolated code that performs more or less equally bad all the time. The code has implementation of both sync and async calls with huge differences between them. We have read about the issues for big fields, but we see the problem on all our queries (without any big fields) and the reproduced slow calls below has no large fields.

We run in Azure with an App Service plan of P1V2 scaled to two instances and an SQL Azure Premium P1: 125 DTUs. We don't see any performance problems at all in SQL Azure. We have a VNET with a private link between the App Service and SQL Azure. We use Azure Load test to run the tests (and we can share the tests too if needed). We also get similar results when testing locally.
We use Stackify Retrace for monitoring and have enabled that through their extension.

**Sync code performance when running 10 instances with 250 threads:**
90%: 4,69 sek
Average: 1,417 sek
Open Connection: <0,001 sek
Select Logins: 0,002 sek

![Azure_load_test_sync_10_instances](https://user-images.githubusercontent.com/1352330/160374830-bcbdb03f-ac92-40be-b4eb-85d68d50a213.png)
![Stackify_overview_sync_10_instances](https://user-images.githubusercontent.com/1352330/160374917-b0738f4c-6211-4e6b-bf1c-35b9cdc17d53.png)
![Stackify_details_sync_10_instances](https://user-images.githubusercontent.com/1352330/160374947-51b96edb-04a3-48bf-9c86-6182d8b3d131.png)

**Async code performance when running 10 instances with 250 threads:**
90%: 5,38 sek
Average: 2,235 sek
Open Connection: 1,624 sek
Select Logins: 0,298 sek

![Azure_load_test_async_10_instances](https://user-images.githubusercontent.com/1352330/160375016-d1163e3b-e049-4a5b-b0c3-e0200c414ef7.png)
![Stackify_overview_async_10_instances](https://user-images.githubusercontent.com/1352330/160375040-c7f27114-b2a7-4623-ba8e-96e8e50da2fd.png)
![Stackify_details_async_10_instances](https://user-images.githubusercontent.com/1352330/160375069-b7db5512-1d32-4cfc-9448-46e3bb0af4e7.png)

**Sync code performance when running 45 instances with 250 threads, each thread making 10 calls after each other:**
90%: 40,21 sek
Average: 17,502 sek
Open Connection: <0,001 sek
Select Logins: 0,006 sek

![Azure_load_test_sync_45_instances](https://user-images.githubusercontent.com/1352330/160375218-1ea42384-18b6-49d0-b424-42ac307a8b27.png)
![Stackify_overview_sync_45_instances](https://user-images.githubusercontent.com/1352330/160375256-12a15df2-d39c-436d-9e70-faba2e5e4cc0.png)
![Stackify_details_sync_45_instances](https://user-images.githubusercontent.com/1352330/160375283-e3070b0c-ac88-413a-ae63-f81b8cc72a8d.png)

**Async code performance when running 45 instances with 250 threads, each thread making 10 calls after each other:**
90%: 80,97 sek
Average: 35,080 sek
Open Connection: 19,634 sek
Select Logins: 7,975 sek

![Azure_load_test_async_45_instances](https://user-images.githubusercontent.com/1352330/160375423-0ed6808c-5c90-4e36-bac1-a68a41fed5a3.png)
![Stackify_overview_async_45_instances](https://user-images.githubusercontent.com/1352330/160375461-a0db568e-cc6b-4b53-bc8b-599bd2f6ddef.png)
![Stackify_details_async_45_instances](https://user-images.githubusercontent.com/1352330/160375498-d37e36f8-7972-4714-8529-a4ab5f7b10c5.png)

**Table script**:

```sql
CREATE TABLE [security].[AspNetUserLogin](
[LoginProvider] [nvarchar] (100) NOT NULL,
[ProviderKey] [nvarchar] (100) NOT NULL,
[TenantId] [uniqueidentifier] NOT NULL,
[ProviderDisplayName] [nvarchar] (450) NULL,
[UserId] [uniqueidentifier] NOT NULL,
CONSTRAINT [PK_AspNetUserLogin] PRIMARY KEY CLUSTERED
(
[LoginProvider] ASC,
[ProviderKey] ASC,
[TenantId] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
GO

ALTER TABLE [security].[AspNetUserLogin] WITH CHECK ADD CONSTRAINT [FK_AspNetUserLogin_AspNetUser_UserId] FOREIGN KEY([UserId])
REFERENCES [security].[AspNetUser] ([Id])
ON DELETE CASCADE
GO

ALTER TABLE [security].[AspNetUserLogin] CHECK CONSTRAINT [FK_AspNetUserLogin_AspNetUser_UserId]
GO
```

**Data**: The table is empty

**ASP.NET Code (.NET 6 and Microsoft.Data.SqlClient 4.1.0** (we have tried with version 5 too). We started with a default .NET 6 template, added support for controllers and removed authorization, https redirection and exception handler.
The controller:

```cs
using Microsoft.AspNetCore.Identity;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Data.SqlClient;

namespace SyncAsyncPerf.Controllers
{
[Route("api/[controller]")]
[ApiController]
public class LoadTestController : ControllerBase
{
private const string _connectionString =
"Server=tcp:ourserver.database.windows.net;Authentication=Active Directory Managed Identity;Initial Catalog=oursqldb;MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;Max Pool Size=5000;Min Pool Size=100";

[HttpGet]
public IActionResult Get(Guid userId)
{
var sql = $"SELECT [LoginProvider], [ProviderKey], [ProviderDisplayName] FROM [security].[AspNetUserLogin]" +
$"WHERE [UserId] = @UserId";

using var connection = new SqlConnection(_connectionString);
using var command = connection.CreateCommand();

command.CommandText = sql;
var parameter = command.CreateParameter();
parameter.ParameterName = "@UserId";
parameter.DbType = System.Data.DbType.Guid;
parameter.Value = userId;
command.Parameters.Add(parameter);

connection.Open();

using var reader = command.ExecuteReader(System.Data.CommandBehavior.SingleResult | System.Data.CommandBehavior.CloseConnection);

var result = new List();
while (reader.Read())
{
result.Add(
new UserLoginInfo(reader.GetString(0), reader.GetString(1), reader.IsDBNull(2) ? null : reader.GetString(2))
);
}

return Ok(result);
}

[HttpGet("async")]
public async Task GetAsync(Guid userId)
{
var sql = $"SELECT [LoginProvider], [ProviderKey], [ProviderDisplayName] FROM [security].[AspNetUserLogin]" +
$"WHERE [UserId] = @UserId";

using var connection = new SqlConnection(_connectionString);
using var command = connection.CreateCommand();

command.CommandText = sql;
var parameter = command.CreateParameter();
parameter.ParameterName = "@UserId";
parameter.DbType = System.Data.DbType.Guid;
parameter.Value = userId;
command.Parameters.Add(parameter);

await connection.OpenAsync(HttpContext.RequestAborted);

using var reader = await command.ExecuteReaderAsync(System.Data.CommandBehavior.SingleResult | System.Data.CommandBehavior.CloseConnection, HttpContext.RequestAborted);

var result = new List();
while (await reader.ReadAsync())
{
result.Add(
new UserLoginInfo(reader.GetString(0), reader.GetString(1), reader.IsDBNull(2) ? null : reader.GetString(2))
);
}

return Ok(result);
}
}
}
```

Finally, some notes on the connection string. We have tried many different configurations without effecting the underlying problem. If we use defaults for connection pooling the number of errors increases rapidly, especially for async code because of connection timeouts.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.