dotnet / dotnet/SqlClient

Managed SNI + MARS: unobserved NullReferenceException in SniTcpHandle.ReceiveAsync when the physical connection is disposed while the demuxer is still receiving (reliable repro)

Open
#4,679 1 comment 0 reactions 1 assignee Claimed by @cheenamalhotra View on GitHub
Repro Available :heavy_check_mark:
Dominant language
C#
Stars
989
Forks
340
Avg merge
4d 18h
Merged PRs (30d)
69

Description

### Describe the bug

On Linux (managed SNI) with `MultipleActiveResultSets=true`, tearing down the physical connection while the SMUX demuxer is still receiving data for another session produces a `NullReferenceException` inside the SNI receive pump. It escapes into the fire-and-forget `ContinueWith` continuation created by `SniPacket.ReadFromStreamAsync`, so it surfaces as `TaskScheduler.UnobservedTaskException` on the finalizer thread, with no application frames. We see it several times a week in production (ASP.NET Core on .NET 10, Ubuntu 24.04 containers); it is harmless but noisy and impossible to correlate to a caller.

This is the same signature as #1171 (2021) and #2026 (2024, closed not-planned) and is closely related to #3720 (open). The difference here is a small, reliable reproduction.

```
Exception message: Object reference not set to an instance of an object.

Stack trace (as reported in production, .NET 10.0.11, Microsoft.Data.SqlClient 7.0.2):
System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread.
---> System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.Data.SqlClient.ManagedSni.SniTcpHandle.ReceiveAsync(SniPacket& packet) SniTcpHandle.netcore.cs:1025
at Microsoft.Data.SqlClient.ManagedSni.SniMarsConnection.ReceiveAsync(SniPacket& packet) SniMarsConnection.netcore.cs:131
at Microsoft.Data.SqlClient.ManagedSni.SniMarsConnection.HandleReceiveComplete(SniPacket packet, UInt32 sniErrorCode) SniMarsConnection.netcore.cs:346
at Microsoft.Data.SqlClient.ManagedSni.SniPacket.ReadFromStreamAsyncContinuation(Task`1 task, Object state) SniPacket.netcore.cs:302
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)

Stack trace from the repro when the JIT does not inline (shows the exact throw site):
at Microsoft.Data.SqlClient.ManagedSni.SniPacket.ReadFromStreamAsync(Stream stream)
at Microsoft.Data.SqlClient.ManagedSni.SniTcpHandle.ReceiveAsync(SniPacket& packet)
at Microsoft.Data.SqlClient.ManagedSni.SniMarsConnection.ReceiveAsync(SniPacket& packet)
at Microsoft.Data.SqlClient.ManagedSni.SniMarsConnection.HandleReceiveComplete(SniPacket packet, UInt32 sniErrorCode)
at Microsoft.Data.SqlClient.ManagedSni.SniPacket.InvokeAsyncIOCompletionCallback(UInt32 sniErrorCode)
at Microsoft.Data.SqlClient.ManagedSni.SniPacket.ReadFromStreamAsyncContinuation(Task`1 task, Object state)
```

**Mechanism** (v7.0.2, `src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/ManagedSni/`):

1. `SniTcpHandle.Dispose()` sets `_stream = null` under `lock(this)`.
2. `SniMarsConnection.HandleReceiveComplete` runs under `DemuxerSync` (a different lock). After a **successful** receive with `packet.DataLeft == 0` it re-arms the read (`ReceiveAsync(ref packet)`, line 346) -> `SniTcpHandle.ReceiveAsync` -> `packet.ReadFromStreamAsync(_stream)` (line 1025).
3. If the handle was disposed in between, `_stream` is null, so `Stream.ReadAsync` is invoked on a null reference. The `catch` at that site only filters `ObjectDisposedException | SocketException | IOException`, so the `NullReferenceException` propagates.
4. It unwinds into `ReadFromStreamAsyncContinuation`, i.e. the `ContinueWith` task created in `SniPacket.ReadFromStreamAsync`, which nothing observes.

There is no disposed check in `SniTcpHandle.ReceiveAsync`, and nothing serialises `Dispose` against the demuxer re-arm.

### To reproduce

Complete project: https://github.com/billpeet/SqlClientMarsReceiveNreRepro (single `Program.cs`, `Repro.csproj`, README with results). Point it at a throwaway SQL Server; it raises severity-20 errors and needs a login allowed to do so.

```
export SQLCLIENT_REPRO_CONNECTION="Server=localhost,1433;User Id=sa;Password=...;Encrypt=False;TrustServerCertificate=True"
dotnet run -c Release -- 300 mars pooled # exit code 1 when reproduced
dotnet run -c Release -- 300 no-mars # control: never reproduces
```

Core of the loop:

```c#
var conn = new SqlConnection(cs); // MultipleActiveResultSets=true
await conn.OpenAsync();

// Session 1: stream a large result set on a background task.
var cmd = new SqlCommand("SELECT TOP (200000) a.object_id, b.name, REPLICATE('x', 200) AS pad FROM sys.all_objects a CROSS JOIN sys.all_columns b", conn);
var reader = await cmd.ExecuteReaderAsync();
var consumer = Task.Run(async () => { while (await reader.ReadAsync()) { } });

await Task.Delay(rng.Next(1, 30));

// Session 2 on the same physical connection: severity 20 makes the server drop the connection,
// so the client dooms it and disposes the SniTcpHandle while session 1's packets are still being demuxed.
using (var boom = new SqlCommand("RAISERROR('boom', 20, 1) WITH LOG", conn))
{
try { await boom.ExecuteNonQueryAsync(); } catch (SqlException) { }
}

try { await consumer; } catch (Exception ex) when (ex is SqlException or InvalidOperationException or ObjectDisposedException or IOException) { }
try { conn.Dispose(); } catch (Exception ex) when (ex is SqlException or InvalidOperationException) { }
```

A `TaskScheduler.UnobservedTaskException` handler counts NREs whose trace contains `SniTcpHandle.ReceiveAsync` and `SniMarsConnection.HandleReceiveComplete`; `GC.Collect()` / `GC.WaitForPendingFinalizers()` every 25 iterations flushes them.

Observed:

| Platform | MARS | Pooling | Result |
|---|---|---|---|
| Windows 11, .NET 10.0.11, `UseManagedNetworkingOnWindows=true` | on | off | 6, 16 and 8 hits per 200 iterations (three runs) |
| Windows 11, .NET 10.0.11, `UseManagedNetworkingOnWindows=true` | on | on | 11 hits per 200 |
| Windows 11, .NET 10.0.11, `UseManagedNetworkingOnWindows=true` | off | off | 0 hits per 200 |
| Ubuntu 24.04 (`mcr.microsoft.com/dotnet/sdk:10.0`), .NET 10.0.12 | on | off | first hit within 125 iterations |
| Ubuntu 24.04 (`mcr.microsoft.com/dotnet/sdk:10.0`), .NET 10.0.12 | on | on | first hit within 50 iterations |

Controls that do **not** trigger it (0/200 each): a severity-16 error on session 2; a `CommandTimeout` on session 2. Only a connection-terminating teardown while another session is still streaming does. Disposing while the reader is idle also does not trigger it, because SMUX flow control stops the server sending and the demuxer takes the clean error path.

### Expected behavior

No exception, or at most an observed/handled error: `SniTcpHandle.ReceiveAsync` should treat a disposed handle (`_stream == null`) the same way it treats `ObjectDisposedException` and report `SNI_ERROR` via `ReportErrorAndReleasePacket`, so the demuxer takes the `HandleReceiveError` path instead of faulting an unobserved task.

### Further technical details

Microsoft.Data.SqlClient version: 7.0.2 (`7.0.2+8c70cec98444338ddb0b97be94c34fde93970241`); the relevant SNI code is unchanged in 7.0.3 and 7.1.0-preview3 / `main`
.NET target: net10.0 (.NET 10.0.11 in production, 10.0.12 in the Linux repro)
SQL Server version: SQL Server 2025 (RTM-CU5) 17.0.4045 in `mcr.microsoft.com/mssql/server:latest` (repro); on-prem SQL Server (production)
Operating system: Ubuntu 24.04 Docker container (production and repro); also reproduces on Windows 11 with the managed SNI switch

**Additional context**

Production events are reported by the Sentry .NET SDK with `mechanism: UnobservedTaskException`, finalizer thread, no request context. Connection string has `MultipleActiveResultSets=true;Encrypt=False`, pooling on, EF Core 10 + Dapper. Related: #1171, #2026, #3720, #4374, PR #4460 (repro test for the unobserved-callback mechanism).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.