HttpSys AsyncAcceptContext can repeatedly fail after stale NativeOverlapped is freed
- Dominant language
- C#
- Stars
- 38.4k
- Forks
- 10.9k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 276
Description
### Is there an existing issue for this?
- [x] I have searched the existing issues
### Describe the bug
This appears related to:
* dotnet/aspnetcore#46439
* dotnet/aspnetcore#46504
I am reporting a suspected stale-overlapped failure mechanism in ASP.NET Core HttpSys.
This is not a full HTTP.sys server repro. The relevant ASP.NET Core type, `AsyncAcceptContext`, is internal and depends on a real HTTP.sys request queue. Instead, I created a minimal Windows-only .NET console repro that models the low-level field pattern used by `AsyncAcceptContext`:
* a reusable `_overlapped` field,
* `ThreadPoolBoundHandle.FreeNativeOverlapped(_overlapped)`,
* reallocation into the same field,
* failure or re-entry after the native overlapped has already been freed,
* retry behavior that repeatedly hits the same stale field.
Minimal repro repository:
https://github.com/sjovanovic974/HttpSysOverlappedRepro
The repro demonstrates that once `_overlapped` points to an already-freed native overlapped, repeated calls to `FreeNativeOverlapped` produce the same `ArgumentException` shape:
```text
ArgumentException: 'overlapped' was not allocated by this ThreadPoolBoundHandle instance. (Parameter 'overlapped')
```
The repro has three sections:
1. Deterministic stale `_overlapped` double-free.
2. Concurrent free/reallocation model.
3. Tight retry loop after `_overlapped` has become stale.
The most important parts are sections 1 and 3.
Section 2 is intentionally only a model of a possible concurrent interaction. I do not want to overstate it as proof of the exact production interleaving.
Steps to reproduce on Windows:
```powershell
git clone https://github.com/sjovanovic974/HttpSysOverlappedRepro.git
cd HttpSysOverlappedRepro
dotnet clean -c Release
dotnet build -c Release
dotnet run -c Release
```
Example run output:
```text
HttpSys AsyncAcceptContext stale-overlapped mechanism model
https://github.com/dotnet/aspnetcore/issues/46439
=== 1. Deterministic: stale _overlapped freed twice ===
[1] AllocateNativeRequest() -- _overlapped allocated
[2] Simulated failure between free and realloc
_overlapped is now stale: freed but not nulled
[3] ArgumentException: 'overlapped' was not allocated by this ThreadPoolBoundHandle instance. (Parameter 'overlapped')
REPRODUCED: double-free of stale _overlapped
=== 2. Concurrent: two threads race free/realloc ===
(models accept path vs. stale IOCP callback)
accept-path: ArgumentException: 'overlapped' was not allocated by this ThreadPoolBoundHandle instance. (Parameter 'overlapped')
Note: one thread did not complete before timeout; this can happen because the demo forces a barrier while the other thread may throw during the duplicate free.
REPRODUCED: concurrent free/realloc can corrupt _overlapped in this model
With PreAllocatedOverlapped(state: this), IOWaitCallback cannot
distinguish stale callbacks -- no per-operation identity exists.
=== 3. Spin loop: tight retry loop on one core ===
(models AcceptLoop.ExecuteAsync catch+continue after corruption)
_overlapped corrupted -- running accept loop for 2 seconds...
1s: 246,419 exceptions so far
521,362 ArgumentExceptions in 2.00s (260,679/sec)
Loop never recovers: _overlapped is never nulled after the failed free.
Operational risk: the accept loop can consume one core until the affected process is restarted.
The fix (PR #46504, never merged): set _overlapped = null before freeing.
Fuller investigation should consider per-operation overlapped identity
so stale IOCP callbacks can be detected or discarded.
```
### Expected Behavior
`AsyncAcceptContext` should not remain in a state where `_overlapped` points to a native overlapped that has already been freed.
At minimum, after a native overlapped is selected for freeing, the field should not continue to reference that same pointer if anything fails or re-enters before the next successful allocation.
A defensive pattern would be similar to the previously proposed PR:
```csharp
var overlapped = _overlapped;
if (overlapped != null)
{
_overlapped = null;
boundHandle.FreeNativeOverlapped(overlapped);
}
```
This would not necessarily solve every possible race, but it would prevent a stale field from being repeatedly freed after the first failure.
In production, this exception can be catastrophic if it occurs inside the HttpSys accept loop and the loop catches/logs/continues. Once the stale `_overlapped` state exists, retries can repeatedly hit the same exception and consume CPU until the affected process is restarted.
In our production case, the failure manifested as a self-sustaining error storm in one affected ASP.NET Core HttpSys-hosted service. The service kept logging the same accept-loop exception repeatedly instead of recovering, which caused extremely rapid log growth, high CPU usage on the affected process, and operational degradation until the service process was restarted. In other words, the impact was not a single failed request, but a persistent failure mode where the accept loop continued retrying against corrupted/stale accept state.
### Steps To Reproduce
Minimal repro repository:
https://github.com/sjovanovic974/HttpSysOverlappedRepro
On Windows:
```powershell
git clone https://github.com/sjovanovic974/HttpSysOverlappedRepro.git
cd HttpSysOverlappedRepro
dotnet clean -c Release
dotnet build -c Release
dotnet run -c Release
```
The project is intentionally small and contains only:
```text
.gitignore
HttpSysOverlappedRepro.csproj
Program.cs
README.md
```
It is a low-level model repro, not a full HTTP.sys server repro, because `AsyncAcceptContext` is internal and depends on a real HTTP.sys request queue.
### Exceptions (if any)
The repro produces the same exception type and message shape as the previously reported HttpSys failure:
```text
ArgumentException: 'overlapped' was not allocated by this ThreadPoolBoundHandle instance. (Parameter 'overlapped')
```
Relevant stack shape from the prior ASP.NET Core issue:
```text
System.ArgumentException: 'overlapped' was not allocated by this ThreadPoolBoundHandle instance. (Parameter 'overlapped')
at System.Threading.ThreadPoolBoundHandle.FreeNativeOverlapped(NativeOverlapped* overlapped)
at Microsoft.AspNetCore.Server.HttpSys.AsyncAcceptContext.AllocateNativeRequest(...)
```
Example output from the repro:
```text
=== 1. Deterministic: stale _overlapped freed twice ===
[1] AllocateNativeRequest() -- _overlapped allocated
[2] Simulated failure between free and realloc
_overlapped is now stale: freed but not nulled
[3] ArgumentException: 'overlapped' was not allocated by this ThreadPoolBoundHandle instance. (Parameter 'overlapped')
REPRODUCED: double-free of stale _overlapped
=== 3. Spin loop: tight retry loop on one core ===
(models AcceptLoop.ExecuteAsync catch+continue after corruption)
_overlapped corrupted -- running accept loop for 2 seconds...
1s: 246,419 exceptions so far
521,362 ArgumentExceptions in 2.00s (260,679/sec)
Loop never recovers: _overlapped is never nulled after the failed free.
```
### .NET Version
8.0.413
### Anything else?
Related prior issue and PR:
* dotnet/aspnetcore#46439
* dotnet/aspnetcore#46504
Important scope clarification:
This repro does not directly instantiate `AsyncAcceptContext`. It is an isolated low-level model of the suspected mechanism because:
* `AsyncAcceptContext` is internal,
* it depends on a real HTTP.sys request queue,
* the original production failure is timing-sensitive.
The repro should be read as evidence that the stale-overlapped explanation is mechanically valid, not as proof of the exact production interleaving.
Environment used for the repro:
```text
.NET SDK:
Version: 8.0.413
MSBuild version: 17.11.38+901dc04e4
Runtime Environment:
OS Name: Windows
OS Version: 10.0.26200
OS Platform: Windows
RID: win-x64
Host:
Version: 8.0.19
Architecture: x64
.NET runtimes installed:
Microsoft.AspNetCore.App 8.0.19
Microsoft.NETCore.App 8.0.19
Microsoft.WindowsDesktop.App 8.0.19
```
IDE / editor:
JetBrains Rider 2026.1.3
Contributor guide
Research direction
Start by running the Windows repro and then inspect AsyncAcceptContext, especially AllocateNativeRequest and the AcceptLoop.ExecuteAsync retry path. Compare the behavior with related issue #46439 and proposed PR #46504. Done means the accept context cannot retain a freed native overlapped and the retry path no longer repeatedly produces the reported ArgumentException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100