dotnet / dotnet/runtime

Bounded channel under CPU contention: NullReferenceException from WriteAsync, InvalidOperationException from the reader, or parked reader never wakes (.NET 10/11; .NET 9 unaffected)

Open
#131,141 2 comments 1 reaction 0 assignees View on GitHub
area-System.Threading.Channels regression-from-last-release
Dominant language
C#
Stars
18.3k
Forks
5.6k
PR merge metrics
PR metrics pending

Description

### Description

We run MassTransit with the Amazon SQS transport. Since moving to .NET 10, SQS receive endpoints occasionally stop polling while the process and the bus keep running, usually with nothing in the logs (MassTransit/MassTransit#6226). Investigating that symptom led to `System.Threading.Channels`: the transport funnels SQS client calls through a bounded channel with one waiting reader and many writers, and under CPU contention that channel intermittently fails.

Stress rig that reproduces it: **https://github.com/PooryaCareExpert/repro** — many independent "batchers" (a `bounded(100)` channel with `SingleReader = true`, `FullMode.Wait`, `AllowSynchronousContinuations = false`; one parked reader plus 10 paced writers, mirroring MassTransit's SQS `Batcher`) in CPU-constrained Linux containers.

### Reproduction Steps

```sh
git clone https://github.com/PooryaCareExpert/repro
cd repro
docker compose --profile 9 --profile 10 --profile 11 up --build -d --scale stress-9-0=5 --scale stress-10-0=5 --scale stress-11-0=5
docker compose --profile 9 --profile 10 --profile 11 logs -f
```

Hits are probabilistic; with 12–16 instances sharing a 14-core `cpuset` the first hit typically lands within the hour. Knobs are in the repo README; captured runs are in RESULTS.md.

### Actual behavior

Three failure shapes, all on the same bounded channel:

**1. Writer side** — `WriteAsync` throws:

```
System.NullReferenceException: Object reference not set to an instance of an object.
at System.Threading.Channels.ChannelUtilities.DangerousSetOperations[TAsyncOp,TResult](TAsyncOp head, TResult result)
at System.Threading.Channels.BoundedChannel`1.BoundedChannelWriter.WriteAsync(T item, CancellationToken cancellationToken)
at StressRepro.Batcher.Produce()
```

**2. Reader side** — the single reader's `WaitToReadAsync` await throws, killing the consume loop (in MassTransit this kills the endpoint):

```
System.InvalidOperationException: The asynchronous operation has not completed.
at System.Threading.Channels.AsyncOperation.ThrowIncompleteOperationException()
at System.Threading.Channels.AsyncOperation`2.GetResult(Int16 token)
at StressRepro.Batcher.Consume()
```

**3. Silent** — the parked reader is never woken and nothing is thrown: consumption on that channel stops permanently while writers park forever. This matches the MassTransit#6226 reports where an endpoint dies with no log line at all.

On 11.0.0-preview.6 the reader has additionally thrown `OperationCanceledException` carrying a cancellation token the application never created.

Shapes 2/3 usually land on a channel that had already produced a shape-1 writer exception (10 of 11 same-channel pairings across two runs).

### Results

25 container instances (details in the repo's RESULTS.md):

| Runtime | Writer NREs | Reader kills / strands |
|---|---|---|
| 9.0.18 | 0 | 0 — control, clean on all instances |
| 10.0.10 | 28 | 1 (shape 2) |
| 11.0.0-preview.6 | 50 | 10 (shape 3) |

### Expected behavior

No exceptions out of `WriteAsync` / `WaitToReadAsync`, and a parked reader always wakes when items are written.

### Regression?

The identical rig built for `net9.0` runs clean in every control run; .NET 10.0.10 and 11.0.0-preview.6.26359.118 both hit.

### Known Workarounds

Targeting `net9.0`.

### Configuration

- .NET 10.0.10 and 11.0.0-preview.6.26359.118 (affected); 9.0.18 (clean)
- linux/amd64 containers (Docker Desktop on Windows 11, x64 host), 8-CPU quota, shared `cpuset`
- Also observed in production: x64 AWS Fargate (1 vCPU), .NET 10, MassTransit Amazon SQS

### Other information

In production the writer-side `NullReferenceException` appears roughly every other day across 5 environments, always the same `DangerousSetOperations` → `BoundedChannelWriter.WriteAsync` stack (reached via MassTransit's `DeleteMessage` batching; each hit means a failed SQS delete and a duplicate delivery). Reader-side endpoint deaths are rarer and match the stopped-endpoint reports in MassTransit/MassTransit#6226.

Contributor guide

Open the contributing guide

Research direction

Start with the linked stress repro, its README and RESULTS.md, then inspect System.Threading.Channels entry points named in the traces: BoundedChannelWriter.WriteAsync, WaitToReadAsync, AsyncOperation, and DangerousSetOperations. Run the Docker Compose profiles under CPU contention on .NET 10 and 11, using .NET 9 as the clean control. Done means no writer or reader exceptions and no parked reader that fails to wake.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, docker, linux
Domain
backend, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.