dotnet / dotnet/runtime

PortableThreadPool permanently stops dispatching: single worker idle in `LowLevelLifoSemaphore.Wait` with `NumProcessingWork=1`, global queue non-empty, GateThread starvation check cannot recover (.NET 8.0.27, Linux x64, Server GC, 32 cores)

Open
#128,879 3 comments 0 reactions 0 assignees View on GitHub
area-System.Threading needs-author-action
Dominant language
C#
Stars
18.3k
Forks
5.6k
PR merge metrics
PR metrics pending

Description

### Description

A long-running Orleans 8 service on .NET 8.0.27 (Linux x64, Server GC, 32 cores) hit a state where PortableThreadPool permanently stopped dispatching queued work. The single worker thread sits in LowLevelLifoSemaphore.Wait, the global queue holds
ready items, and the pool never recovers (~28h until killed). Internal counters are desynced: ThreadCounts.NumProcessingWork == 1 while that worker is provably asleep and no semaphore permit is outstanding. The GateThread starvation heuristic is alive
but structurally cannot recover this shape. Appears to be the same family as #117877 and #99896.

### Reproduction Steps

Natural occurrence: intermittent, on a long-running silo under high churn of short work items (per-second grain timers, ~12k active grains) on a 32-core box. The originating interleaving is a sub-microsecond race and we have no deterministic natural
repro (neither do #117877 / #99896).

Forced-state demonstration (reproduces the consequence / non-recovery, not the natural trigger). On .NET 8.0.27, a console app uses reflection to set the exact counter values observed in the dump, without releasing the semaphore:

1. Warm up the pool (queue a few items, let workers park in semaphore.Wait).
2. Reflectively set:
- ThreadPoolWorkQueue._separated.hasOutstandingThreadRequest = 1
- PortableThreadPool._separated.numRequestedWorkers = 1
- PortableThreadPool._separated.counts._data = 429496795137 (NumProcessingWork=1, NumExistingThreads=1, NumThreadsGoal=100)
3. ThreadPool.QueueUserWorkItem(...) x 8.

Observed: 0/8 run over 8s, all workers idle, GateThread alive but never rescues. Clearing the flag + one EnsureThreadRequested drains everything instantly. Full ~120-line repro available on request.

### Expected behavior

With work items in the global queue and idle/parked worker threads, the pool should dispatch them — and even if internal counters transiently desync, the pool (e.g. via GateThread) should eventually detect "queue non-empty + numRequestedWorkers > 0 +
no dispatch progress for a long time" and recover. It should not hang permanently.

### Actual behavior

The pool stops dispatching forever. Direct evidence from two independent full dumps taken ~28h apart, byte-identical in every field below:

- Workers Total=1 / Running=1 / Idle=0, CPU ~1%; the single worker parked in LowLevelLifoSemaphore.Wait.
- ThreadCounts: NumProcessingWork=1, NumExistingThreads=1, NumThreadsGoal=100; numRequestedWorkers=1; gateThreadRunningState=6.
- Global queue (threadpool -wi): 35 ready items never drained — 32 x System.Threading.TimerQueue, 1 x SocketAsyncEngine, 1 x Task, 1 x Kestrel HttpProtocol+.
- dumpasync --stats: tens of thousands of stalled state machines (~12k per-activation message-loop pairs) waiting on the stuck pool.

Why no recovery: (1) EnsureThreadRequested is gated by the 0/1 hasOutstandingThreadRequest flag (stuck at 1 → no new request); (2) GateThread starvation rescue only fires while NumProcessingWork >= NumThreadsGoal, but 1 >= 100 is false, so it loops
every 500ms and never adds a worker; (3) RemoveWorkingWorker never runs (no worker in WorkerDoWork) so NumProcessingWork is never corrected.

### Regression?

Not identified as a regression from a specific version. The wake path in PortableThreadPool.WorkerThread.cs is byte-identical between release/8.0 and release/10.0 (only an unrelated ThreadsToKeepAlive feature and an ARM spin-count constant differ), so
.NET 10 is affected too. main rewrites this area (Saturated state / _hasOutstandingThreadRequest, via #122726).

### Known Workarounds

None internal to the runtime (the deadlock is not self-healing). Only an external watchdog that detects the hang and restarts the process. Reducing the rate of short queued work items lowers the hit probability but does not eliminate it.

### Configuration

- .NET 8.0.27
- Linux x64 (kernel 5.14)
- Server GC enabled
- 32 logical processors
- Microsoft Orleans 8.0.0
- ThreadPool.SetMinThreads(100, 100) set at startup (NumThreadsGoal floored at 100)
- Not specific to this configuration as far as we can tell, but high core count + Server GC + high churn appear to raise the probability.

### Other information

Likely same root cause as #117877 (.NET 9, same LowLevelLifoSemaphore.Wait stack) and #99896 (.NET 8). New data here: two byte-identical dumps proving permanence, decoded counter values, and the concrete reason GateThread cannot recover.

Questions: (1) Is this fixed by the #122726 rewrite on main? (2) If so, any backport to 8.x / 10.x? (3) Independent of the originating race, should GateThread/a watchdog recover the "queue non-empty + numRequestedWorkers>0 + no progress +
NumProcessingWork < NumThreadsGoal" shape, which is currently a permanent silent hang?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.