NetworkChange on Linux: permanent deadlock between CloseSocket() under s_gate and the netlink reader's ProcessEvent callback
- Dominant language
- C#
- Stars
- 18.3k
- Forks
- 5.6k
- PR merge metrics
- PR metrics pending
Description
### Description
On Linux, removing the last `NetworkAddressChanged` / `NetworkAvailabilityChanged` subscriber can deadlock permanently against the netlink reader loop. Two threads form a cycle:
**Unsubscriber** — the remove accessor takes `lock (s_gate)` and, finding both subscriber collections empty, calls `CloseSocket()` → `Socket.Dispose()`. `SafeSocketHandle.CloseAsIs` then spins in `while (!_released) { canceledOperations |= TryUnblockSocket(abortive); sw.SpinOnce(); }` waiting for the last `SafeHandle` reference to be released. **It holds `s_gate` for the entire spin.**
**Netlink reader** — `ReadEventsAsync` is inside `Interop.Sys.ReadEvents(socket.SafeHandle, &ProcessEvent)`, so it *holds* that reference, and the `ProcessEvent` callback's first act is `lock (s_gate)`. The source comments on exactly this reference:
> It's safe to compare raw handle values because ProcessEvents gets called from ReadEvents which holds a reference on the SafeHandle.
Neither side can progress, ever. `TryUnblockSocket` cannot break it — the reader is not blocked in a syscall, it is parked on a managed monitor.
Relevant code (permalinks pinned to `8ffe515`):
- [`CloseSocket()` call under `lock (s_gate)`, `NetworkAddressChanged` remove accessor](https://github.com/dotnet/runtime/blob/8ffe51558dc6bf2e78cbf0c603628ed8620bb0a9/src/libraries/System.Net.NetworkInformation/src/System/Net/NetworkInformation/NetworkAddressChange.Unix.cs#L83)
- [`CloseSocket()` call under `lock (s_gate)`, `NetworkAvailabilityChanged` remove accessor](https://github.com/dotnet/runtime/blob/8ffe51558dc6bf2e78cbf0c603628ed8620bb0a9/src/libraries/System.Net.NetworkInformation/src/System/Net/NetworkInformation/NetworkAddressChange.Unix.cs#L144)
- [`CloseSocket()` → `Socket.Dispose()`](https://github.com/dotnet/runtime/blob/8ffe51558dc6bf2e78cbf0c603628ed8620bb0a9/src/libraries/System.Net.NetworkInformation/src/System/Net/NetworkInformation/NetworkAddressChange.Unix.cs#L177-L182)
- [`ProcessEvent` taking `lock (s_gate)` from inside `ReadEvents`](https://github.com/dotnet/runtime/blob/8ffe51558dc6bf2e78cbf0c603628ed8620bb0a9/src/libraries/System.Net.NetworkInformation/src/System/Net/NetworkInformation/NetworkAddressChange.Unix.cs#L222-L240)
- [`SafeSocketHandle.CloseAsIs` spin](https://github.com/dotnet/runtime/blob/8ffe51558dc6bf2e78cbf0c603628ed8620bb0a9/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SafeSocketHandle.cs#L106-L122)
The current `Socket`-based design dates from #64614.
### Why this is worse than a single hung call
The unsubscribe frequently runs **on the finalizer thread**, via `HttpConnectionPoolManager+NetworkChangeCleanup.Finalize()`. When it deadlocks there, the finalizer thread is gone for the life of the process:
- every `GC.WaitForPendingFinalizers()` anywhere in the process blocks forever;
- no finalizer ever runs again, so `SafeHandle`s, sockets and file handles stop being reclaimed.
In our case 13 of 22 threads ended up parked in `GC.WaitForPendingFinalizers()` behind that one finalizer, and the process stopped doing useful work at all. It presented as an unexplained stall, not as an error.
### Actual behaviour
From `dotnet-dump analyze --command clrthreads` on a heap dump of the stalled process:
```
Thread (finalizer):
System.Private.CoreLib!System.Threading.Thread.Sleep(int32)
System.Private.CoreLib!System.Threading.SpinWait.SpinOnceCore(int32)
System.Net.Sockets!System.Net.Sockets.Socket.Dispose(bool)
System.Net.Sockets!System.Net.Sockets.Socket.Dispose()
System.Net.NetworkInformation!System.Net.NetworkInformation.NetworkChange.CloseSocket()
System.Net.NetworkInformation!System.Net.NetworkInformation.NetworkChange.remove_NetworkAddressChanged(...)
System.Net.Http!System.Net.Http.HttpConnectionPoolManager+NetworkChangeCleanup.Finalize()
System.Private.CoreLib!System.GC.RunFinalizers()
Thread (threadpool):
System.Private.CoreLib!System.Threading.Monitor.Enter_Slowpath(class System.Object)
System.Private.CoreLib!System.Threading.Monitor.Enter(class System.Object,bool&)
System.Net.NetworkInformation!System.Net.NetworkInformation.NetworkChange.ProcessEvent(int,value class NetworkChangeKind)
System.Net.NetworkInformation!Interop+Sys.ReadEvents(class System.Runtime.InteropServices.SafeHandle,fnptr void(int,value class NetworkChangeKind))
System.Net.NetworkInformation!System.Net.NetworkInformation.NetworkChange+d__29.MoveNext()
...
```
Plus 13 further threads blocked in `System.GC.WaitForPendingFinalizers()`.
### Expected behaviour
Unsubscribing the last handler completes, whether or not a netlink event is in flight.
### Reproduction steps
No minimal repro — it is a race, hit in CI rather than constructed. The two conditions that make it reachable:
1. Repeatedly drive the subscriber count to **zero**, since each 1→0 transition is another `CloseSocket()`. Creating and releasing many `SocketsHttpHandler`s does this implicitly, because each pool manager subscribes and unsubscribes from its finalizer.
2. Generate address-change events continuously, so the reader is usually mid-dispatch — e.g. `ip link add dummy0 type dummy` plus `ip addr add` / `ip addr del` in a loop, or heavy container churn (every veth pair is an event).
Our occurrence was an integration test suite: several hundred host boots (so several hundred `SocketsHttpHandler`s created and finalized) while 12 Docker containers were being created and destroyed concurrently.
### Workaround
Hold one `NetworkAddressChanged` subscription for the lifetime of the process. Both `CloseSocket()` call sites are guarded by "both subscriber collections are now empty", so a subscriber that is never removed makes them unreachable and the window never opens.
### Possible direction for a fix
Offered tentatively — this is a reading of the code, not something I have built or tested.
Avoid disposing while holding `s_gate`: capture the socket and null the field under the lock, then dispose outside it. `ProcessEvent` already re-checks `Socket != null && socket == Socket.Handle` under the lock and would no-op, and `ReadEventsAsync` already catches `ObjectDisposedException` and `SocketError.OperationAborted`.
### Configuration
- .NET 10.0.10, linux-x64
- Ubuntu 24.04.1 LTS, x64 VM
### Regression?
Not known to be. The shape appears long-standing and is present on `main`.
Contributor guide
Research direction
Start in System.Net.NetworkInformation/NetworkAddressChange.Unix.cs at the two remove accessors, CloseSocket(), ReadEventsAsync, and ProcessEvent; then read SafeSocketHandle.CloseAsIs in System.Net.Sockets. Exercise the race with repeated subscriber removal and Linux address-change events such as ip link and ip addr churn. Done means removing the last handler completes even while a netlink event is in flight, without leaving finalization blocked.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, linux
- Domain
- networking, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100