dotnet / dotnet/runtime

NetworkChange on Linux: permanent deadlock between CloseSocket() under s_gate and the netlink reader's ProcessEvent callback

Open
#131,935 2 comments 0 reactions 0 assignees View on GitHub
area-System.Net.Sockets
Dominant language
C#
Stars
18.3k
Forks
5.6k
PR merge metrics
PR metrics pending

Description

### Description

On Linux, removing the last `NetworkAddressChanged` / `NetworkAvailabilityChanged` subscriber can deadlock permanently against the netlink reader loop. Two threads form a cycle:

**Unsubscriber** — the remove accessor takes `lock (s_gate)` and, finding both subscriber collections empty, calls `CloseSocket()` → `Socket.Dispose()`. `SafeSocketHandle.CloseAsIs` then spins in `while (!_released) { canceledOperations |= TryUnblockSocket(abortive); sw.SpinOnce(); }` waiting for the last `SafeHandle` reference to be released. **It holds `s_gate` for the entire spin.**

**Netlink reader** — `ReadEventsAsync` is inside `Interop.Sys.ReadEvents(socket.SafeHandle, &ProcessEvent)`, so it *holds* that reference, and the `ProcessEvent` callback's first act is `lock (s_gate)`. The source comments on exactly this reference:

> It's safe to compare raw handle values because ProcessEvents gets called from ReadEvents which holds a reference on the SafeHandle.

Neither side can progress, ever. `TryUnblockSocket` cannot break it — the reader is not blocked in a syscall, it is parked on a managed monitor.

Relevant code (permalinks pinned to `8ffe515`):

- [`CloseSocket()` call under `lock (s_gate)`, `NetworkAddressChanged` remove accessor](https://github.com/dotnet/runtime/blob/8ffe51558dc6bf2e78cbf0c603628ed8620bb0a9/src/libraries/System.Net.NetworkInformation/src/System/Net/NetworkInformation/NetworkAddressChange.Unix.cs#L83)
- [`CloseSocket()` call under `lock (s_gate)`, `NetworkAvailabilityChanged` remove accessor](https://github.com/dotnet/runtime/blob/8ffe51558dc6bf2e78cbf0c603628ed8620bb0a9/src/libraries/System.Net.NetworkInformation/src/System/Net/NetworkInformation/NetworkAddressChange.Unix.cs#L144)
- [`CloseSocket()` → `Socket.Dispose()`](https://github.com/dotnet/runtime/blob/8ffe51558dc6bf2e78cbf0c603628ed8620bb0a9/src/libraries/System.Net.NetworkInformation/src/System/Net/NetworkInformation/NetworkAddressChange.Unix.cs#L177-L182)
- [`ProcessEvent` taking `lock (s_gate)` from inside `ReadEvents`](https://github.com/dotnet/runtime/blob/8ffe51558dc6bf2e78cbf0c603628ed8620bb0a9/src/libraries/System.Net.NetworkInformation/src/System/Net/NetworkInformation/NetworkAddressChange.Unix.cs#L222-L240)
- [`SafeSocketHandle.CloseAsIs` spin](https://github.com/dotnet/runtime/blob/8ffe51558dc6bf2e78cbf0c603628ed8620bb0a9/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SafeSocketHandle.cs#L106-L122)

The current `Socket`-based design dates from #64614.

### Why this is worse than a single hung call

The unsubscribe frequently runs **on the finalizer thread**, via `HttpConnectionPoolManager+NetworkChangeCleanup.Finalize()`. When it deadlocks there, the finalizer thread is gone for the life of the process:

- every `GC.WaitForPendingFinalizers()` anywhere in the process blocks forever;
- no finalizer ever runs again, so `SafeHandle`s, sockets and file handles stop being reclaimed.

In our case 13 of 22 threads ended up parked in `GC.WaitForPendingFinalizers()` behind that one finalizer, and the process stopped doing useful work at all. It presented as an unexplained stall, not as an error.

### Actual behaviour

From `dotnet-dump analyze --command clrthreads` on a heap dump of the stalled process:

```
Thread (finalizer):
System.Private.CoreLib!System.Threading.Thread.Sleep(int32)
System.Private.CoreLib!System.Threading.SpinWait.SpinOnceCore(int32)
System.Net.Sockets!System.Net.Sockets.Socket.Dispose(bool)
System.Net.Sockets!System.Net.Sockets.Socket.Dispose()
System.Net.NetworkInformation!System.Net.NetworkInformation.NetworkChange.CloseSocket()
System.Net.NetworkInformation!System.Net.NetworkInformation.NetworkChange.remove_NetworkAddressChanged(...)
System.Net.Http!System.Net.Http.HttpConnectionPoolManager+NetworkChangeCleanup.Finalize()
System.Private.CoreLib!System.GC.RunFinalizers()

Thread (threadpool):
System.Private.CoreLib!System.Threading.Monitor.Enter_Slowpath(class System.Object)
System.Private.CoreLib!System.Threading.Monitor.Enter(class System.Object,bool&)
System.Net.NetworkInformation!System.Net.NetworkInformation.NetworkChange.ProcessEvent(int,value class NetworkChangeKind)
System.Net.NetworkInformation!Interop+Sys.ReadEvents(class System.Runtime.InteropServices.SafeHandle,fnptr void(int,value class NetworkChangeKind))
System.Net.NetworkInformation!System.Net.NetworkInformation.NetworkChange+d__29.MoveNext()
...
```

Plus 13 further threads blocked in `System.GC.WaitForPendingFinalizers()`.

### Expected behaviour

Unsubscribing the last handler completes, whether or not a netlink event is in flight.

### Reproduction steps

No minimal repro — it is a race, hit in CI rather than constructed. The two conditions that make it reachable:

1. Repeatedly drive the subscriber count to **zero**, since each 1→0 transition is another `CloseSocket()`. Creating and releasing many `SocketsHttpHandler`s does this implicitly, because each pool manager subscribes and unsubscribes from its finalizer.
2. Generate address-change events continuously, so the reader is usually mid-dispatch — e.g. `ip link add dummy0 type dummy` plus `ip addr add` / `ip addr del` in a loop, or heavy container churn (every veth pair is an event).

Our occurrence was an integration test suite: several hundred host boots (so several hundred `SocketsHttpHandler`s created and finalized) while 12 Docker containers were being created and destroyed concurrently.

### Workaround

Hold one `NetworkAddressChanged` subscription for the lifetime of the process. Both `CloseSocket()` call sites are guarded by "both subscriber collections are now empty", so a subscriber that is never removed makes them unreachable and the window never opens.

### Possible direction for a fix

Offered tentatively — this is a reading of the code, not something I have built or tested.

Avoid disposing while holding `s_gate`: capture the socket and null the field under the lock, then dispose outside it. `ProcessEvent` already re-checks `Socket != null && socket == Socket.Handle` under the lock and would no-op, and `ReadEventsAsync` already catches `ObjectDisposedException` and `SocketError.OperationAborted`.

### Configuration

- .NET 10.0.10, linux-x64
- Ubuntu 24.04.1 LTS, x64 VM

### Regression?

Not known to be. The shape appears long-standing and is present on `main`.

Contributor guide

Open the contributing guide

Research direction

Start in System.Net.NetworkInformation/NetworkAddressChange.Unix.cs at the two remove accessors, CloseSocket(), ReadEventsAsync, and ProcessEvent; then read SafeSocketHandle.CloseAsIs in System.Net.Sockets. Exercise the race with repeated subscriber removal and Linux address-change events such as ip link and ip addr churn. Done means removing the last handler completes even while a netlink event is in flight, without leaving finalization blocked.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, linux
Domain
networking, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.