SDK causes occasional .NET crash on WSManCloseCommand

Prerequisites

• Write a descriptive title.
• Make sure you are able to repro it on the latest released version
• Search the existing issues.
• Refer to the FAQ.
• Refer to Differences between Windows PowerShell 5.1 and PowerShell.

Steps to reproduce

We have a .Net Windows Service that uses the Powershell SDK to connect to many remote machines and maintain connections for long periods of time.

We experience in some environments a crash every couple of days, and as such, the exact steps to reproduce are not exactly clear, but there is a strong lead to the exact issue in the code, to be described below.

It is not possible for us at this point to use the latest major SDK version, as the required .Net version is not yet supported in the production environment, and it does not reproduce in our smaller test environments. However, we will show below that the suspected issue in the codebase exists on the latest version as well.

The dump stack trace:

CoreCLR Version: 8.0.1325.6609
.NET Version: 8.0.13
Description: The process was terminated due to an unhandled exception.
Exception Info: System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
Stack:
   at System.Management.Automation.Remoting.Client.WSManNativeApi.WSManCloseCommand(IntPtr, Int32, IntPtr)
   at System.Management.Automation.Remoting.Client.WSManNativeApi.WSManCloseCommand(IntPtr, Int32, IntPtr)
   at System.Management.Automation.Remoting.Client.WSManClientCommandTransportManager.CloseAsync()
   at System.Management.Automation.Runspaces.Internal.ClientRemotePowerShell.HandleInvocationStateInfoReceived(System.Object, System.Management.Automation.RemoteDataEventArgs`1<System.Management.Automation.PSInvocationStateInfo>)
   at System.Management.Automation.Remoting.Client.WSManClientCommandTransportManager.RaiseErrorHandler(System.Management.Automation.Remoting.TransportErrorOccuredEventArgs)
   at System.Management.Automation.Remoting.Client.BaseClientTransportManager.ServicePendingCallbacks(System.Object)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()

Code analysis -
We have a strong suspicion that the crash is caused by a race condition in WSManClientCommandTransportManager between Dispose and CloseAsync methods:
WSManClientCommandTransportManager.CloseAsync may be called internally by the SDK when the runspace pool state becomes broken, for example. Before invoking WSManCloseCommand, it initializes the _closeCmdCompleted member.

However, calling RunspacePool.Dispose from the application code ends up in WSManClientCommandTransportManager.Dispose, which disposes the _closeCmdCompleted. This means that the WSManCloseCommand, which may hang for a long time, would end up invoking a callback that has already been freed, resulting in access violation.

Expected behavior

Invoking WSManCloseCommand does not result in .Net crash

Actual behavior

Invoking WSManCloseCommand may result in .Net crash

Error details

Environment data

Powershell SDK 7.4.10 on .NET 8

Visuals

No response

Start by reading WSManClientCommandTransportManager.CloseAsync and Dispose, focusing on their handling of _closeCmdCompleted and the WSManCloseCommand callback. Use the provided stack trace and race-condition description to investigate concurrent runspace-pool shutdown. Done means invoking WSManCloseCommand during disposal no longer produces the reported .NET access violation.