RemoteExecutor crashes with "`AbandonedMutexException`: The wait completed due to an abandoned mutex."
- Dominant language
- C#
- Stars
- 729
- Forks
- 397
- Avg merge
- 3d 15m
- Merged PRs (30d)
- 149
Description
- [ ] This issue is blocking
- [x] This issue is causing unreasonable pain
Version: **5.0.0-beta.20201.2**
Windows Forms has a number of tests that rely on `RemoteExecutor`. Yesterday all of the sudden we tests started randomly crashing with either ([System.Windows.Forms.Tests_netcoreapp5.0_x64.log](https://dev.azure.com/dnceng/public/_build/results?buildId=610393&view=artifacts&type=publishedArtifacts)):
```
Unhandled exception. System.Threading.AbandonedMutexException: The wait completed due to an abandoned mutex.
at System.Threading.WaitHandle.WaitOneNoCheck(Int32 millisecondsTimeout)
at System.Threading.WaitHandle.WaitOne()
at Coverlet.Core.Instrumentation.Tracker.System.Windows.Forms_4cd81754-6b57-47be-9c9b-f84bb7e27431.UnloadModule(Object , EventArgs )
at System.AppContext.OnProcessExit()
System.Windows.Forms.Tests.ParkingWindowTests.ParkingWindow_DoesNotThrowOnGarbageCollecting [FAIL]
Assert.Equal() Failure
Expected: 0
Actual: -532462766
Stack Trace:
D:\a\1\s\src\System.Windows.Forms\tests\UnitTests\System\Windows\Forms\Application.ParkingWindowTests.cs(37,0): at System.Windows.Forms.Tests.ParkingWindowTests.ParkingWindow_DoesNotThrowOnGarbageCollecting()
```
or ([System.Windows.Forms.Tests_netcoreapp5.0_x64.log](https://dev.azure.com/dnceng/public/_build/results?buildId=611135&view=artifacts&type=publishedArtifacts)):
```
System.Windows.Forms.Tests.ApplicationTests.Application_EnableVisualStyles_InvokeBeforeGettingRenderWithVisualStyles_Success [FAIL]
Half-way through waiting for remote process.
Memory load: 23
Image Name PID Services
========================= ======== ============================================
svchost.exe 916 PlugPlay
svchost.exe 964 BrokerInfrastructure, DcomLaunch, Power,
SystemEventsBroker
svchost.exe 500 RpcEptMapper, RpcSs
svchost.exe 636 LSM
svchost.exe 1164 TermService
svchost.exe 1208 lmhosts
svchost.exe 1216 CoreMessagingRegistrar
svchost.exe 1288 Schedule
svchost.exe 1304 NcbService
svchost.exe 1328 TimeBrokerSvc
svchost.exe 1400 ProfSvc
svchost.exe 1500 EventLog
svchost.exe 1532 UmRdpService
svchost.exe 1552 UserManager
svchost.exe 1620 nsi
svchost.exe 1744 vmicheartbeat
svchost.exe 1752 vmickvpexchange
svchost.exe 1772 Dhcp
svchost.exe 1780 vmicrdv
svchost.exe 1808 vmicshutdown
svchost.exe 1908 vmicvss
svchost.exe 1916 vmictimesync
svchost.exe 1108 CertPropSvc
svchost.exe 2068 EventSystem
svchost.exe 2076 SysMain
svchost.exe 2116 Themes
svchost.exe 2248 LanmanWorkstation
svchost.exe 2292 NlaSvc
svchost.exe 2308 SENS
svchost.exe 2384 AudioEndpointBuilder
svchost.exe 2392 Dnscache
svchost.exe 2420 FontCache
svchost.exe 2428 SessionEnv
svchost.exe 2516 Audiosrv
svchost.exe 2580 netprofm
svchost.exe 2736 DusmSvc
svchost.exe 2748 Wcmsvc
svchost.exe 2828 ShellHWDetection
Unhandled exception. svchost.exe 2900 WinHttpAutoProxySvc
svchost.exe 3008 BFE, mpssvc
svchost.exe 3140 SEMgrSvc
svchost.exe 3224 CryptSvc
svchost.exe 3232 DiagTrack
svchost.exe 3244 DPS
svchost.exe 3280 Winmgmt
svchost.exe 3320 LanmanServer
svchost.exe 3424 TrkWks
svchost.exe 3432 W32Time
svchost.exe 3544 WinRM
svchost.exe 3612 WpnService
svchost.exe 3636 iphlpsvc
svchost.exe 3688 WdiServiceHost
svchost.exe 4368 PolicyAgent
svchost.exe 4640 StateRepository
svchost.exe 5060 CDPUserSvc_4247c
svchost.exe 5104 WpnUserService_4247c
svchost.exe 3080 TokenBroker
svchost.exe 5176 TabletInputService
svchost.exe 5300 CDPSvc
svchost.exe 5564 cbdhsvc_4247c
svchost.exe 6500 LicenseManager
svchost.exe 6648 WarpJITSvc
svchost.exe 7292 pla
svchost.exe 7244 SSDPSRV
svchost.exe 8816 Appinfo
svchost.exe 9020 PcaSvc
svchost.exe 5432 WdiSystemHost
svchost.exe 3600 wuauserv
svchost.exe 7160 DoSvc
svchost.exe 5584 StorSvc
svchost.exe 8172 UsoSvc
svchost.exe 7388 wscsvc
svchost.exe 6720 OneSyncSvc_4247c
svchost.exe 452 NgcSvc
svchost.exe 8440 NgcCtnrSvc
svchost.exe 8524 gpsvc
Timed out at 4/22/2020 1:28:05 AM after 60000ms waiting for remote process.
Wrote mini dump to: F:\h\w\B2D20A03\w\C1B00A1B\uploads\9308.1u2vt2qb.ljc.dmp
Process ID: 9308
Handle: 9888
Name: dotnet
MainModule: F:\workspace\_work\1\s\.dotnet\dotnet.exe
StartTime: 4/22/2020 1:26:52 AM
TotalProcessorTime: 00:01:04.8281250
Stack Trace:
/_/src/Microsoft.DotNet.RemoteExecutor/src/RemoteInvokeHandle.cs(219,0): at Microsoft.DotNet.RemoteExecutor.RemoteInvokeHandle.Dispose(Boolean disposing)
/_/src/Microsoft.DotNet.RemoteExecutor/src/RemoteInvokeHandle.cs(54,0): at Microsoft.DotNet.RemoteExecutor.RemoteInvokeHandle.Dispose()
F:\workspace\_work\1\s\src\System.Windows.Forms\tests\UnitTests\System\Windows\Forms\ApplicationTests.cs(17,0): at System.Windows.Forms.Tests.ApplicationTests.Application_EnableVisualStyles_InvokeBeforeGettingRenderWithVisualStyles_Success()
System.Threading.AbandonedMutexException: The wait completed due to an abandoned mutex.
at System.Threading.WaitHandle.WaitOneNoCheck(Int32 millisecondsTimeout)
at System.Threading.WaitHandle.WaitOne()
at Coverlet.Core.Instrumentation.Tracker.System.Windows.Forms.Primitives_27ec1ab9-8067-4be0-9359-fb23fef04f49.UnloadModule(Object , EventArgs )
at System.AppContext.OnProcessExit()
```
I can only guess that something in .NET runtime has changed that started causing this.
Contributor guide
Assessment
This issue has not been assessed yet.