[Problem/Bug] Browser process hangs for up to 10 minutes when login.live.com is blackholed

What happened?

When a Windows install has login.live.com blackholed (meaning packets are dropped by a firewall and not rejected), WebView2 browser processes generally do not exit for up to 10 minutes. We've (painfully) traced the reason for this end-to-end; TL;DR: it's blocking on the Microsoft Account Sign-in Assistant service (aka wlidsvc) which is in turn trying to determine the device identity.

What specifically happens in our application, which embeds a WebView2, is that some HWNDs were still open in the browser process (e.g. the IME window), which prevented standard cross-process message loop processing since the browser process was already in its shutdown. We have confirmed that all CoreWebView2, CoreWebView2Controller and CoreWebView2Environment instances on our side were shut down already.

Furthermore, wlidsvc ostensibly stalls completely while it's waiting (for up to 10 minutes) on a device identity request. It can't even be shut down via SCM.

We think what happened is an interplay of a few small bugs:

• msedgewebview2.exe's OneAuthEdgeAuthenticationManager should, upon shutdown, cancel outgoing calls (IAsyncInfo::Cancel or such)
• msedgewebview2.exe should tear down window handles it owns before shutting down the rest, in case shutdown ever stalls for another reason.
• wlidsvcshould probably not completely lock up when it can't phone home to provision an device ID. I realize this is not your team, but maybe you can pass it on.
• Windows_Security_Authentication_OnlineId should probably support cancelling the RPC call as well if it's being cancelled (via RpcAsyncCancelCall or such, presumably). Also not your team, I know.

The combination of these led to a few weeks of fairly frustrating debugging on a very locked down enterprise machine - however, we also now know unreasonably more about the interplay than we should. I've attached a stripped down analysis below, but please feel free to ask anything. I can't attach customer dumps for privacy reasons but I can extract whatever is needed for further analysis, though I think there's enough information to act.

Importance

Hard to gauge. We mitigated it customer-side for now by disabling wlidsvc completely; rerouting login.live.com via DNS works also. I would assume any application that embeds WebView2 and is run in a corporate setting could be affected by the hang.

Runtime Channel

Stable

Runtime Version

151.0.4129.101

SDK Version

1.0.3650.58

Framework

Win32 (C++ / MFC)

Operating System

Windows Server 2022

OS Version

10.0.20348

Repro steps and sample app

Note: We haven't reproduced this in lab conditions yet, but our analysis team is working on it. This repro is how we assume it can be reproduced; I'll edit when I hear back from our analysis team.

How to (presumably) repro

1. Use a machine that has no cached MSA device identity (wlidsvc has never successfully provisioned one).
2. Block outbound TCP 443 to login.live.com with a DROP rule (not a REJECT rule, or the connect fails immediately and the bug does not reproduce). Note that login.live.com still needs to be resolvable via DNS.
3. Launch any WebView2 host, create a CoreWebView2Controller, navigate anywhere.
4. Close the controller and release the environment so the browser process should exit.
5. Watch msedgewebview2.exe in Task Manager.

Expected behavior

The browser process exits promptly. Shutdown does not wait on background identity
work.

Actual behavior

The browser process persists for up to ~10 minutes with 0% CPU. It may still have hidden windows remain enumerable on the desktop that are unresponsive, because the message loop has already been destroyed. The process then exits normally on its own once the 600s MSA timeout expires. The wlidsvc also does not respond to shutdown requests during that time.

Regression

Unknown / not verified against an earlier runtime.

Last working version

n/a

Analysis

I'll present the evidence (narratively) from three sides:

• Our app (WG)
• msedgewebview2.exe
• wlidsvc inside svchost.exe

Our App (WG)

This is where it started. A big corporate customer was testing the new version of our app, and reported that after logging in, WG would often (but not always) hang for anywhere between about 2 and 7 minutes completely, before suddenly resuming normally as if nothing had happened. For context, in this particular site configuration, an embedded WebView2 "Hello"-window is shown before logging in, and the actual login dialog is a separate window embedding a WebView2 instance.

We received several dump files from the running WG process as well as a separate executable associated with WG (which also hosts UI components embedded into WG), but not of anything else. The dump files showed seemingly random Win32-API calls hanging for a long time; sometimes SendMessage(), sometimes GetMessage(), and in one case WG itself was not even hanging but progressing through a ListCtrl update incredibly slowly (could be e.g. something else persistently running into the timeout scenario of a SendMessageTimeout in a message handler, but that's pure speculation).

We unfortunately did not get dumps for anything else, but we did get the notice that Task Manager's Wait Chain Analysis showed WG waiting on msedgewebview2.exe. So, we asked for more information.

msedgewebview2.exe

The customer, at a later point in time, reproduced again and this time gave us dumps not just for WG but also for the offending browser process. Unfortunately only for one reproduction, but that was enough.

WG was hanging on an innocent WM_FLOATSTATUS on activation, sent out by MFC and not even WG code itself. The window it was trying to send to was msctf::UI, itself owned by an IME window which had WG's main window set as its owner. (Figuring that out from the dump was incredibly painful and it would take me pages upon pages to explain how - please just trust me on this and don't make me revisit the horror)

But let's look at the browser process. The main thread:

00 ntdll!NtWaitForSingleObject+0x14
01 KERNELBASE!WaitForSingleObjectEx+0x8e
02 oneauth+0x5cd33
03 oneauth!OneAuthUninitialize+0x3578
04 oneauth!OneAuthUninitialize+0x2d89
05 oneauth!OneAuthUninitialize+0x9c6
06 oneauth!OneAuthUninitialize+0x839
07 oneauth!OneAuthUninitialize+0x4ec
08 oneauth!OneAuthUninitialize+0x3d6
09 oneauth!OneAuthUninitialize+0x82
0a oneauth!OneAuthUninitialize+0x25
0b msedge!OneAuthProxy::`vector deleting destructor'+0x38
0c msedge!OneAuthEdgeAuthenticationManager::~OneAuthEdgeAuthenticationManager+0x93
0d msedge!base::Singleton<OneAuthEdgeAuthenticationManager,OneAuthEdgeAuthenticationManager::SingletonTraits,OneAuthEdgeAuthenticationManager>::OnExit+0x19
0e msedge!base::AtExitManager::ProcessCallbacksNow+0xfb
0f msedge!base::AtExitManager::~AtExitManager+0x24
10 msedge!content::ContentMainRunnerImpl::Shutdown+0xa6
11 msedge!content::RunContentProcess+0x270
12 msedge!content::ContentMain+0x5f
13 msedge!ChromeMain+0x2e1
14 msedgewebview2!MainDllLoader::Launch+0x3e4
15 msedgewebview2!wWinMain+0x1de
16 msedgewebview2!__scrt_common_main_seh+0x106
17 kernel32!BaseThreadInitThunk+0x10
18 ntdll!RtlUserThreadStart+0x2b

Clearly, the main thread is in the middle of shutting down. It can be assumed that there are other pending AtExit's running, though I did not bother confirming this (Claude Opus 5 says it's 98, including base::LazyInstance<base::win::MessageWindow::WindowClass> and base::Singleton<gfx::'anonymous'>, but I did not dig into WinDbg to verify this myself).

Anyway, the thread is doing an INFINITE wait; !handle reports:

Handle 00000000000010a8
  Type          Event
  Attributes    0
  GrantedAccess 0x1f0003:
         Delete,ReadControl,WriteDac,WriteOwner,Synch
         QueryState,ModifyState
  HandleCount   2
  PointerCount  65536
  Name          <none>
  Object specific information
    Event Type Auto Reset
    Event is Waiting

There's a rather laborious but effective way in WinDbg/CDB to figure out what type something may be:

• !heap -x to figure out heap block
• dps to look at it; the first entry is usually the vtable
• dps <vtable>-8; the _RTTICompleteObjectLocator is here
• dd <rtti locator> + 0xc; that's the pTypeDescriptor entry, relative to modbase
• !lmi <module> to get the base of the module
• da <module>+<pTypeDescriptor>+0x10 to get the mangled symbol name.
  (see include\rttidata.h in your local MSVC install for details on the RTTI structures)

This gave us shared_ptr<Msai::AutoResetEvent>; not helpful on its own. But other threads help form the bigger picture:

Two threads are identically in this callstack:

[0x0]   ntdll!NtWaitForSingleObject+0x14   0xe6977ff3b8   0x7fffb3acd5ae   
[0x1]   KERNELBASE!WaitForSingleObjectEx+0x8e   0xe6977ff3c0   0x7fff5f5a2933   
[0x2]   oneauth+0x62933   0xe6977ff460   0x7fff5f5a08de   
[0x3]   oneauth+0x608de   0xe6977ff530   0x7fff5f59ec34   
[0x4]   oneauth+0x5ec34   0xe6977ff6d0   0x7fff5f5bdfca   
[0x5]   oneauth+0x7dfca   0xe6977ff810   0x7fff5f5a32a7   
[0x6]   oneauth+0x632a7   0xe6977ff8d0   0x7fff5f59cdc2   
[0x7]   oneauth+0x5cdc2   0xe6977ffa10   0x7fff5f59c991   
[0x8]   oneauth+0x5c991   0xe6977ffbe0   0x7fff5f548486   
[0x9]   oneauth+0x8486   0xe6977ffc90   0x7fff5f6530ef   
[0xa]   oneauth!OneAuthGetVersion+0x1439f   0xe6977ffce0   0x7fffb4374cb0   
[0xb]   kernel32!BaseThreadInitThunk+0x10   0xe6977ffd10   0x7fffb641edeb   
[0xc]   ntdll!RtlUserThreadStart+0x2b   0xe6977ffd40   0x0   

They're each waiting on a handle like:

Handle 0000000000000db8
  Type         	Event
  Attributes   	0
  GrantedAccess	0x1f0003:
         Delete,ReadControl,WriteDac,WriteOwner,Synch
         QueryState,ModifyState
  HandleCount  	2
  PointerCount 	65536
  Name         	<none>
  Object specific information
    Event Type Manual Reset
    Event is Waiting

The above laborious sequence + a demangler gives us:

winrt::impl::delegate<winrt::Windows::Foundation::AsyncOperationCompletedHandler<winrt::Windows::Security::Authentication::Web::Core::FindAllAccountsResult>, shared_type>

And that leads us to this thread:

00 ntdll!NtWaitForMultipleObjects+0x14
01 KERNELBASE!WaitForMultipleObjectsEx+0xec
02 Windows_Security_Authentication_OnlineId!WLIDCCreateContextEx+0x1fc
03 Windows_Security_Authentication_OnlineId!...::UserHostAuthenticationOperation::SetupIdentityHandle+0x151
04 Windows_Security_Authentication_OnlineId!...::UserHostAuthenticationOperation::RetrieveDeviceData+0x1e1c8
05 Windows_Security_Authentication_OnlineId!...::UserHostAuthenticationOperation::DoWork+0x3a4
06 Windows_Security_Authentication_OnlineId!CAsyncOperationBase<...UserHostIdentity*...>::DoWorkStub+0x7c
07 ntdll!TppWorkpExecuteCallback+0x13a
08 ntdll!TppWorkerThread+0x686
09 kernel32!BaseThreadInitThunk+0x10
0a ntdll!RtlUserThreadStart+0x2b

Interestingly, this worker thread was started almost immediately after process start and has been idle (presumably blocked) since:

0:040> !runaway 7
 User Mode Time
  Thread       Time
    0:24a4     0 days 0:00:00.593
   42:23f0     0 days 0:00:00.000
 Kernel Mode Time
  Thread       Time
    0:24a4     0 days 0:00:02.828
   42:23f0     0 days 0:00:00.031
 Elapsed Time
  Thread       Time
    0:24a4     0 days 0:02:04.446 <-- main thread, for comparison
   42:23f0     0 days 0:02:00.615 <-- the thread above

So, this shows:

• the process is shutting down
• it's blocked on a wlidsvc call
• wlidsvc is not answering

wlidsvc

We got live access briefly, and this time pulled dumps from the svchost belonging to wlidsvc.

One thread is blocked here:

00 ntdll!NtWaitForSingleObject+0x14
01 KERNELBASE!WaitForSingleObjectEx+0x8e
02 winhttp!HTTP_USER_REQUEST::_HandleSyncPending+0xfc
03 winhttp!HTTP_USER_REQUEST::SendRequest+0x281
04 winhttp!WinHttpSendRequest+0x2b5
05 wlidsvc!ServiceWinApi::WinHttpSendRequest+0x41
06 wlidsvc!CWinhttpSessions::SendRequestAndReceiveResponse+0xc5
07 wlidsvc!CTransport::SendImplementation+0x4f3
08 wlidsvc!CTransport::SendInternalHelper+0x289
09 wlidsvc!CTransport::SendInternal+0xe8
0a wlidsvc!CTransport::SendRequest+0x14b
0b wlidsvc!CSingleIdentity::AuthDeviceIdentity+0x443
0c wlidsvc!DeviceIdHelpers::AuthDevice+0xc8
0d wlidsvc!CDeviceIdentityBase::Authenticate+0x8a
0e wlidsvc!CDeviceIdentityBase::Provision+0x1c2
0f wlidsvc!DeviceIdHelpers::CreateNewDeviceIdentityInternal+0x6d7
10 wlidsvc!DeviceIdHelpers::CreateNewDeviceIdentity+0x72
11 wlidsvc!DeviceIdHelpers::ProvisionDeviceId+0x155
12 wlidsvc!DeviceIdHelpers::RetrieveDeviceID+0x4df08
13 wlidsvc!DeviceIdHelpers::GetDeviceAuthToken+0xa3
14 wlidsvc!DeviceIdHelpers::GetLogicalDeviceDAToken+0x39
15 wlidsvc!ProcessUserLogon+0x1c4
16 wlidsvc!CWLIDTimerQueue::TimerDriver+0x10c
17 ntdll!RtlpTpTimerCallback+0x83
18 ntdll!TppTimerpExecuteCallback+0xb3
19 ntdll!TppWorkerThread+0x686
1a kernel32!BaseThreadInitThunk+0x10
1b ntdll!RtlUserThreadStart+0x2b

A bunch of other threads are blocked here:

00 ntdll!NtWaitForSingleObject+0x14
01 KERNELBASE!WaitForSingleObjectEx+0x8e
02 wlidsvc!DeviceIdHelpers::AcquireProvisioningMutex+0x147
03 wlidsvc!DeviceIdHelpers::CreateNewDeviceIdentityInternal+0x1b7
04 wlidsvc!DeviceIdHelpers::CreateNewDeviceIdentity+0x72
05 wlidsvc!DeviceIdHelpers::ProvisionDeviceId+0x155
06 wlidsvc!DeviceIdHelpers::RetrieveDeviceID+0x4df08
07 wlidsvc!GetDeviceIdInternal+0x143
08 wlidsvc!CIdentityStore::GetNewIdentityHandle+0x298
09 wlidsvc!HandleCreateContext+0x179
0a wlidsvc!WLIDCreateContext+0xb2
0b rpcrt4!Invoke+0x73
0c rpcrt4!NdrStubCall2+0x44f
0d rpcrt4!NdrServerCall2+0x1a
0e rpcrt4!DispatchToStubInCNoAvrf+0x22
0f rpcrt4!RPC_INTERFACE::DispatchToStubWorker+0x1af
10 rpcrt4!RPC_INTERFACE::DispatchToStub+0xf8
11 rpcrt4!LRPC_SCALL::DispatchRequest+0x33c
12 rpcrt4!LRPC_SCALL::HandleRequest+0x837
13 rpcrt4!LRPC_SASSOCIATION::HandleRequest+0x24d
14 rpcrt4!LRPC_ADDRESS::HandleRequest+0x181
15 rpcrt4!LRPC_ADDRESS::ProcessIO+0x897
16 rpcrt4!LrpcIoComplete+0xc9
17 ntdll!TppAlpcpExecuteCallback+0x280
18 ntdll!TppDirectExecuteCallback+0xdc
19 ntdll!TppWorkerThread+0x686
1a kernel32!BaseThreadInitThunk+0x10
1b ntdll!RtlUserThreadStart+0x2b

And then the most interesting bit is netstat -ano:

Proto  Local Address          Foreign Address        State           PID
  TCP    10.206.68.56:50465     40.126.31.71:443       SYN_SENT        2768

Digging around the call sites of the thread shows it's trying to get at https://login.live.com/RST2.srf, but that's blackholed by the corporate firewall running on-site, and we run into a timeout. The RPC call timeout is 600s, while the HTTP call connect timeout is 60s (as read out from the WinHTTP parameters in the dump), though presumably it's being retried - in any case, it's hanging for much, much longer than one minute.

Knowing that, we were able to mitigate it. Rerouting login.live.com to something invalid in /etc/hosts helped, as did disabling the service in the first place.

As mentioned, please don't hesitate to ask me anything.