dotnet / dotnet/aspnetcore

Yarp app pool intermittently crash

Open
#63,777 3 comments 0 reactions 0 assignees View on GitHub
area-networking feature-iis
Dominant language
C#
Stars
38.4k
Forks
10.9k
Avg merge
2d 6h
Merged PRs (30d)
290

Description

### Describe the bug
I have a customer opened support ticket for this issue. Latest Yarp hosted on IIS with high volume, the w3wp.exe process for Yarp intermittently crash. crash can happen 30 times a day, and can be also once in 2 months. unpredictable.

### To Reproduce
VM Azure Hosting latest Yarp and a bunch of ASP.Net Core app. Issue is the w3wp process for YARP intermittently crash. crash can happen 30 times a day, and can be also once in 2 months. unpredictable. Recently more frequently.

crash mostly happened on server 04 and 05. Very occasionally, crash can happen on other servers too, there is one server never had crash.

The Yarp takes incoming requests, and routes to web apps (same ASP.Net core version apps) hosted on the same IIS server.
The request count is very high about 1000 concurrent requests per second at peak. But crash can happen any time.

And although there are over 1 dozen ASP.Net core apps hosted on every server, only the YARP app pool crashes. checked YARP app pool and it is set to “No Managed Code”, “Always Running” “Idle Timeout” 0, no any recycle time or condition is set.
When YARP app pool crashes, nothing else crashes.

### Further technical details

Crash dumps were collected and call stack at crash varies a little in each dump, see crash call stacks below:

```
00 (Inline Function) --------`-------- iiscore!W3_CONTEXT_BASE::ContinueNotificationLoop(unsigned long cbCompletion = 0xb60, HRESULT hrCompletionStatus = 0x00000000)+0x4 [servercommon\inetsrv\iis\iisrearc\iis70\core\w3basecontext.cxx @ 1149]
01 000000b3`a51ffbc0 00007ffb`04792e0a iiscore!W3_MAIN_CONTEXT::OnIoCompletion(unsigned long dwIoOperation = 1, void * pvContext = 0x00000368`36da6130, unsigned long cbCompletion = 0xb60, unsigned long dwCompletionStatus = 0, struct _OVERLAPPED * __formal = 0x00000368`36da6100)+0xd8 [servercommon\inetsrv\iis\iisrearc\iis70\core\maincontext.cxx @ 200]
02 (Inline Function) --------`-------- w3dt!UL_NATIVE_REQUEST::DoStateProcess+0x1a3e [servercommon\inetsrv\iis\iisrearc\iisplus\w3dt\nativerequest.cxx @ 741]
03 (Inline Function) --------`-------- w3dt!UL_NATIVE_REQUEST::DoWork+0x1a80 [servercommon\inetsrv\iis\iisrearc\iisplus\w3dt\nativerequest.cxx @ 369]
04 000000b3`a51ffc20 00007ffb`04791b45 w3dt!UL_RECEIVE_CONTEXT::DoWork+0x1a8a [servercommon\inetsrv\iis\iisrearc\iisplus\w3dt\nativerequest.cxx @ 2368]
05 000000b3`a51ffc60 00007ffb`09eb1133 w3dt!WP_CONTEXT::OnCompletion(unsigned long dwErrorCode = 0x36da6130, unsigned long dwNumberOfBytesTransfered = 0xb60, struct _OVERLAPPED * lpOverlapped = 0x00000368`36da6100)+0x45 [servercommon\inetsrv\iis\iisrearc\iisplus\w3dt\wpcontext.cxx @ 348]
06 000000b3`a51ffca0 00007ffb`09eb1048 w3tp!THREAD_POOL_DATA::ThreadPoolThread(void)+0x83 [servercommon\inetsrv\iis\iisrearc\iisplus\w3tp\thread_pool.cxx @ 1108]
07 000000b3`a51ffcf0 00007ffb`09eb1b59 w3tp!THREAD_POOL_DATA::ThreadPoolThread(void * pvThis = 0x000001ea`c7868b10)+0x38 [servercommon\inetsrv\iis\iisrearc\iisplus\w3tp\thread_pool.cxx @ 965]
08 000000b3`a51ffd30 00007ffb`13064cb0 w3tp!THREAD_MANAGER::ThreadManagerThread(void * ThreadParam = 0x000001ea`c7834c50)+0x59 [servercommon\inetsrv\iis\iisrearc\iisplus\w3tp\thread_manager.cxx @ 877]
09 000000b3`a51ffd70 00007ffb`1437edcb kernel32!BaseThreadInitThunk(unsigned long RunProcessInit = , * StartAddress = , void * Argument = )+0x10 [clientcore\base\win32\client\thread.c @ 75]
0a 000000b3`a51ffda0 00000000`00000000 ntdll!RtlUserThreadStart( * StartAddress = 0x00000000`00000000, void * Argument = 0x00000000`00000000)+0x2b [minkernel\ntdll\rtlstrt.c @ 1152]

00 00000043`a7e4fab0 00007ffa`c6be1edd KERNELBASE!RaiseException(unsigned long dwExceptionCode = , unsigned long dwExceptionFlags = 0xcb9377c8, unsigned long nNumberOfArguments = 0xcb9377c8, unsigned int64 * lpArguments = 0x00000000`00000000)+0x6c [minkernel\kernelbase\xcpt.c @ 955]
01 00000043`a7e4fb90 00007ffa`c6b81599 iiscore!W3_MAIN_CONTEXT::OnIoCompletion(unsigned long dwIoOperation = 2, void * pvContext = 0x000002bb`c7fe78e0, unsigned long cbCompletion = 0x20000, unsigned long dwCompletionStatus = , struct _OVERLAPPED * __formal = 0x000002bb`c7fe7888)+0x9d [servercommon\inetsrv\iis\iisrearc\iis70\core\maincontext.cxx @ 184]
02 (Inline Function) --------`-------- w3dt!UL_NATIVE_REQUEST::DoStateProcess(void)+0x3a [servercommon\inetsrv\iis\iisrearc\iisplus\w3dt\nativerequest.cxx @ 741]
03 (Inline Function) --------`-------- w3dt!UL_NATIVE_REQUEST::DoWork(void)+0x6f [servercommon\inetsrv\iis\iisrearc\iisplus\w3dt\nativerequest.cxx @ 369]
04 00000043`a7e4fbf0 00007ffa`c6b81b45 w3dt!UL_SEND_CONTEXT::DoWork(unsigned long cbData = , unsigned long dwError = , struct _OVERLAPPED * lpo = 0x000002bb`c7fe7888)+0x79 [servercommon\inetsrv\iis\iisrearc\iisplus\w3dt\nativerequest.cxx @ 2350]
05 00000043`a7e4fc30 00007ffa`cb3b1133 w3dt!WP_CONTEXT::OnCompletion(unsigned long dwErrorCode = 0xcb9377c8, unsigned long dwNumberOfBytesTransfered = 0x20000, struct _OVERLAPPED * lpOverlapped = 0x000002bb`c7fe7888)+0x45 [servercommon\inetsrv\iis\iisrearc\iisplus\w3dt\wpcontext.cxx @ 348]
06 00000043`a7e4fc70 00007ffa`cb3b1048 w3tp!THREAD_POOL_DATA::ThreadPoolThread(void)+0x83 [servercommon\inetsrv\iis\iisrearc\iisplus\w3tp\thread_pool.cxx @ 1108]
07 00000043`a7e4fcc0 00007ffa`cb3b1b59 w3tp!THREAD_POOL_DATA::ThreadPoolThread(void * pvThis = 0x000001bd`83613b30)+0x38 [servercommon\inetsrv\iis\iisrearc\iisplus\w3tp\thread_pool.cxx @ 965]
08 00000043`a7e4fd00 00007ffa`d4334cb0 w3tp!THREAD_MANAGER::ThreadManagerThread(void * ThreadParam = 0x000001bd`83be7600)+0x59 [servercommon\inetsrv\iis\iisrearc\iisplus\w3tp\thread_manager.cxx @ 877]
09 00000043`a7e4fd40 00007ffa`d5ddedcb kernel32!BaseThreadInitThunk(unsigned long RunProcessInit = , * StartAddress = , void * Argument = )+0x10 [clientcore\base\win32\client\thread.c @ 75]
0a 00000043`a7e4fd70 00000000`00000000 ntdll!RtlUserThreadStart( * StartAddress = 0x00000000`00000000, void * Argument = 0x00000000`00000000)+0x2b [minkernel\ntdll\rtlstrt.c @ 1152]

00 (Inline Function) --------`-------- iiscore!W3_CONTEXT_BASE::ReferenceW3Context(void) [servercommon\inetsrv\iis\iisrearc\iis70\core\w3basecontext.h @ 562]
01 (Inline Function) --------`-------- iiscore!IISCORE_ASYNC_CONTEXT::SetIoCompletionContext(void)+0x22 [servercommon\inetsrv\iis\iisrearc\iis70\core\w3request.cxx @ 3638]
02 0000007b`5b97f2a0 00007ffe`8cfb594a iiscore!NOTIFICATION_SEND_RESPONSE::DoWork(unsigned long cbCompletion = 0, HRESULT hrCompletionStatus = , unsigned long dwCompletedOperation = , int fCompletion = 0n0)+0xbf [servercommon\inetsrv\iis\iisrearc\iis70\core\w3response.cxx @ 3457]
03 (Inline Function) --------`-------- iiscore!W3_CONTEXT_BASE::StartNotificationLoop(void)+0x6a [servercommon\inetsrv\iis\iisrearc\iis70\core\w3basecontext.cxx @ 1083]
04 0000007b`5b97f320 00007ffe`8cfb5cda iiscore!W3_RESPONSE::Flush(int fAsync = 0n1, int fMoreData = , class W3_CONTEXT_BASE * pW3BaseContext = 0x000002bb`babc0790, unsigned long * pcbSent = 0x0000007b`5b97f434, int * pfCompletionExpected = 0x0000007b`5b97f430)+0x39a [servercommon\inetsrv\iis\iisrearc\iis70\core\w3response.cxx @ 2016]
05 0000007b`5b97f3d0 00007ffe`8cfb1f43 iiscore!NOTIFICATION_MAIN::DoWork(unsigned long cbCompletion = 0, HRESULT hrCompletionStatus = 0x00000000, unsigned long dwCompletedOperation = 1, int fCompletion = 0n1)+0x25a [servercommon\inetsrv\iis\iisrearc\iis70\core\mainnotification.cxx @ 255]
06 (Inline Function) --------`-------- iiscore!W3_CONTEXT_BASE::ContinueNotificationLoop(unsigned long cbCompletion = , HRESULT hrCompletionStatus = 0x00000000)+0x2f [servercommon\inetsrv\iis\iisrearc\iis70\core\w3basecontext.cxx @ 1149]
07 0000007b`5b97f6d0 00007ffe`90422e0a iiscore!W3_MAIN_CONTEXT::OnIoCompletion(unsigned long dwIoOperation = 1, void * pvContext = 0x000002bb`babc0790, unsigned long cbCompletion = 0x7b0, unsigned long dwCompletionStatus = , struct _OVERLAPPED * __formal = 0x000002bb`babc0760)+0x103 [servercommon\inetsrv\iis\iisrearc\iis70\core\maincontext.cxx @ 200]
08 (Inline Function) --------`-------- w3dt!UL_NATIVE_REQUEST::DoStateProcess+0x1a3e [servercommon\inetsrv\iis\iisrearc\iisplus\w3dt\nativerequest.cxx @ 741]
09 (Inline Function) --------`-------- w3dt!UL_NATIVE_REQUEST::DoWork+0x1a80 [servercommon\inetsrv\iis\iisrearc\iisplus\w3dt\nativerequest.cxx @ 369]
0a 0000007b`5b97f730 00007ffe`90421b45 w3dt!UL_RECEIVE_CONTEXT::DoWork+0x1a8a [servercommon\inetsrv\iis\iisrearc\iisplus\w3dt\nativerequest.cxx @ 2368]
0b 0000007b`5b97f770 00007ffe`939a1133 w3dt!WP_CONTEXT::OnCompletion(unsigned long dwErrorCode = 0xffffffff, unsigned long dwNumberOfBytesTransfered = 0x7b0, struct _OVERLAPPED * lpOverlapped = 0x000002bb`babc0760)+0x45 [servercommon\inetsrv\iis\iisrearc\iisplus\w3dt\wpcontext.cxx @ 348]
0c 0000007b`5b97f7b0 00007ffe`939a1048 w3tp!THREAD_POOL_DATA::ThreadPoolThread(void)+0x83 [servercommon\inetsrv\iis\iisrearc\iisplus\w3tp\thread_pool.cxx @ 1108]
0d 0000007b`5b97f800 00007ffe`939a1b59 w3tp!THREAD_POOL_DATA::ThreadPoolThread(void * pvThis = 0x000001bd`71a23310)+0x38 [servercommon\inetsrv\iis\iisrearc\iisplus\w3tp\thread_pool.cxx @ 965]
0e 0000007b`5b97f840 00007ffe`9d904cb0 w3tp!THREAD_MANAGER::ThreadManagerThread(void * ThreadParam = 0x000001bd`71ac0010)+0x59 [servercommon\inetsrv\iis\iisrearc\iisplus\w3tp\thread_manager.cxx @ 877]
0f 0000007b`5b97f880 00007ffe`9e41edcb kernel32!BaseThreadInitThunk(unsigned long RunProcessInit = , * StartAddress = , void * Argument = )+0x10 [clientcore\base\win32\client\thread.c @ 75]
10 0000007b`5b97f8b0 00000000`00000000 ntdll!RtlUserThreadStart( * StartAddress = 0x00000000`00000000, void * Argument = 0x00000000`00000000)+0x2b [minkernel\ntdll\rtlstrt.c @ 1152]
```

- Include the version of the packages you are using: upgraded to latest version of Yarp and issue persists.
- The platform (Linux/macOS/Windows) Windows Server 2022, Azure VM.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.