dotnet / dotnet/aspnetcore

IIS AspNetCoreModuleV2: random/endless recycles

Open
#54,531 30 comments 2 reactions 0 assignees View on GitHub
:heavy_check_mark: Resolution: Fixed area-networking feature-iis
Dominant language
C#
Stars
38.4k
Forks
10.9k
Avg merge
2d 5h
Merged PRs (30d)
276

Description

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Describe the bug

After deploying an update of our application at one of our customers in IIS, our application was stuck in an endless loop. It would always start successfully and then be recycled a few seconds later.

In the event log we saw the following:

```
An ISAPI reported an unhealthy condition to its worker process. Therefore, the worker process with process id of '9092' serving application pool '' has requested a recycle.
```

This only stopped after we completely restarted the IIS server. This was not reproducible at first, but happened again a few weeks later after another update.

After some investigation, we noticed that this would still happen randomly a few times a day, even after fixing the "endless loop" by restarting IIS. All auto-recycle settings on the application pool are disabled. After enabling the debug logging for the ASP .NET Core Module, we saw the following log event when a random recycle happened:

```
[2024-03-13T12:40:45.437Z, PID: 11716] [aspnetcorev2.dll] ASPNET_CORE_GLOBAL_MODULE::OnGlobalConfigurationChange ''
[2024-03-13T12:40:45.439Z, PID: 11716] [aspnetcorev2.dll] Stopping application '/LM/W3SVC/2/ROOT/'
```

Relevant code here:

https://github.com/dotnet/aspnetcore/blob/268a2dfc29b33e3fdb73cbac6eb198c05314d77e/src/Servers/IIS/AspNetCoreModuleV2/AspNetCore/globalmodule.cpp#L52-L68

Notice that path in the OnGlobalConfigurationChange log message is empty (as far as I can see it can't be NULL since then the recycle wouldn't be triggered). Our current assumption is that this has something to do with the problem since we don't know when/how this happens. When you edit an IIS configuration file, this contains the path to the relevant config section.

After further investigation we also saw these random recycles on other customer systems in different infrastructures. However the endless recycle loop only happened on a particular system until now. Unfortunately we have no Debug Logs for the endless loop case and can't reproduce it again. We assume that the endless loop was also caused by configuration changes with an empty path, but we can't be sure.

We currently worked around this setting [disallowRotationOnConfigChange to true](https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/iis/advanced?view=aspnetcore-8.0#disallow-rotation-on-config). We have not seen this happen since.

So the questions are:
- what triggers these global configuration events with the empty path? The [API doc](https://learn.microsoft.com/en-us/iis/web-development-reference/native-code-api-reference/iglobalconfigurationchangeprovider-getchangepath-method) gives no indication what an empty path means.
- Should the ASP .NET Core Module maybe ignore the configuration change if the path is empty? From my understanding this would "fix" the issue.

This happened with ASP .NET Core 8.0.2 on Windows Server 2016 with a self-contained x64 app.

### Expected Behavior

_No response_

### Steps To Reproduce

_No response_

### Exceptions (if any)

_No response_

### .NET Version

8.0.2

### Anything else?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.