Azure / Azure/azure-functions-host

Host not starting up after MSDeploy

Open
#7,010 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 12h
Merged PRs (30d)
38

Description

I received a customer issue (see ICM incident 214395142) where the customer's app got 500 Internal Server Errors after deploying through MSDeploy.

The app seemed to have gone offline during the deployment and stayed in that state instead of coming back online. This caused the host to never start up and any pings to it showed the host in an offline state. Looks like the host isn't handling the `App_Offline.htm` file consistently.

This issue is similar to #4702.

#### Investigative information

- Timestamp: Timestamp: 2020-11-04 08:58:00 UTC
- Region: North Europe
- See ICM incident 214395142 for more information.

#### Expected behavior

Host starts up after a deployment through MSDeploy.

#### Actual behavior

Host doesn't startup after a deployment through MSDeploy and the customer's app sees 500 Internal Server Errors.

#### Known workarounds

The customer restarted their app to mitigate the issue and get the host to start up. (Timestamp: 2020-11-04 15:54)

#### Related information
Relevant logs:
```
File change of type 'Changed' detected for 'D:\home\site\wwwroot\App_Offline.htm'
Host configuration has changed. Signaling shutdown
File change of type 'Changed' detected for 'D:\home\site\wwwroot\App_Offline.htm'
Host configuration has changed. Signaling restart
File change of type 'Changed' detected for 'D:\home\site\wwwroot\App_Offline.htm'
Host configuration has changed. Signaling shutdown
File change of type 'Changed' detected for 'D:\home\site\wwwroot\App_Offline.htm'
Host configuration has changed. Signaling restart
File change of type 'Changed' detected for 'D:\home\site\wwwroot\App_Offline.htm'
Host configuration has changed. Signaling shutdown
File change of type 'Changed' detected for 'D:\home\site\wwwroot\App_Offline.htm'
Host configuration has changed. Signaling restart
```
```
Ping Status: {
"hostState": "Offline"
}
```
Relevant code:

1. Code that checks if the app is offline: https://github.com/Azure/azure-functions-host/blob/ea772a8c33bdb4b614ee537fd79ea07ce1c5974d/src/WebJobs.Script/Workers/Rpc/RpcInitializationService.cs#L68
2. Code that evaluates skipJobHostStartup's value and prints out the "Building host" log: https://github.com/Azure/azure-functions-host/blob/ea772a8c33bdb4b614ee537fd79ea07ce1c5974d/src/WebJobs.Script.WebHost/WebJobsScriptHostService.cs#L238
3. Calling TraceFileChangeRestart to create the `File change of type 'Changed' detected for 'D:\home\site\wwwroot\App_Offline.htm'` logs:
https://github.com/Azure/azure-functions-host/blob/ea772a8c33bdb4b614ee537fd79ea07ce1c5974d/src/WebJobs.Script.WebHost/FileMonitoringService.cs#L283
https://github.com/Azure/azure-functions-host/blob/ea772a8c33bdb4b614ee537fd79ea07ce1c5974d/src/WebJobs.Script.WebHost/FileMonitoringService.cs#L306

Contributor guide

Open the contributing guide

Research direction

Start with RpcInitializationService.cs and WebJobsScriptHostService.cs at the linked locations, then trace FileMonitoringService.cs where TraceFileChangeRestart produces the App_Offline.htm change logs. Compare the offline check, skipJobHostStartup handling, and shutdown/restart sequence against the supplied logs. Done means the host starts after MSDeploy and the app no longer remains offline or returns 500 errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
backend, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.