Azure / Azure/azure-functions-host
Host not starting up after MSDeploy
- Dominant language
- C#
- Stars
- 2k
- Forks
- 482
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 38
Description
I received a customer issue (see ICM incident 214395142) where the customer's app got 500 Internal Server Errors after deploying through MSDeploy.
The app seemed to have gone offline during the deployment and stayed in that state instead of coming back online. This caused the host to never start up and any pings to it showed the host in an offline state. Looks like the host isn't handling the `App_Offline.htm` file consistently.
This issue is similar to #4702.
#### Investigative information
- Timestamp: Timestamp: 2020-11-04 08:58:00 UTC
- Region: North Europe
- See ICM incident 214395142 for more information.
#### Expected behavior
Host starts up after a deployment through MSDeploy.
#### Actual behavior
Host doesn't startup after a deployment through MSDeploy and the customer's app sees 500 Internal Server Errors.
#### Known workarounds
The customer restarted their app to mitigate the issue and get the host to start up. (Timestamp: 2020-11-04 15:54)
#### Related information
Relevant logs:
```
File change of type 'Changed' detected for 'D:\home\site\wwwroot\App_Offline.htm'
Host configuration has changed. Signaling shutdown
File change of type 'Changed' detected for 'D:\home\site\wwwroot\App_Offline.htm'
Host configuration has changed. Signaling restart
File change of type 'Changed' detected for 'D:\home\site\wwwroot\App_Offline.htm'
Host configuration has changed. Signaling shutdown
File change of type 'Changed' detected for 'D:\home\site\wwwroot\App_Offline.htm'
Host configuration has changed. Signaling restart
File change of type 'Changed' detected for 'D:\home\site\wwwroot\App_Offline.htm'
Host configuration has changed. Signaling shutdown
File change of type 'Changed' detected for 'D:\home\site\wwwroot\App_Offline.htm'
Host configuration has changed. Signaling restart
```
```
Ping Status: {
"hostState": "Offline"
}
```
Relevant code:
1. Code that checks if the app is offline: https://github.com/Azure/azure-functions-host/blob/ea772a8c33bdb4b614ee537fd79ea07ce1c5974d/src/WebJobs.Script/Workers/Rpc/RpcInitializationService.cs#L68
2. Code that evaluates skipJobHostStartup's value and prints out the "Building host" log: https://github.com/Azure/azure-functions-host/blob/ea772a8c33bdb4b614ee537fd79ea07ce1c5974d/src/WebJobs.Script.WebHost/WebJobsScriptHostService.cs#L238
3. Calling TraceFileChangeRestart to create the `File change of type 'Changed' detected for 'D:\home\site\wwwroot\App_Offline.htm'` logs:
https://github.com/Azure/azure-functions-host/blob/ea772a8c33bdb4b614ee537fd79ea07ce1c5974d/src/WebJobs.Script.WebHost/FileMonitoringService.cs#L283
https://github.com/Azure/azure-functions-host/blob/ea772a8c33bdb4b614ee537fd79ea07ce1c5974d/src/WebJobs.Script.WebHost/FileMonitoringService.cs#L306
Contributor guide
Research direction
Start with RpcInitializationService.cs and WebJobsScriptHostService.cs at the linked locations, then trace FileMonitoringService.cs where TraceFileChangeRestart produces the App_Offline.htm change logs. Compare the offline check, skipJobHostStartup handling, and shutdown/restart sequence against the supplied logs. Done means the host starts after MSDeploy and the app no longer remains offline or returns 500 errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp
- Domain
- backend, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100