Azure / Azure/azure-functions-host
Functions Host goes into bad state after OutOfMemoryException
- Dominant language
- C#
- Stars
- 2k
- Forks
- 482
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 36
Description
#### Investigative information
> INTERNAL ONLY - see IcM \#8756524 for an instance of this issue with all the details
#### Summary
The functions host (1.0) was shutdown due to an OutOfMemoryException. The functions host failed to restart itself after being in this condition. The process was kept alive by AlwaysOn, but the host didn't restart until a restart command was manually sent to the App Service infrastructure
Here is the callstack of the OutOfMemoryException. Note that it happens in user code:
```
Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: Functions.ExtractAndNotify ---> System.Reflection.TargetInvocationException : Exception has been thrown by the target of an invocation. --->
System.OutOfMemoryException : Exception of type 'System.OutOfMemoryException' was thrown.
at Submission#0.PostErrorMessage(String queueItem,String errorMessage,String baseMessage)
at Submission#0.Run(String queueItem,TraceWriter log)
End of inner exception
at System.RuntimeMethodHandle.InvokeMethod(Object target,Object[] arguments,Signature sig,Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj,Object[] parameters,Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj,BindingFlags invokeAttr,Binder binder,Object[] parameters,CultureInfo culture)
at async Microsoft.Azure.WebJobs.Script.Description.DotNetFunctionInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context)
at async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters)
at async Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker`2.InvokeAsync[TReflected,TReturnType](TReflected instance,Object[] arguments)
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync[TReflected,TReturnValue](Object instance,Object[] arguments)
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeAsync(IFunctionInvoker invoker,ParameterHelper parameterHelper,CancellationTokenSource timeoutTokenSource,CancellationTokenSource functionCancellationTokenSource,Boolean throwOnTimeout,TimeSpan timerInterval,IFunctionInstance instance)
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstance instance,ParameterHelper parameterHelper,TraceWriter traceWriter,CancellationTokenSource functionCancellationTokenSource)
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(??)
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(??)
End of inner exception
```
Functions logs at the time:
| TIMESTAMP | Pid | Summary |
| --- | --- | --- |
| 2018-10-11 06:30:31.1958763 | 16512 | An unhandled exception has occurred. Host is shutting down. |
| 2018-10-11 06:30:31.2273143 | 16512 | Message has reached MaxDequeueCount of 1. Moving message to queue '**redacted**-queue-poison'. |
| 2018-10-11 06:30:31.2709304 | 16512 | Stopping Host |
| 2018-10-11 06:30:31.2808059 | 16512 | Singleton lock released (**redacted**/Host.Functions.MaterialsReceiveSubscriber.Listener) |
| 2018-10-11 06:30:31.3614276 | 16512 | Job host stopped |
| 2018-10-11 06:30:52.6076518 | 16512 | EnvSettings: SettingsProcessor.Start apply environments to appSettings or connectionStrings |
| 2018-10-11 06:30:52.6076949 | 16512 | EnvSettings: SettingsProcessor.Start completed.|
| (..long delay – no data..) | | |
| 2018-10-11 09:02:30.3865452 | 11360 | EnvSettings: SettingsProcessor.Start apply environments to appSettings or connectionStrings |
| 2018-10-11 09:02:30.3868933 | 11360 | EnvSettings: SettingsProcessor.Start completed. |
| 2018-10-11 09:02:30.7642940 | 11360 | Created directory snapshot. |
| 2018-10-11 09:02:30.8362800 | 11360 | Reading host configuration file 'D:\home\site\wwwroot\host.json' |
#### Expected behavior
The functions host should automatically restart after shutting down.
#### Actual behavior
The functions host stayed in a shutdown state. It continued to respond successfully to ping requests from AlwaysOn.
#### Known workarounds
Manually restart the function app when it gets into this state.
Contributor guide
Research direction
Start with the host shutdown/restart lifecycle and the startup path evidenced by the EnvSettings and host.json logs. Compare behavior after an unhandled OutOfMemoryException with the expected automatic restart, using the AlwaysOn and ping scenario as reproduction signals; done means the host leaves its shutdown state and restarts automatically.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100