Azure / Azure/azure-functions-host

Functions Host goes into bad state after OutOfMemoryException

Open
#3,624 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 10h
Merged PRs (30d)
36

Description

#### Investigative information

> INTERNAL ONLY - see IcM \#8756524 for an instance of this issue with all the details

#### Summary
The functions host (1.0) was shutdown due to an OutOfMemoryException. The functions host failed to restart itself after being in this condition. The process was kept alive by AlwaysOn, but the host didn't restart until a restart command was manually sent to the App Service infrastructure

Here is the callstack of the OutOfMemoryException. Note that it happens in user code:
```
Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: Functions.ExtractAndNotify ---> System.Reflection.TargetInvocationException : Exception has been thrown by the target of an invocation. --->
System.OutOfMemoryException : Exception of type 'System.OutOfMemoryException' was thrown.
at Submission#0.PostErrorMessage(String queueItem,String errorMessage,String baseMessage)
at Submission#0.Run(String queueItem,TraceWriter log)
End of inner exception
at System.RuntimeMethodHandle.InvokeMethod(Object target,Object[] arguments,Signature sig,Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj,Object[] parameters,Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj,BindingFlags invokeAttr,Binder binder,Object[] parameters,CultureInfo culture)
at async Microsoft.Azure.WebJobs.Script.Description.DotNetFunctionInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context)
at async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters)
at async Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker`2.InvokeAsync[TReflected,TReturnType](TReflected instance,Object[] arguments)
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync[TReflected,TReturnValue](Object instance,Object[] arguments)
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeAsync(IFunctionInvoker invoker,ParameterHelper parameterHelper,CancellationTokenSource timeoutTokenSource,CancellationTokenSource functionCancellationTokenSource,Boolean throwOnTimeout,TimeSpan timerInterval,IFunctionInstance instance)
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstance instance,ParameterHelper parameterHelper,TraceWriter traceWriter,CancellationTokenSource functionCancellationTokenSource)
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(??)
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(??)
End of inner exception
```

Functions logs at the time:

| TIMESTAMP | Pid | Summary |
| --- | --- | --- |
| 2018-10-11 06:30:31.1958763 | 16512 | An unhandled exception has occurred. Host is shutting down. |
| 2018-10-11 06:30:31.2273143 | 16512 | Message has reached MaxDequeueCount of 1. Moving message to queue '**redacted**-queue-poison'. |
| 2018-10-11 06:30:31.2709304 | 16512 | Stopping Host |
| 2018-10-11 06:30:31.2808059 | 16512 | Singleton lock released (**redacted**/Host.Functions.MaterialsReceiveSubscriber.Listener) |
| 2018-10-11 06:30:31.3614276 | 16512 | Job host stopped |
| 2018-10-11 06:30:52.6076518 | 16512 | EnvSettings: SettingsProcessor.Start apply environments to appSettings or connectionStrings |
| 2018-10-11 06:30:52.6076949 | 16512 | EnvSettings: SettingsProcessor.Start completed.|
| (..long delay – no data..) | | |
| 2018-10-11 09:02:30.3865452 | 11360 | EnvSettings: SettingsProcessor.Start apply environments to appSettings or connectionStrings |
| 2018-10-11 09:02:30.3868933 | 11360 | EnvSettings: SettingsProcessor.Start completed. |
| 2018-10-11 09:02:30.7642940 | 11360 | Created directory snapshot. |
| 2018-10-11 09:02:30.8362800 | 11360 | Reading host configuration file 'D:\home\site\wwwroot\host.json' |

#### Expected behavior

The functions host should automatically restart after shutting down.

#### Actual behavior

The functions host stayed in a shutdown state. It continued to respond successfully to ping requests from AlwaysOn.

#### Known workarounds

Manually restart the function app when it gets into this state.

Contributor guide

Open the contributing guide

Research direction

Start with the host shutdown/restart lifecycle and the startup path evidenced by the EnvSettings and host.json logs. Compare behavior after an unhandled OutOfMemoryException with the expected automatic restart, using the AlwaysOn and ping scenario as reproduction signals; done means the host leaves its shutdown state and restarts automatically.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.