Azure / Azure/azure-functions-host

deadlock in TableExtension.GetTable

Open
#1,656 6 comments 0 reactions 0 assignees View on GitHub
bug needs-investigation
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 12h
Merged PRs (30d)
38

Description

Deadlock call stack is below. I also have a memory dump. More context in an email with subject `Function monitoring page stuck`

```
[Managed to Native Transition]
> mscorlib.dll!System.Threading.Monitor.Wait(object obj, int millisecondsTimeout, bool exitContext) Unknown
mscorlib.dll!System.Threading.Monitor.Wait(object obj, int millisecondsTimeout) Unknown
mscorlib.dll!System.Threading.ManualResetEventSlim.Wait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Unknown
mscorlib.dll!System.Threading.Tasks.Task.SpinThenBlockingWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Unknown
mscorlib.dll!System.Threading.Tasks.Task.InternalWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Unknown
mscorlib.dll!System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task) Unknown
Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Tables.TableExtension.GetTable(Microsoft.Azure.WebJobs.TableAttribute attribute) Unknown
Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Tables.TableExtension.BuildFromTableAttribute(Microsoft.Azure.WebJobs.TableAttribute attribute) Unknown
Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Bindings.BindingFactory.DelegateConverterBuilder.Convert(System.__Canon input) Unknown
Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Bindings.PatternMatcher.Invoker>.Work.AnonymousMethod__0(object input) Unknown
Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Bindings.AsyncCollectorBindingProvider.ExactBinding.BuildAsync(Microsoft.Azure.WebJobs.TableAttribute attrResolved, Microsoft.Azure.WebJobs.Host.Bindings.ValueBindingContext context) Unknown
Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Bindings.BindingBase.BindAsync(Microsoft.Azure.WebJobs.Host.Bindings.BindingContext context) Unknown
mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start.d__7>(ref Microsoft.Azure.WebJobs.Host.Bindings.BindingBase.d__7 stateMachine) Unknown
Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Bindings.BindingBase.BindAsync(Microsoft.Azure.WebJobs.Host.Bindings.BindingContext context) Unknown
Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Triggers.TriggeredFunctionBinding.BindCoreAsync(Microsoft.Azure.WebJobs.Host.Bindings.ValueBindingContext context, object value, System.Collections.Generic.IDictionary parameters) Unknown
mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder>.Start.d__8>(ref Microsoft.Azure.WebJobs.Host.Triggers.TriggeredFunctionBinding.d__8 stateMachine) Unknown
Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Triggers.TriggeredFunctionBinding.BindCoreAsync(Microsoft.Azure.WebJobs.Host.Bindings.ValueBindingContext context, object value, System.Collections.Generic.IDictionary parameters) Unknown
Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Triggers.TriggeredFunctionBinding.BindAsync(Microsoft.Azure.WebJobs.Host.Bindings.ValueBindingContext context, System.Collections.Generic.IDictionary parameters) Unknown
mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder>.Start.d__7>(ref Microsoft.Azure.WebJobs.Host.Triggers.TriggeredFunctionBinding.d__7 stateMachine) Unknown
Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Triggers.TriggeredFunctionBinding.BindAsync(Microsoft.Azure.WebJobs.Host.Bindings.ValueBindingContext context, System.Collections.Generic.IDictionary parameters) Unknown
Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Executors.BindingSource.BindAsync(Microsoft.Azure.WebJobs.Host.Bindings.ValueBindingContext context) Unknown
Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(Microsoft.Azure.WebJobs.Host.Executors.IFunctionInstance instance, Microsoft.Azure.WebJobs.Host.Protocols.FunctionStartedMessage message, Microsoft.Azure.WebJobs.Host.Loggers.FunctionInstanceLogEntry fastItem, System.Collections.Generic.IDictionary parameterLogCollector, System.Diagnostics.TraceLevel functionTraceLevel, System.Threading.CancellationToken cancellationToken) Unknown
mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Startd__14>(ref Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.d__14 stateMachine) Unknown
Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(Microsoft.Azure.WebJobs.Host.Executors.IFunctionInstance instance, Microsoft.Azure.WebJobs.Host.Protocols.FunctionStartedMessage message, Microsoft.Azure.WebJobs.Host.Loggers.FunctionInstanceLogEntry fastItem, System.Collections.Generic.IDictionary parameterLogCollector, System.Diagnostics.TraceLevel functionTraceLevel, System.Threading.CancellationToken cancellationToken) Unknown
Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(Microsoft.Azure.WebJobs.Host.Executors.IFunctionInstance functionInstance, System.Threading.CancellationToken cancellationToken) Unknown
mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Startd__12>(ref Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.d__12 stateMachine) Unknown
Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(Microsoft.Azure.WebJobs.Host.Executors.IFunctionInstance functionInstance, System.Threading.CancellationToken cancellationToken) Unknown
Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Executors.ShutdownFunctionExecutor.TryExecuteAsync(Microsoft.Azure.WebJobs.Host.Executors.IFunctionInstance instance, System.Threading.CancellationToken cancellationToken) Unknown
mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Startd__3>(ref Microsoft.Azure.WebJobs.Host.Executors.ShutdownFunctionExecutor.d__3 stateMachine) Unknown
Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.Host.Executors.ShutdownFunctionExecutor.TryExecuteAsync(Microsoft.Azure.WebJobs.Host.Executors.IFunctionInstance instance, System.Threading.CancellationToken cancellationToken) Unknown
Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.JobHost.CallAsyncCore(System.Reflection.MethodInfo method, System.Collections.Generic.IDictionary arguments, System.Threading.CancellationToken cancellationToken) Unknown
mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Startd__38>(ref Microsoft.Azure.WebJobs.JobHost.d__38 stateMachine) Unknown
Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.JobHost.CallAsyncCore(System.Reflection.MethodInfo method, System.Collections.Generic.IDictionary arguments, System.Threading.CancellationToken cancellationToken) Unknown
Microsoft.Azure.WebJobs.Host.dll!Microsoft.Azure.WebJobs.JobHost.CallAsync(System.Reflection.MethodInfo method, System.Collections.Generic.IDictionary arguments, System.Threading.CancellationToken cancellationToken) Unknown
Microsoft.Azure.WebJobs.Script.dll!Microsoft.Azure.WebJobs.Script.ScriptHost.CallAsync(string method, System.Collections.Generic.Dictionary arguments, System.Threading.CancellationToken cancellationToken) Unknown
mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Startd__70>(ref Microsoft.Azure.WebJobs.Script.ScriptHost.d__70 stateMachine) Unknown
Microsoft.Azure.WebJobs.Script.dll!Microsoft.Azure.WebJobs.Script.ScriptHost.CallAsync(string method, System.Collections.Generic.Dictionary arguments, System.Threading.CancellationToken cancellationToken) Unknown
Microsoft.Azure.WebJobs.Script.WebHost.dll!Microsoft.Azure.WebJobs.Script.WebHost.WebScriptHostManager.HandleRequestAsync(Microsoft.Azure.WebJobs.Script.Description.FunctionDescriptor function, System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) Unknown
mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Startd__37>(ref Microsoft.Azure.WebJobs.Script.WebHost.WebScriptHostManager.d__37 stateMachine) Unknown
Microsoft.Azure.WebJobs.Script.WebHost.dll!Microsoft.Azure.WebJobs.Script.WebHost.WebScriptHostManager.HandleRequestAsync(Microsoft.Azure.WebJobs.Script.Description.FunctionDescriptor function, System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) Unknown
Microsoft.Azure.WebJobs.Script.WebHost.dll!Microsoft.Azure.WebJobs.Script.WebHost.Controllers.FunctionsController.ProcessRequestAsync(System.Net.Http.HttpRequestMessage request, Microsoft.Azure.WebJobs.Script.Description.FunctionDescriptor function, System.Threading.CancellationToken cancellationToken) Unknown
mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Startd__5>(ref Microsoft.Azure.WebJobs.Script.WebHost.Controllers.FunctionsController.d__5 stateMachine) Unknown
Microsoft.Azure.WebJobs.Script.WebHost.dll!Microsoft.Azure.WebJobs.Script.WebHost.Controllers.FunctionsController.ProcessRequestAsync(System.Net.Http.HttpRequestMessage request, Microsoft.Azure.WebJobs.Script.Description.FunctionDescriptor function, System.Threading.CancellationToken cancellationToken) Unknown
Microsoft.Azure.WebJobs.Script.WebHost.dll!Microsoft.Azure.WebJobs.Script.WebHost.Controllers.FunctionsController.ExecuteAsync.AnonymousMethod__0(System.Net.Http.HttpRequestMessage req, System.Threading.CancellationToken ct) Unknown
mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Startc__DisplayClass3_0.<b__0>d stateMachine) Unknown
Microsoft.Azure.WebJobs.Script.WebHost.dll!Microsoft.Azure.WebJobs.Script.WebHost.Controllers.FunctionsController.ExecuteAsync.AnonymousMethod__0(System.Net.Http.HttpRequestMessage req, System.Threading.CancellationToken ct) Unknown
Microsoft.Azure.WebJobs.Extensions.Http.dll!Microsoft.Azure.WebJobs.Extensions.Http.HttpRequestManager.ProcessRequestAsync(System.Net.Http.HttpRequestMessage request, System.Func> processRequestHandler, System.Threading.CancellationToken cancellationToken) Unknown
mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Startd__8>(ref Microsoft.Azure.WebJobs.Extensions.Http.HttpRequestManager.d__8 stateMachine) Unknown
Microsoft.Azure.WebJobs.Extensions.Http.dll!Microsoft.Azure.WebJobs.Extensions.Http.HttpRequestManager.ProcessRequestAsync(System.Net.Http.HttpRequestMessage request, System.Func> processRequestHandler, System.Threading.CancellationToken cancellationToken) Unknown
Microsoft.Azure.WebJobs.Script.WebHost.dll!Microsoft.Azure.WebJobs.Script.WebHost.Controllers.FunctionsController.ExecuteAsync(System.Web.Http.Controllers.HttpControllerContext controllerContext, System.Threading.CancellationToken cancellationToken) Unknown
mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Startd__3>(ref Microsoft.Azure.WebJobs.Script.WebHost.Controllers.FunctionsController.d__3 stateMachine) Unknown
Microsoft.Azure.WebJobs.Script.WebHost.dll!Microsoft.Azure.WebJobs.Script.WebHost.Controllers.FunctionsController.ExecuteAsync(System.Web.Http.Controllers.HttpControllerContext controllerContext, System.Threading.CancellationToken cancellationToken) Unknown
System.Web.Http.dll!System.Web.Http.Dispatcher.HttpControllerDispatcher.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) Unknown
mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Startd__1>(ref System.Web.Http.Dispatcher.HttpControllerDispatcher.d__1 stateMachine) Unknown
System.Web.Http.dll!System.Web.Http.Dispatcher.HttpControllerDispatcher.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) Unknown
System.Net.Http.dll!System.Net.Http.HttpMessageInvoker.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) Unknown
System.Web.Http.dll!System.Web.Http.Dispatcher.HttpRoutingDispatcher.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) Unknown
System.Net.Http.dll!System.Net.Http.DelegatingHandler.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) Unknown
Microsoft.Azure.WebJobs.Script.WebHost.dll!Microsoft.Azure.WebJobs.Script.WebHost.Handlers.SystemTraceHandler.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) Unknown
mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Startd__5>(ref Microsoft.Azure.WebJobs.Script.WebHost.Handlers.SystemTraceHandler.d__5 stateMachine) Unknown
Microsoft.Azure.WebJobs.Script.WebHost.dll!Microsoft.Azure.WebJobs.Script.WebHost.Handlers.SystemTraceHandler.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) Unknown
System.Net.Http.dll!System.Net.Http.DelegatingHandler.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) Unknown
Microsoft.Azure.WebJobs.Script.WebHost.dll!Microsoft.Azure.WebJobs.Script.WebHost.Handlers.WebScriptHostHandler.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) Unknown
mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Startd__6>(ref Microsoft.Azure.WebJobs.Script.WebHost.Handlers.WebScriptHostHandler.d__6 stateMachine) Unknown
Microsoft.Azure.WebJobs.Script.WebHost.dll!Microsoft.Azure.WebJobs.Script.WebHost.Handlers.WebScriptHostHandler.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) Unknown
System.Net.Http.dll!System.Net.Http.DelegatingHandler.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) Unknown
System.Web.Http.dll!System.Web.Http.HttpServer.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) Unknown
mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Startd__0>(ref System.Web.Http.HttpServer.d__0 stateMachine) Unknown
System.Web.Http.dll!System.Web.Http.HttpServer.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) Unknown
System.Net.Http.dll!System.Net.Http.HttpMessageInvoker.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) Unknown
System.Web.Http.WebHost.dll!System.Web.Http.WebHost.HttpControllerHandler.ProcessRequestAsyncCore(System.Web.HttpContextBase contextBase) Unknown
mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Startd__0>(ref System.Web.Http.WebHost.HttpControllerHandler.d__0 stateMachine) Unknown
System.Web.Http.WebHost.dll!System.Web.Http.WebHost.HttpControllerHandler.ProcessRequestAsyncCore(System.Web.HttpContextBase contextBase) Unknown
System.Web.Http.WebHost.dll!System.Web.Http.WebHost.HttpControllerHandler.ProcessRequestAsync(System.Web.HttpContext context) Unknown
System.Web.dll!System.Web.HttpTaskAsyncHandler.System.Web.IHttpAsyncHandler.BeginProcessRequest.AnonymousMethod__0() Unknown
System.Web.dll!System.Web.TaskAsyncHelper.BeginTask(System.Func taskFunc, System.AsyncCallback callback, object state) Unknown
System.Web.dll!System.Web.HttpTaskAsyncHandler.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext context, System.AsyncCallback cb, object extraData) Unknown
System.Web.dll!System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() Unknown
System.Web.dll!System.Web.HttpApplication.ExecuteStep(System.Web.HttpApplication.IExecutionStep step, ref bool completedSynchronously) Unknown
System.Web.dll!System.Web.HttpApplication.PipelineStepManager.ResumeSteps(System.Exception error) Unknown
System.Web.dll!System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext context, System.AsyncCallback cb) Unknown
System.Web.dll!System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest wr, System.Web.HttpContext context) Unknown
System.Web.dll!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(System.IntPtr rootedObjectsPointer, System.IntPtr nativeRequestContext, System.IntPtr moduleData, int flags) Unknown
System.Web.dll!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(System.IntPtr rootedObjectsPointer, System.IntPtr nativeRequestContext, System.IntPtr moduleData, int flags) Unknown
[Native to Managed Transition]
[Managed to Native Transition]
System.Web.dll!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(System.IntPtr rootedObjectsPointer, System.IntPtr nativeRequestContext, System.IntPtr moduleData, int flags) Unknown
System.Web.dll!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(System.IntPtr rootedObjectsPointer, System.IntPtr nativeRequestContext, System.IntPtr moduleData, int flags) Unknown
[Native to Managed Transition]

```

Contributor guide

Open the contributing guide

Research direction

Start at the TableExtension.GetTable entry point and its caller, BuildFromTableAttribute, then inspect the supplied deadlock call stack and memory dump; the email titled `Function monitoring page stuck` contains additional context. Done means the cause of the blocked table binding is identified and the deadlock no longer occurs in the reported function-monitoring scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.