Azure / Azure/azure-functions-host
Durable function - Unhandled exception while executing task: System.ObjectDisposedException: Cannot access a disposed object. Object name: 'LoggerFactory'
- Dominant language
- C#
- Stars
- 2k
- Forks
- 482
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 36
Description
Durable function fails because of `System.ObjectDisposedException` for `LoggerFactory` in an activity function. Potentially related to #1690.
#### Investigative information
Please provide the following:
- Timestamp:
- Function App version (1.0 or 2.0): 2.0
- Function name(s): `CosmosReprocessProcedureRuns` invoking `CosmosReprocessProcedureRuns_ProcessRun`
- Invocation ID: 4b7d91ec-eace-47f5-b3c0-62d3effde5e6
- Region: West Europe
#### Stack trace
```
Exception while executing function: CosmosReprocessProcedureRuns Orchestrator function 'CosmosReprocessProcedureRuns' failed: The activity function 'CosmosReprocessProcedureRuns_ProcessRun' failed: "Failed to deserialize exception from TaskActivity: Unhandled exception while executing task: System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'LoggerFactory'.
at Microsoft.Extensions.Logging.LoggerFactory.CreateLogger(String categoryName)
at Microsoft.Azure.WebJobs.Script.WebHost.ScriptLoggerFactory.CreateLogger(String categoryName) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Diagnostics\ScriptLoggerFactory.cs:line 26
at Microsoft.Azure.WebJobs.Host.Loggers.FunctionInstanceLogger.Log(LogLevel level, FunctionDescriptor descriptor, Guid functionId, String message, Exception exception) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Loggers\FunctionInstanceLogger.cs:line 68
at Microsoft.Azure.WebJobs.Host.Loggers.FunctionInstanceLogger.LogFunctionCompletedAsync(FunctionCompletedMessage message, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Loggers\FunctionInstanceLogger.cs:line 55
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsyncCore(IFunctionInstanceEx functionInstance, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 158
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(IFunctionInstance instance, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 87
at Microsoft.Azure.WebJobs.Host.Executors.TriggeredFunctionExecutor`1.TryExecuteAsync(TriggeredFunctionData input, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\TriggeredFunctionExecutor.cs:line 56
at Microsoft.Azure.WebJobs.Extensions.DurableTask.TaskActivityShim.RunAsync(TaskContext context, String rawInput) in C:\GitHub\azure-functions-durable-extension\src\WebJobs.Extensions.DurableTask\Listener\TaskActivityShim.cs:line 57
at DurableTask.Core.TaskActivityDispatcher.<>c__DisplayClass12_1.<b__1>d.MoveNext() in C:\source\durabletask\src\DurableTask.Core\TaskActivityDispatcher.cs:line 135
at Microsoft.Extensions.Logging.LoggerFactory.CreateLogger(String categoryName)
at Microsoft.Azure.WebJobs.Script.WebHost.ScriptLoggerFactory.CreateLogger(String categoryName) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Diagnostics\ScriptLoggerFactory.cs:line 26
at Microsoft.Azure.WebJobs.Host.Loggers.FunctionInstanceLogger.Log(LogLevel level, FunctionDescriptor descriptor, Guid functionId, String message, Exception exception) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Loggers\FunctionInstanceLogger.cs:line 68
at Microsoft.Azure.WebJobs.Host.Loggers.FunctionInstanceLogger.LogFunctionCompletedAsync(FunctionCompletedMessage message, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Loggers\FunctionInstanceLogger.cs:line 55
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsyncCore(IFunctionInstanceEx functionInstance, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 158
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(IFunctionInstance instance, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 87
at Microsoft.Azure.WebJobs.Host.Executors.TriggeredFunctionExecutor`1.TryExecuteAsync(TriggeredFunctionData input, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\TriggeredFunctionExecutor.cs:line 56
at Microsoft.Azure.WebJobs.Extensions.DurableTask.TaskActivityShim.RunAsync(TaskContext context, String rawInput) in C:\GitHub\azure-functions-durable-extension\src\WebJobs.Extensions.DurableTask\Listener\TaskActivityShim.cs:line 57
at DurableTask.Core.TaskActivityDispatcher.<>c__DisplayClass12_1.<b__1>d.MoveNext() in C:\source\durabletask\src\DurableTask.Core\TaskActivityDispatcher.cs:line 135". See the function execution logs for additional details.
```
#### Related information
* Programming language used: C#
* Bindings used:
* Durable function (triggered by a `QueueTrigger`)
* Calls activity function, which has two `CosmosDb` bindings (one `DocumentClient` and one `IAsyncCollector`)
Has happened at least three times and the durable function execution (within the `CosmosReprocessProcedureRuns_ProcessRun` activity function) also contains multiple exceptions for connecting to Cosmos:
```
A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
```
```
System.Net.Http.HttpRequestException:
at System.Net.Http.ConnectHelper+d__2.MoveNext (System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Threading.Tasks.ValueTask`1.get_Result (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1+ConfiguredValueTaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.HttpConnectionPool+d__44.MoveNext (System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Threading.Tasks.ValueTask`1.get_Result (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1+ConfiguredValueTaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.HttpConnectionPool+d__49.MoveNext (System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Threading.Tasks.ValueTask`1.get_Result (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1+ConfiguredValueTaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.HttpConnectionPool+d__39.MoveNext (System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.RedirectHandler+d__4.MoveNext (System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.DiagnosticsHandler+d__2.MoveNext (System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
Inner exception System.Net.Sockets.SocketException handled at System.Net.Http.ConnectHelper+d__2.MoveNext:
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.ConnectHelper+d__2.MoveNext (System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
```
Contributor guide
Research direction
Start with ScriptLoggerFactory.cs and trace the LoggerFactory lifecycle through FunctionInstanceLogger, FunctionExecutor, and DurableTask TaskActivityShim.RunAsync. Reproduce the CosmosReprocessProcedureRuns activity failure path and determine why logging reaches a disposed LoggerFactory; done means the reported activity failure no longer produces this ObjectDisposedException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp
- Domain
- backend, cloud, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100