Azure / Azure/durabletask

Handling of OutOfMemoryException in work items

Open
#629 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
1.7k
Forks
335
Avg merge
2d 23h
Merged PRs (30d)
6

Description

The implementation seems to be a bit "of two minds" when it comes to handling OutOfMemoryException thrown by user work items (activities or orchestrations).

Based on common sense and guidance (https://docs.microsoft.com/en-us/dotnet/api/system.outofmemoryexception?redirectedfrom=MSDN&view=net-5.0) the only reasonable thing to do inside a handler that catches OutOfMemoryException is to do some quick logging (without allocating objects in the process) and then failing fast.

This is not quite what the code seems to do in all cases:
- For activity work items, the exception is not caught and will terminate the threadpool task allocated in WorkItemDispatcher.cs:334. That seems fine to me. The result would be that the activity is retried at some later point of time, since it never acknowledges the processing of the work item in storage.
- For orchestration work items, the exception is caught and `this.context.FailOrchestration(exception)` is invoked. This seems like it has unpredictable results, as it could sometimes lead to a retry (if more OutOfMemoryException are thrown later in that code path, which is quite likely) and sometimes lead to the orchestration being terminated with an OutOfMemoryResult (if the storage access completes). I think we should fix this and handle it the same way as for activities - fail fast.

It would be a very good idea to create tests that inject OutOfMemoryErrors to validate that the implementation behaves as designed.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.