RetryOptions error-handler in Orchestration cannot capture right exception-type from Sub-Orchestration
- Dominant language
- C#
- Stars
- 1.7k
- Forks
- 335
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 6
Description
My question is related to [436](https://github.com/Azure/durabletask/issues/436) and [807](https://github.com/Azure/durabletask/issues/807)- this is explained below:
**Problem**:
Cannot get the right exception (via InnerException or FailureDetails) in the exception handler of RetryOptions (Handle)
**Scenario**:
I have an Orchestration that calls a sub-orchestration using RetryOptions, like:
```
RetryOptions retryOptions = new RetryOptions(TimeSpan.FromMilliseconds(2500),3){
Handle = e =>
{
/*
This exception never captures MyCustomException, which was originally thrown,
either in FailureDetails or the InnerException property
*/
SubOrchestrationFailedException tfe = e as SubOrchestrationFailedException;
if (tfe != null && tfe.InnerException != null)
{
e = tfe.InnerException;
}
MyCustomException ce = e as MyCustomException;
if (ce != null)
{
LatestException = ce; //LatestException is a variable of type MyCustomException
return true;
}
return false;
}};
// IPayload is a custom type that is supposed to be returned from my sub-orchestration(if it ran successfully)
return await context.CreateSubOrchestrationInstanceWithRetry(typeof(FetchRatesSubOrchestration), retryOptions, 1);
```
The sub-orchestration's RunTask() looks like:
```
public override async Task RunTask(OrchestrationContext context, object input)
{
List rates = new List();
Reference1 reference1 = await context.ScheduleTask(typeof(TaskActivity1));
await context.ScheduleTask(typeof(TaskActivity2), reference1);
rates = await context.ScheduleTask>(typeof(TaskActivity3), reference1);
List datasReceived = rates.Select(x => x.TickerName).ToList();
List validDataPoints = _dbContext.SourceKeys.Select(t => t.SourceKeyValue).ToList();
List missingDataPoints = datasReceived.Except(validDataPoints).ToList();
if (missingDataPoints.Count() > 0)
{
_logger.LogError($"Requested data points {string.Join(",", missingDataPoints)} were not returned in the response. Retrying.");
throw new MyCustomException($"Requested data points {string.Join(",", missingDataPoints)} were not returned in the response. Retrying.");
//OrchestrationFailureException innerExceptionToThrow = new OrchestrationFailureException($"Requested data points {string.Join(",", missingDataPoints)} were not returned in the response. Retrying.");
//throw innerExceptionToThrow;
// var exc = new DurableTask.Core.Exceptions.SubOrchestrationFailedException($"Orchestration failed - {GetType().Name}", innerExceptionToThrow);
// exc.FailureDetails
// throw exc;
}
return new RatesAvailablePayload(rates);
}
```
The error thrown within the if-block in the sub-orchestration never bubbles-up to the Handle error-handler of the defined RetryOptions (within the parent orchestration).
**Additionally, what else have I tried?**
**a**. Setting the ErrorPropagationMode to ErrorPropagationMode.UseFailureDetails (or ErrorPropagationMode.SerializeExceptions) doesn't help - the FailureDetails object of the exception and the InnerException property are always null.
**b**. I have tried debugging against the source of the framework, and, I believe to have seen the "_Multiple ExecutionCompletedEvent found, potential corruption in state storage_" message from the SerMarkerEvents() method in DurableTask.Core.OrchestrationRuntimeState class - but that was during one of the times I was debugging it.
**c**. While debugging the framework source, I noticed that the FailureDetails object is populated, for the most part, but due to the parallel-execution of the code it is a little difficult to debug as the control jumps from one class to the other, and then to the third - which makes following a trail difficult, at best
**d**. I have looked at most, if not all, of the tests in the [DurableTask](https://github.com/Azure/durabletask.git) and the [DurableTask-MSSQL](https://github.com/microsoft/durabletask-mssql.git) repositories, but these haven't helped me in figuring out what I may still be missing
I'm using the following packages:
**Microsoft.DurableTask.SqlServer** - 1.1.1
**Microsoft.Azure.DurableTask.Core** - 2.10.0 (dependency brought in via Microsoft.DurableTask.SqlServer)
Please advise on how to get this resolved.
Thank you.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.