microsoft / microsoft/durabletask-netherite

NetheriteOrchestrationService.StartAsync throws cached exception on retry if first call fails

Open
#352 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug P1
Dominant language
C#
Stars
239
Forks
37
Avg merge
1d 10h
Merged PRs (30d)
2

Description

The following was observed in a customer support case.

  1. The function app starts up and Netherite throws an exception because of a transient DNS issue when connecting to Event Hubs.
  2. The Functions host handles the failure, and tries to restart the host startup process.
  3. NetheriteOrchestrationService.StartAsync gets called again with the same NetheriteOrchestrationService object instance, but the same exception gets thrown. This time, however, we don't log it. The DNS error only shows up in the FunctionsLogs - not the DurableFunctionsEvents log, indicating that we're not actually running the startup logic a second time.
  4. This continues indefinitely.
  5. The problem stops only when the customer forcefully restarts the function app

As discussed, this appears to be a caching issue in NetheriteOrchestrationService. The Azure Functions Host assumes that we can retry startup failures, which is why we see this behavior in production. However, NetheriteOrchestrationService.StartAsync doesn't appear to support retrying.

In order to be compatible with the Azure Functions host retry logic, we should change this behavior.

Context from @sebastianburckhardt

...this breaks down into two internal tasks (for starting client and for starting workers) which are being cached. I believe the reason for this caching is to deal with early and/or concurrent client calls and make exceptions in the various startup tasks visible to the application. I agree this is complicated and I would love something simpler but it has been tricky last time I tried because the environment does not follow what would be easy to implement (wait for successful StartAsync before calling any client operations)

It would probably not be all that difficult to just clear all the cached failed transitions when StartAsync() is retried

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at NetheriteOrchestrationService.StartAsync and trace how the cached client and worker startup tasks handle exceptions. Reproduce a transient Event Hubs connection failure followed by a retry on the same service instance; done means the retry runs startup logic again instead of returning the cached exception, with the failure visible in the expected logs.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.