Azure / Azure/azure-functions-host

Improve host retry behaviour

Open
#9,924 0 comments 1 reaction 0 assignees View on GitHub
area: configuration
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 10h
Merged PRs (30d)
36

Description

As part of some of the flex work happening, we recently made a change in the host to capture an "AppFailure" count when there is a "permanent" app failure in the host - this typically ends up happening when we see an exception in the host startup:

https://github.com/Azure/azure-functions-host/blob/726c20c29b3433097226f3742eb9b3297f171e6e/src/WebJobs.Script.WebHost/WebJobsScriptHostService.cs#L399-L423

But when this kind of failure occurs, we retry to start the host. The host as an indefinite exponential retry (up to a max of 2 minutes), this was required as the platform couldn't react efficiently to unhealthy instances. However today, we have a different platform that is aware of the permanent app failures and is able to manage instances better than before. As Fabio mentioned, "we should have behavior consistent with the signals we emit. If we state we're in a permanent failure state, we our logic should probably match that."

**Goal:**

We should look into what the platform is capable of today in terms of instance management, and make changes to how the host retries:

1. Should we cap out retries to a specific number?
- i.e. If we retry 5 times and we are still not able to restart the host, emit a permanent failure count here then stop retrying
2. Should we make the retry threshold configurable?

To help support any design decisions made here, we should look at logs to see how often we see the host coming back after 3 attempts vs how often it retries indefinitely.

Contributor guide

Open the contributing guide

Research direction

Start with src/WebJobs.Script.WebHost/WebJobsScriptHostService.cs at lines 399-423 and trace the host-startup retry path that records AppFailure. Review available platform instance-management behavior and logs showing recovery after three attempts versus indefinite retries. Done means an agreed retry threshold or configurability decision is implemented consistently with the permanent-failure signal and supported by the observed logs.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.