Azure / Azure/azure-functions-host
Dynamically adjust consumption scale
- Dominant language
- C#
- Stars
- 2k
- Forks
- 482
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 38
Description
#### What problem would the feature you're requesting solve? Please describe.
When targeting an endpoint that is not infinitely scalable, it is difficult to determine the correct scale values to use and ideal scale values may change over time.
Consider these scenarios:
1. Assume you are hitting a service endpoint and are assuming you have XX% of the service's resources to work with. You set the scale settings appropriately and move on. Later, someone performs a large job that eats at your assumed resources and the Function app starts experiencing a high number of failures. It keeps trying and failing, and, eventually, the host gives up and lots of triggers never get processed.
2. Assume two Functions in a single Function app that are hitting the same endpoint. Traffic is low so the scale setting is working fine. Traffic starts increasing and the Function app starts experiencing a high number of failures. It keeps trying and failing, and, eventually, the host gives up and lots of triggers never get processed.
3. Assume a Function app's scale settings were set by assuming a particular tier at the endpoint service. Traffic starts to increase and the business can now justify a higher tier endpoint service. The endpoint service is scaled up, but the Function app can't dynamically scale with it. The Function app ends up with a giant backlog, because the Function app isn't working fast enough now. Once this is diagnosed, the consumption app can't be scaled up as simply as the endpoint service because a new commit to source control and a new deployment will be necessary to adjust the Function app settings.
#### Describe the solution you'd like
Function app can return a TimeoutException that will cause the host to scale back the number of instances running.
#### Describe alternatives you've considered
1. Host detects that function instances are taking longer than usual and scales down.
2. Function app sets a timeout value for each function. If the timeout value is exceeded, host should scale down the number of instances. Maybe
3. Triggers never stop occurring, even after X errors (maybe only for certain exception types).
4. The Function app is in control of the trigger retry strategy, allowing exponential or random retry periods.
#### Additional context
This is related to https://github.com/Azure/azure-functions-host/issues/4301.
Contributor guide
Research direction
Start by reading the related issue #4301 and compare its context with the three scaling scenarios and proposed alternatives here. This issue needs an agreed design and acceptance criteria for how the host should react to timeouts, failures, or changing endpoint capacity before implementation can be considered done.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp
- Domain
- backend, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100