Azure / Azure/azure-functions-host

Restart unresponsive workers during WorkerStatus heartbeat request

Open
#8,417 2 comments 0 reactions 0 assignees View on GitHub
design
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 12h
Merged PRs (30d)
38

Description

The [HostPerformanceManager](https://github.com/Azure/azure-functions-host/blob/dev/src/WebJobs.Script/Scale/HostPerformanceManager.cs#L115) starts a timer which periodically makes a `GetWorkerStatusAsync` request. The worker status here is used to determine if workers are overloaded and if we need to start a new worker channel. This is determined through latency, if the worker is taking too long to process grpc requests then it's overloaded/unhealthy.

We want to add to this feature by restarting unresponsive workers, currently we might only be able to do this to multi-threaded workers (dotnet and java)

Contributor guide

Open the contributing guide

Research direction

Start with src/WebJobs.Script/Scale/HostPerformanceManager.cs around the timer and its GetWorkerStatusAsync request. Trace how latency identifies overloaded or unhealthy workers and how new worker channels are started. Done means defining and implementing the restart behavior for unresponsive workers, initially considering multi-threaded .NET and Java workers; the issue does not name tests or a more specific entry point.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.