Azure / Azure/azure-functions-host

Improved telemetry for worker initialization timeouts

Open
#10,745 0 comments 0 reactions 0 assignees View on GitHub
Logging and Metrics supportability
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 10h
Merged PRs (30d)
36

Description

It is difficult to pin down which methods or tasks are timing out during worker initialization. One example is that during a recent investigation, a timeout likely related to the wait between `WorkerInitRequest` and `WorkerInitResponse` bubbled up into a timeout error pointing at [this line](https://github.com/Azure/azure-functions-host/blob/969f76134a60c1b5bb13b77d56169bf324a8f1c6/src/WebJobs.Script/Host/WorkerFunctionMetadataProvider.cs#L115).

```
System.AggregateException : One or more errors occurred. (The operation has timed out.) ---> The operation has timed out.
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Script.WorkerFunctionMetadataProvider.GetFunctionMetadataAsync(IEnumerable`1 workerConfigs,Boolean forceRefresh) at /src/azure-functions-host/src/WebJobs.Script/Host/WorkerFunctionMetadataProvider.cs : 115 ---> (Inner Exception #0) System.TimeoutException : The operation has timed out.
at Microsoft.Azure.WebJobs.Script.Grpc.GrpcWorkerChannel.PendingItem.OnTimeout() at /src/azure-functions-host/src/WebJobs.Script.Grpc/Channel/GrpcWorkerChannel.cs : 1799
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Script.Grpc.GrpcWorkerChannel.StartWorkerProcessAsync(CancellationToken cancellationToken) at /src/azure-functions-host/src/WebJobs.Script.Grpc/Channel/GrpcWorkerChannel.cs : 379<---
```

Potential next steps:

- Review of worker initialization timeout flows
- Review of the logging in this flow
- Design proposal needed for improvements

Contributor guide

Open the contributing guide

Research direction

Start with WebJobs.Script/Host/WorkerFunctionMetadataProvider.cs around line 115 and WebJobs.Script.Grpc/Channel/GrpcWorkerChannel.cs around lines 1799 and 379. Trace the worker initialization timeout flow between WorkerInitRequest and WorkerInitResponse, then review the related logging. Done should be defined by a design proposal that identifies which method or task timed out and improves telemetry for the flow.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, grpc
Domain
backend, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.