Azure / Azure/azure-functions-host
Improved telemetry for worker initialization timeouts
- Dominant language
- C#
- Stars
- 2k
- Forks
- 482
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 36
Description
It is difficult to pin down which methods or tasks are timing out during worker initialization. One example is that during a recent investigation, a timeout likely related to the wait between `WorkerInitRequest` and `WorkerInitResponse` bubbled up into a timeout error pointing at [this line](https://github.com/Azure/azure-functions-host/blob/969f76134a60c1b5bb13b77d56169bf324a8f1c6/src/WebJobs.Script/Host/WorkerFunctionMetadataProvider.cs#L115).
```
System.AggregateException : One or more errors occurred. (The operation has timed out.) ---> The operation has timed out.
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Script.WorkerFunctionMetadataProvider.GetFunctionMetadataAsync(IEnumerable`1 workerConfigs,Boolean forceRefresh) at /src/azure-functions-host/src/WebJobs.Script/Host/WorkerFunctionMetadataProvider.cs : 115 ---> (Inner Exception #0) System.TimeoutException : The operation has timed out.
at Microsoft.Azure.WebJobs.Script.Grpc.GrpcWorkerChannel.PendingItem.OnTimeout() at /src/azure-functions-host/src/WebJobs.Script.Grpc/Channel/GrpcWorkerChannel.cs : 1799
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Script.Grpc.GrpcWorkerChannel.StartWorkerProcessAsync(CancellationToken cancellationToken) at /src/azure-functions-host/src/WebJobs.Script.Grpc/Channel/GrpcWorkerChannel.cs : 379<---
```
Potential next steps:
- Review of worker initialization timeout flows
- Review of the logging in this flow
- Design proposal needed for improvements
Contributor guide
Research direction
Start with WebJobs.Script/Host/WorkerFunctionMetadataProvider.cs around line 115 and WebJobs.Script.Grpc/Channel/GrpcWorkerChannel.cs around lines 1799 and 379. Trace the worker initialization timeout flow between WorkerInitRequest and WorkerInitResponse, then review the related logging. Done should be defined by a design proposal that identifies which method or task timed out and improves telemetry for the flow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, grpc
- Domain
- backend, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100