Azure / Azure/azure-functions-host

Improve logging on Host startup and shutdown host if start up exceeds a threshold

Open
#5,445 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 12h
Merged PRs (30d)
38

Description

This is follow up item from CRI investigation. When new function runtime `2.0.12888.0` rolled out, customer's application restarted but Jobhost took ~16 minutes to start causing all inflight http requests to timeout.

```
 PreciseTimeStamp Source Summary
2019-12-15 03:18:31.6108487 Host.Startup Host started (984193ms)
```

Logs do not have any details on where the startup got stuck.

Only exception in the logs:

```
The listener for function 'Functions.HopperHubTrigger' was unable to start.
System.OperationCanceledException : The operation was canceled.
at System.Net.Http.HttpClient.HandleFinishSendAsyncError(Exception e,CancellationTokenSource cts)
at async System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask,HttpRequestMessage request,CancellationTokenSource cts,Boolean disposeCts)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand`1 cmd,IRetryPolicy policy,OperationContext operationContext,CancellationToken token)
```

Contributor guide

Open the contributing guide

Research direction

Start by tracing the Host.Startup logging and the listener startup path described in the issue, including the HopperHubTrigger failure. Review the existing startup duration log and cancellation behavior; done means startup progress is diagnosable in logs and an over-threshold startup causes the host to shut down.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
backend, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.