Azure / Azure/azure-functions-host

Storage errors causes the runtime to crash

Open
#6,385 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 10h
Merged PRs (30d)
36

Description

While investigating an incident where the network had very high timeouts when connecting to azure storage (3 minute timeout) The runtime process enters a crash loop with the following exception

```
fail: Microsoft.Azure.WebJobs.Script.WebHost.WebScriptHostExceptionHandler[0]
      An unhandled exception has occurred. Host is shutting down.
Microsoft.WindowsAzure.Storage.StorageException: This request is not authorized to perform this operation.
   at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)
   at Microsoft.WindowsAzure.Storage.Blob.CloudBlobContainer.ListBlobsSegmentedAsync(String prefix, Boolean useFlatBlobListing, BlobListingDetails blobListingDetails, Nullable`1 maxResults, BlobContinuationToken currentToken, BlobRequestOptions options, OperationContext operationContext, CancellationToken cancellationToken)
   at Microsoft.Azure.WebJobs.Extensions.Storage.TimeoutHandler.ExecuteWithTimeout[T](String operationName, String clientRequestId, IWebJobsExceptionHandler exceptionHandler, Func`1 operation)
   at Microsoft.Azure.WebJobs.Host.Blobs.Listeners.CloudBlobClientExtensions.ListBlobsAsync(CloudBlobClient client, String prefix, Boolean useFlatBlobListing, BlobListingDetails blobListingDetails, String operationName, IWebJobsExceptionHandler exceptionHandler, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Extensions.Storage\Blobs\Listeners\CloudBlobClientExtensions.cs:line 33
   at Microsoft.Azure.WebJobs.Host.Blobs.Listeners.BlobLogListener.GetLogsWithPrefixAsync(List`1 selectedLogs, CloudBlobClient blobClient, String prefix, IWebJobsExceptionHandler exceptionHandler, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Extensions.Storage\Blobs\Listeners\BlobLogListener.cs:line 171
   at Microsoft.Azure.WebJobs.Host.Blobs.Listeners.BlobLogListener.ListRecentLogFilesAsync(CloudBlobClient blobClient, DateTime startTimeForSearch, CancellationToken cancellationToken, Int32 hoursWindow, IWebJobsExceptionHandler exceptionHandler) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Extensions.Storage\Blobs\Listeners\BlobLogListener.cs:line 159
   at Microsoft.Azure.WebJobs.Host.Blobs.Listeners.BlobLogListener.GetRecentBlobWritesAsync(CancellationToken cancellationToken, Int32 hoursWindow) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Extensions.Storage\Blobs\Listeners\BlobLogListener.cs:line 60
   at Microsoft.Azure.WebJobs.Host.Blobs.Listeners.PollLogsStrategy.ExecuteAsync(CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Extensions.Storage\Blobs\Listeners\PollLogsStrategy.cs:line 115
   at Microsoft.Azure.WebJobs.Host.Blobs.Listeners.ScanBlobScanLogHybridPollingStrategy.ExecuteAsync(CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Extensions.Storage\Blobs\Listeners\ScanBlobScanLogHybridPollingStrategy.cs:line 131
   at Microsoft.Azure.WebJobs.Host.Timers.TaskSeriesTimer.RunAsync(CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Timers\TaskSeriesTimer.cs:line 147
Request Information
RequestID:00000000-0000-0000-0000-000000000000
RequestDate:Sat,00 Jul 0000 00:00:00 GMT
StatusMessage:This request is not authorized to perform this operation.
ErrorCode:AuthorizationFailure
ErrorMessage:This request is not authorized to perform this operation.
RequestId:00000000-0000-0000-0000-000000000000
Time:0000-00-00T00:00:00.0000000Z
```

Contributor guide

Open the contributing guide

Research direction

Start with CloudBlobClientExtensions.cs, BlobLogListener.cs, PollLogsStrategy.cs, and ScanBlobScanLogHybridPollingStrategy.cs at the stack-trace entry points, then trace how authorization failures and timeouts propagate through the logging poller. Reproduce or test the storage failure if existing coverage allows; done means the host does not enter a crash loop when blob log access fails.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.