Azure / Azure/azure-functions-host
Storage errors causes the runtime to crash
- Dominant language
- C#
- Stars
- 2k
- Forks
- 482
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 36
Description
While investigating an incident where the network had very high timeouts when connecting to azure storage (3 minute timeout) The runtime process enters a crash loop with the following exception
```
fail: Microsoft.Azure.WebJobs.Script.WebHost.WebScriptHostExceptionHandler[0]
An unhandled exception has occurred. Host is shutting down.
Microsoft.WindowsAzure.Storage.StorageException: This request is not authorized to perform this operation.
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)
at Microsoft.WindowsAzure.Storage.Blob.CloudBlobContainer.ListBlobsSegmentedAsync(String prefix, Boolean useFlatBlobListing, BlobListingDetails blobListingDetails, Nullable`1 maxResults, BlobContinuationToken currentToken, BlobRequestOptions options, OperationContext operationContext, CancellationToken cancellationToken)
at Microsoft.Azure.WebJobs.Extensions.Storage.TimeoutHandler.ExecuteWithTimeout[T](String operationName, String clientRequestId, IWebJobsExceptionHandler exceptionHandler, Func`1 operation)
at Microsoft.Azure.WebJobs.Host.Blobs.Listeners.CloudBlobClientExtensions.ListBlobsAsync(CloudBlobClient client, String prefix, Boolean useFlatBlobListing, BlobListingDetails blobListingDetails, String operationName, IWebJobsExceptionHandler exceptionHandler, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Extensions.Storage\Blobs\Listeners\CloudBlobClientExtensions.cs:line 33
at Microsoft.Azure.WebJobs.Host.Blobs.Listeners.BlobLogListener.GetLogsWithPrefixAsync(List`1 selectedLogs, CloudBlobClient blobClient, String prefix, IWebJobsExceptionHandler exceptionHandler, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Extensions.Storage\Blobs\Listeners\BlobLogListener.cs:line 171
at Microsoft.Azure.WebJobs.Host.Blobs.Listeners.BlobLogListener.ListRecentLogFilesAsync(CloudBlobClient blobClient, DateTime startTimeForSearch, CancellationToken cancellationToken, Int32 hoursWindow, IWebJobsExceptionHandler exceptionHandler) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Extensions.Storage\Blobs\Listeners\BlobLogListener.cs:line 159
at Microsoft.Azure.WebJobs.Host.Blobs.Listeners.BlobLogListener.GetRecentBlobWritesAsync(CancellationToken cancellationToken, Int32 hoursWindow) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Extensions.Storage\Blobs\Listeners\BlobLogListener.cs:line 60
at Microsoft.Azure.WebJobs.Host.Blobs.Listeners.PollLogsStrategy.ExecuteAsync(CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Extensions.Storage\Blobs\Listeners\PollLogsStrategy.cs:line 115
at Microsoft.Azure.WebJobs.Host.Blobs.Listeners.ScanBlobScanLogHybridPollingStrategy.ExecuteAsync(CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Extensions.Storage\Blobs\Listeners\ScanBlobScanLogHybridPollingStrategy.cs:line 131
at Microsoft.Azure.WebJobs.Host.Timers.TaskSeriesTimer.RunAsync(CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Timers\TaskSeriesTimer.cs:line 147
Request Information
RequestID:00000000-0000-0000-0000-000000000000
RequestDate:Sat,00 Jul 0000 00:00:00 GMT
StatusMessage:This request is not authorized to perform this operation.
ErrorCode:AuthorizationFailure
ErrorMessage:This request is not authorized to perform this operation.
RequestId:00000000-0000-0000-0000-000000000000
Time:0000-00-00T00:00:00.0000000Z
```
Contributor guide
Research direction
Start with CloudBlobClientExtensions.cs, BlobLogListener.cs, PollLogsStrategy.cs, and ScanBlobScanLogHybridPollingStrategy.cs at the stack-trace entry points, then trace how authorization failures and timeouts propagate through the logging poller. Reproduce or test the storage failure if existing coverage allows; done means the host does not enter a crash loop when blob log access fails.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100