The system lacked sufficient buffer space or because a queue was full
- Dominant language
- PowerShell
- Stars
- 1.1k
- Forks
- 215
- Avg merge
- 4h 2m
- Merged PRs (30d)
- 1
Description
Summary: **All of the Azure Functions under a shared App Service plan suddenly started failing in production with error: "An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full"**
@tohling - opening a bug similar to #1112 . We are a team inside Microsoft facing issues **in production**, so any timely help on this would be most appreciated. We already have an open support ticket number. for reference: 119021321000198
**Azure Functions version**: v2 (.NET core)
**App Service plan**: FastPath-prod-east-us
**Affected Function Apps**: All of them, but if you want a specific example, use FastPathIceHockey-prod-east-us, function: NHL-Game-GameEndNotification
We already do follow the best practices in using static clients when making outbound connections, as described here: https://github.com/Azure/azure-functions-host/wiki/Managing-Connections
Here are some examples on how we define the clients:
In each function we define static clients like so:
```c#
private static BlobStorage blobStorageClient;
private static CosmosDbClient cosmosDbClient;
private static HttpClient httpClient;
```
Then inside **static constructors** for **each** function we make instances:
```c#
blobStorageClient = new BlobStorage(config.BlobStorageConnectionString);
cosmosDbClient = new CosmosDbClient(endpointUrl: config.CosmosDbEndpointUrl, primaryKey: config.CosmosDbPrimaryKey);
var clientHandler = new HttpClientHandler();
httpClient = new HttpClient(clientHandler);
```
Is there anything else we can do to mitigate this? Having a bug suddenly take down all of our app functions sounds like a P0 bug to me.
You have our permission to poke around all the logs etc. You can find the exception in Application Insights. Please feel free to bring in more folks to look into this. I will also separately send you an e-mail with our entire source code.
Thanks!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.