Azure / Azure/azure-functions-host

CPU-bound tasks seem to starve network-bound tasks

Open
#6,983 20 comments 5 reactions 2 assignees Claimed by @v-bbalaiagar View on GitHub
needs-investigation question
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 10h
Merged PRs (30d)
36

Description

#### Investigative information

Please provide the following:

- Timestamp: 12/12/20 2:56:49 PM ET
- Function App version: 3
- Function App name: NpgsqlHeavyCPU
- Function name(s) (as appropriate): NpgsqlOrchestration
- Invocation ID: 9948a45e-dcf9-410f-a167-ea81f2dc4682
- Region: East US 2

#### Repro steps

I created a sample project to demonstrate the problem: https://github.com/Methuselah96/NpgsqlHeavyCpu

The sample only shows Npgsql, but I've also seen issues with Cosmos and Blob Storage as well.

#### Expected behavior

The Azure Function to complete without failure.

#### Actual behavior

Various socket-related errors:
Npgsql:
```
Exception message: Npgsql.NpgsqlException (0x80004005): Exception while writing to stream
Stack trace:
---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..

---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.

--- End of inner exception stack trace ---

at Npgsql.NpgsqlWriteBuffer.Flush(Boolean async)

at Npgsql.NpgsqlWriteBuffer.Flush(Boolean async)

at Npgsql.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)

at Npgsql.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)

at Npgsql.ConnectorPool.AllocateLong(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)

at Npgsql.NpgsqlConnection.c__DisplayClass32_0.g__OpenLong|0>d.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at Dapper.SqlMapper.QueryRowAsync[T](IDbConnection cnn, Row row, Type effectiveType, CommandDefinition command) in /_/Dapper/SqlMapper.Async.cs:line 482
```

Npgsql:
```
Exception message: System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
Stack trace:
---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..

---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.

--- End of inner exception stack trace ---

---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.

--- End of inner exception stack trace ---

at System.Net.FixedSizeReader.ReadPacketAsync(Stream transport, AsyncProtocolRequest request)

at System.Net.Security.SslStream.ThrowIfExceptional()

at System.Net.Security.SslStream.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)

at System.Net.Security.SslStream.EndProcessAuthentication(IAsyncResult result)

at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult)

at System.Net.Security.SslStream.c.b__64_2(IAsyncResult iar)

at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)

--- End of stack trace from previous location where exception was thrown ---

at Npgsql.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)

at Npgsql.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)

at Npgsql.ConnectorPool.AllocateLong(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)

at Npgsql.NpgsqlConnection.c__DisplayClass32_0.g__OpenLong|0>d.MoveNext()
```

Cosmos:
```
System.Net.Http.HttpRequestException: An attempt was made to access a socket in a way forbidden by its access permissions.

---> System.Net.Sockets.SocketException (10013): An attempt was made to access a socket in a way forbidden by its access permissions.

at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)

--- End of inner exception stack trace ---

at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)

at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)

at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)

at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)

at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)

at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)

at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)

at Microsoft.Azure.Cosmos.DocumentClient.HttpRequestMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)

at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)

at Microsoft.Azure.Cosmos.GatewayAccountReader.GetDatabaseAccountAsync(Uri serviceEndpoint)

at Microsoft.Azure.Cosmos.Routing.GlobalEndpointManager.GetDatabaseAccountFromAnyLocationsAsync(Uri defaultEndpoint, IList`1 locations, Func`2 getDatabaseAccountFn)

at Microsoft.Azure.Cosmos.GatewayAccountReader.InitializeReaderAsync()

at Microsoft.Azure.Cosmos.CosmosAccountServiceConfiguration.InitializeAsync()

at Microsoft.Azure.Cosmos.DocumentClient.InitializeGatewayConfigurationReaderAsync()

at Microsoft.Azure.Cosmos.DocumentClient.GetInitializationTaskAsync(IStoreClientFactory storeClientFactory)

at Microsoft.Azure.Cosmos.DocumentClient.EnsureValidClientAsync()

at Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler.EnsureValidClientAsync(RequestMessage request)

at Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken)

at Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler.SendAsync(Uri resourceUri, ResourceType resourceType, OperationType operationType, RequestOptions requestOptions, ContainerCore cosmosContainerCore, Nullable`1 partitionKey, Stream streamPayload, Action`1 requestEnricher, CosmosDiagnosticsContext diagnosticsContext, CancellationToken cancellationToken)

at Microsoft.Azure.Cosmos.CosmosClient.c__DisplayClass27_0.b__0>d.MoveNext()
```

Blob storage:
```
Microsoft.WindowsAzure.Storage.StorageException: An attempt was made to access a socket in a way forbidden by its access permissions.

---> System.Net.Http.HttpRequestException: An attempt was made to access a socket in a way forbidden by its access permissions.

---> System.Net.Sockets.SocketException (10013): An attempt was made to access a socket in a way forbidden by its access permissions.

at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)

--- End of inner exception stack trace ---

at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)

at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)

at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)

at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)

at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)

at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)

at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)

at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)

at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)
```
#### Known workarounds

For Npgsql, if I opened all the connections before running the CPU-bound code, then I wouldn't have any errors. I haven't found any workarounds for Cosmos or Blob storage yet.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.