grpc / grpc/grpc-dotnet

Getting intermittent errors when performing parallel gRPC calls.

Open
#2,289 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
C#
Stars
4.5k
Forks
836
Avg merge
6d 3h
Merged PRs (30d)
7

Description

### What version of gRPC and what language are you using?
C#

### What operating system (Linux, Windows,...) and version?
Client: Windows Server 2012, Server: Docker Linux container

### What runtime / compiler are you using (e.g. .NET Core SDK version `dotnet --info`)
client: .Net Framework 4.6 Grpc.Core
server: .NET 7

### What did you do?
I have multiple parallel gRPC calls.
Quite often (but not always) I see an error for some of the calls:
```
{
Type: 'System.IO.IOException',
Data: {
ActivityId: '00-2f69e7db02bab0398d5790f160790820-4a685c674cb0fe0b-01'
},
HResult: -2146232800,
Message: 'The request stream was aborted.',
Source: 'System.Private.CoreLib',
TargetSite: 'Void Throw()',
InnerException: {
HResult: -2146233029,
Message: 'The HTTP/2 connection faulted.',
Source: null,
CancellationToken: {
IsCancellationRequested: false,
CanBeCanceled: false,
WaitHandle: {
Handle: { },
SafeWaitHandle: {
IsInvalid: false,
IsClosed: false
}
}
},
Type: 'Microsoft.AspNetCore.Connections.ConnectionAbortedException'
}
}
```

With stack trace:
> System.IO.IOException: The request stream was aborted.
---> Microsoft.AspNetCore.Connections.ConnectionAbortedException: The HTTP/2 connection faulted.
--- End of inner exception stack trace ---
at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
at System.IO.Pipelines.Pipe.GetReadAsyncResult()
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2MessageBody.ReadAsync(CancellationToken cancellationToken)
at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at Grpc.AspNetCore.Server.Internal.PipeExtensions.ReadStreamMessageAsync[T](PipeReader input, HttpContextServerCallContext serverCallContext, Func`2 deserializer, CancellationToken cancellationToken)
at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at Grpc.AspNetCore.Server.Internal.HttpContextStreamReader`1.g__MoveNextAsync|11_0(ValueTask`1 readStreamTask)
at Grpc.Core.AsyncStreamReaderExtensions.ReadAllAsyncCore[T](IAsyncStreamReader`1 streamReader, CancellationToken cancellationToken)+MoveNext()
at Grpc.Core.AsyncStreamReaderExtensions.ReadAllAsyncCore[T](IAsyncStreamReader`1 streamReader, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource.GetResult()
at DataProvider.Service.Endpoint.RadioDataProvider.RespondWithEntitiesHavingIdsFromStream[TSource,TMapped](IAsyncStreamReader`1 requestStream, IAsyncStreamWriter`1 responseStream, Task`1 availableDataTask, Func`3 mapper, CancellationToken cancellationToken) in /src/src/DataProvider.Service.Endpoint/RadioDataProvider.cs:line 473
at DataProvider.Service.Endpoint.RadioDataProvider.RespondWithEntitiesHavingIdsFromStream[TSource,TMapped](IAsyncStreamReader`1 requestStream, IAsyncStreamWriter`1 responseStream, Task`1 availableDataTask, Func`3 mapper, CancellationToken cancellationToken) in /src/src/DataProvider.Service.Endpoint/RadioDataProvider.cs:line 473
at Grpc.Shared.Server.DuplexStreamingServerMethodInvoker`3.ResolvedInterceptorInvoker(IAsyncStreamReader`1 requestStream, IServerStreamWriter`1 responseStream, ServerCallContext resolvedContext)
at Grpc.Shared.Server.DuplexStreamingServerMethodInvoker`3.ResolvedInterceptorInvoker(IAsyncStreamReader`1 requestStream, IServerStreamWriter`1 responseStream, ServerCallContext resolvedContext)
at CalculationEngine.Shared.Infrastructure.ClientInitiatedCancellationInterceptor.DuplexStreamingServerHandler[TRequest,TResponse](IAsyncStreamReader`1 requestStream, IServerStreamWriter`1 responseStream, ServerCallContext context, DuplexStreamingServerMethod`2 continuation) in /src/src/CalculationEngine.Shared.Infrastructure/ClientInitiatedCancellationInterceptor.cs:line 84
at Grpc.Shared.Server.InterceptorPipelineBuilder`2.<>c__DisplayClass3_0.<b__1>d.MoveNext()
--- End of stack trace from previous location ---
at Grpc.Shared.Server.InterceptorPipelineBuilder`2.<>c__DisplayClass3_0.<b__1>d.MoveNext()
--- End of stack trace from previous location ---
at CalculationEngine.Shared.Server.Grpc.StatusFromDomainExceptionAugmentingInterceptor.DuplexStreamingServerHandler[TRequest,TResponse](IAsyncStreamReader`1 requestStream, IServerStreamWriter`1 responseStream, ServerCallContext context, DuplexStreamingServerMethod`2 continuation) in /src/src/CalculationEngine.Shared.Server.Grpc/StatusFromDomainExceptionAugmentingInterceptor.cs:line 55
at Grpc.Shared.Server.InterceptorPipelineBuilder`2.<>c__DisplayClass3_0.<b__1>d.MoveNext()
--- End of stack trace from previous location ---
at Grpc.Shared.Server.InterceptorPipelineBuilder`2.<>c__DisplayClass3_0.<b__1>d.MoveNext()
--- End of stack trace from previous location ---
at CalculationEngine.Shared.Infrastructure.GrpcCallLoggingInterceptor.DuplexStreamingServerHandler[TRequest,TResponse](IAsyncStreamReader`1 requestStream, IServerStreamWriter`1 responseStream, ServerCallContext context, DuplexStreamingServerMethod`2 continuation) in /src/src/CalculationEngine.Shared.Infrastructure/GrpcCallLoggingInterceptor.cs:line 94
at Grpc.Shared.Server.InterceptorPipelineBuilder`2.<>c__DisplayClass3_0.<b__1>d.MoveNext()
--- End of stack trace from previous location ---
at Grpc.Shared.Server.InterceptorPipelineBuilder`2.<>c__DisplayClass3_0.<b__1>d.MoveNext()
--- End of stack trace from previous location ---
at Grpc.Shared.Server.DuplexStreamingServerMethodInvoker`3.Invoke(HttpContext httpContext, ServerCallContext serverCallContext, IAsyncStreamReader`1 requestStream, IServerStreamWriter`1 responseStream)
at Grpc.AspNetCore.Server.Internal.CallHandlers.DuplexStreamingServerCallHandler`3.HandleCallAsyncCore(HttpContext httpContext, HttpContextServerCallContext serverCallContext)
at Grpc.AspNetCore.Server.Internal.CallHandlers.ServerCallHandlerBase`3.g__AwaitHandleCall|8_0(HttpContextServerCallContext serverCallContext, Method`2 method, Task handleCall)

### What did you expect to see?
Successful calls without errors.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.