gRPC stream "Response ended prematurely (ResponseEnded)"
- Dominant language
- C#
- Stars
- 4.5k
- Forks
- 836
- Avg merge
- 6d 3h
- Merged PRs (30d)
- 7
Description
### What version of gRPC and what language are you using?
C#, `Grpc.Net.Client` version `2.60.0`
### What operating system (Linux, Windows,...) and version?
macOS 14.2.1, Linux (Ubuntu on Google Cloud Kubernetes Engine)
### What runtime / compiler are you using (e.g. .NET Core SDK version `dotnet --info`)
```
.NET SDK:
Version: 8.0.100
```
### What did you do?
> This may potentially be related to #2358
We have a long-lived gRPC streaming call that gets canceled after receiving a few messages with the following:
```
Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="Error reading next message. HttpIOException: The response ended prematurely while waiting for the next frame from the server. (ResponseEnded)", DebugException="System.Net.Http.HttpIOException: The response ended prematurely while waiting for the next frame from the server. (ResponseEnded)")
---> System.Net.Http.HttpIOException: The response ended prematurely while waiting for the next frame from the server. (ResponseEnded)
at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException)
at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()
at System.Net.Http.Http2Connection.Http2Stream.TryReadFromBuffer(Span`1 buffer, Boolean partOfSyncRead)
at System.Net.Http.Http2Connection.Http2Stream.ReadDataAsync(Memory`1 buffer, HttpResponseMessage responseMessage, CancellationToken cancellationToken)
at Grpc.Net.Client.Internal.StreamExtensions.ReadMessageAsync[TResponse](Stream responseStream, GrpcCall call, Func`2 deserializer, String grpcEncoding, Boolean singleMessage, CancellationToken cancellationToken)
at Grpc.Net.Client.Internal.GrpcCall`2.ReadMessageAsync(Stream responseStream, String grpcEncoding, Boolean singleMessage, CancellationToken cancellationToken)
at Grpc.Net.Client.Internal.HttpContentClientStreamReader`2.MoveNextCore(CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at Grpc.Net.Client.Internal.HttpContentClientStreamReader`2.MoveNextCore(CancellationToken cancellationToken)
at Grpc.Core.AsyncStreamReaderExtensions.ReadAllAsyncCore[T](IAsyncStreamReader`1 streamReader, CancellationToken cancellationToken)+MoveNext()
at Grpc.Core.AsyncStreamReaderExtensions.ReadAllAsyncCore[T](IAsyncStreamReader`1 streamReader, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource.GetResult()
```
So far, we can reproduce this when using a [L7 external load balancer in Google Cloud (Classic Application Load Balancer)](https://cloud.google.com/load-balancing/docs/https). When using an L4 external load balancer, it works as expected.
We have built [an extensive reproduction](https://github.com/taxfyle/lb-issue-repro), including multiple servers and clients in different languages to isolate the root cause as being the gRPC dotnet client library. The following clients have been tested and work without issue:
* grpcurl
* custom Golang client (see repo)
* custom Rust client (see repo)
The linked repository also includes information about the infrastructure setup.
The following proto is used:
```protobuf
syntax = "proto3";
option csharp_namespace = "LbIssue";
option go_package = "github.com/taxfyle/lb-issue-repro/src/go/pb";
package lb_issue;
service Demo {
rpc StreamMessages (StreamMessagesRequest) returns (stream StreamMessagesResponse);
}
message StreamMessagesRequest {
string name = 1;
}
message StreamMessagesResponse {
string message = 1;
}
```
At first, we thought the root cause was the L7 load balancer, however, given other gRPC client libraries work without issue, we figured it might be something specific in the dotnet one.
The issue occurs after having received between 5-8 messages on the stream when sending a message roughly every second. 6 messages is the most common one we've observed.
If we send messages faster (using a `while` loop that calls a HTTP endpoint on the server to trigger messages), we get between 20-25 messages before the connection is broken.
We also observed that the connection breaks _after_ receiving a message, never while waiting for one.
Contributor guide
Research direction
The issue names no repository files or tests; start with the Grpc.Net.Client stack entries HttpContentClientStreamReader.MoveNextCore and Http2Connection, then run the linked lb-issue-repro setup. Compare the L7 and L4 load balancer cases and the working grpcurl, Go, and Rust clients. Done means identifying why the .NET stream ends prematurely and validating a fix against the reproduction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, google-cloud, grpc, kubernetes
- Domain
- backend-api-design, cloud, distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100