Sends wrong HTTP/2 response when RpcException happens and custom HTTP/2 trailer appended in middleware
- Dominant language
- C#
- Stars
- 4.5k
- Forks
- 836
- Avg merge
- 6d 3h
- Merged PRs (30d)
- 7
Description
### What version of gRPC and what language are you using?
Should work on every version, but I tested on `2.67.0`
### What operating system (Linux, Windows,...) and version?
It doesn't matters, but I tested on Tested on Windows, Linux, MacOS M1
### What runtime / compiler are you using (e.g. .NET Core SDK version `dotnet --info`)
It doesn't matters, but I tested on:
SDK: 9.0.300
Runtime: 9.0.5
### Reproduction
Create simplest gRPC service with Unary handler and throw `RpcException` from the inside like this:
```c#
public override async Task Unary(Empty _, ServerCallContext context)
{
// this is not important
await Task.Yield();
// RpcException with custom status code and some details
throw new RpcException(new Status(StatusCode.InvalidArgument, "My Exception"));
}
```
Now if you call this method with dotnet client or external tool, written with another languages, then it works just fine.
I use grpc_cli (but also tried Postman):
```bash
$ grpc_cli call --noremotedb --proto_path /Users/username/dev/sandbox/Sandbox.Grpc/Protos --protofiles echo.proto --timeout 2 localhost:6001 GrpcService/Unary ""
connecting to localhost:6001
Received trailing metadata from server:
content-length : 0
date : Wed, 25 Jun 2025 21:28:19 GMT
server : Kestrel
Rpc failed with status code 3, error message: My Exception
```
Both Status code and error message works fine.
Then I add middleware which write custom HTTP/2 trailer:
```c#
public class Startup
{
...
public void Configure(IApplicationBuilder app)
{
...
app.Use(static async (context, next) =>
{
await next(context);
if (context.Response.SupportsTrailers())
{
// append custom trailer
context.Response.AppendTrailer("x-my-custom-trailer", "42");
}
});
...
}
...
}
```
I can't write trailer from grpc-interceptor, because I need to do it after another middlewares which calculate some values for this trailer, so middleware in defined position is the only option I have.
Then if I call method I don't get correct error code:
```bash
$ grpc_cli call --noremotedb --proto_path /Users/username/dev/sandbox/Sandbox.Grpc/Protos --protofiles echo.proto --timeout 2 localhost:6001 GrpcService/Unary ""
connecting to localhost:6001
Received trailing metadata from server:
x-my-custom-trailer : 42
Rpc failed with status code 2, error message: No status received
```
Note that I don't get correct error code or status message.
I also don't get other metadata like `content-length`, `date` and `server`.
In Wireshark it looks like this:
```js
HyperText Transfer Protocol 2
Stream: HEADERS, Stream ID: 1, Length 121, 200 OK
...
Type: HEADERS (1)
Flags: 0x04, End Headers
...
[Header Count: 8]
Header: :status: 200 OK
Header: content-type: application/grpc
Header: date: Wed, 25 Jun 2025 21:38:30 GMT
Header: server: Kestrel
Header: content-length: 0
Header: grpc-status: 3
Header: grpc-message: My Exception
...
HyperText Transfer Protocol 2
Stream: HEADERS, Stream ID: 1, Length 24
...
Type: HEADERS (1)
Flags: 0x05, End Headers, End Stream
...
[Header Length: 29]
[Header Count: 1]
Header: x-my-custom-trailer: 42
```
Note that there is 2 HTTP/2 HEADERS sections: `Flags: 0x04, End Headers` and `Flags: 0x05, End Headers, End Stream`. And all metadata from gRPC library placed in the first section but my custom trailer placed in the second section.
Interesting that if I try to read it with another grpc-dotnet then I get correct `RpcException` but I don't get my custom trailer:
```c#
using var channel = GrpcChannel.ForAddress(url);
var client = new FixtureService.FixtureServiceClient(channel);
var request = new Empty();
using var call = client.UnaryCallAsync(request, cancellationToken: CancellationToken.None);
// it's empty here
var responseHeaders = await call.ResponseHeadersAsync;
try
{
await call.ResponseAsync;
}
catch(Exception ex)
{
// true
if (ex is RpcException rpcException)
{
// this 3 values are correct
rpcException.StatusCode; // InvalidArgument
rpcException.Status.StatusCode; // InvalidArgument
rpcException.Status.Detail; // My Exception
// but this is empty
var errorTrailers = rpcException.Trailers;
}
}
// and this is empty
var responseTrailers = call.GetTrailers();
```
### The essence of the error
If I append HTTP/2 thailer to `HTTPContext` then grpc-dotnet library does not add its own metadata to trailers but it writes it to headers instead. Then we get HTTP/2 answer with two HEADERs sections (one with grpc-dotnet metadata and one with custom trailers) instead of one with all trailers.
### Cause of error
I believe that all this magic came from this method: https://github.com/grpc/grpc-dotnet/blob/master/src/Grpc.AspNetCore.Server/Internal/GrpcProtocolHelpers.cs#L157
We check that `response.HasStarted` is `false`, believe that response is `Trailers-only` and write metadata to headers. But in our case someone already started another header with custom trailers.
So it seems that `response.HasStarted == false` is not enough to write metadata to headers. Maybe we need to check that there is no trailers already added like this:
```c#
public static IHeaderDictionary GetTrailersDestination(HttpResponse response)
{
var feature = response.HttpContext.Features.Get();
if (response.HasStarted)
{
// The response has content so write trailers to a trailing HEADERS frame
if (feature?.Trailers == null || feature.Trailers.IsReadOnly)
{
throw new InvalidOperationException("Trailers are not supported for this response. The server may not support gRPC.");
}
return feature.Trailers;
}
else
{
if (feature?.Trailers.Count > 0)
{
return feature.Trailers;
}
else
{
// The response is "Trailers-Only". There are no gRPC messages in the response so the status
// and other trailers can be placed in the header HEADERS frame
return response.Headers;
}
}
}
```
### Workaround
Now I use workaround - I copied method `GetTrailersDestination` and add HTTP/2 trailer to headers if response has not started but it seems that it's not clean solution.
I'll be happy if you could look at this issue in not too distant future.
Thank you for your attention
Contributor guide
Research direction
Start in src/Grpc.AspNetCore.Server/Internal/GrpcProtocolHelpers.cs at GetTrailersDestination and reproduce the issue with an RpcException plus middleware that appends an HTTP/2 trailer. Trace how response.HasStarted and existing trailers select the metadata destination; done means the custom trailer and gRPC status metadata are delivered together with the correct error status and message.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100