dotnet / dotnet/yarp

"Stream was already consumed" exception thrown even with enabled buffering.

Open
#2,721 18 comments 1 reaction 0 assignees View on GitHub
Type: Bug
Dominant language
C#
Stars
9.6k
Forks
933
Avg merge
12d 18h
Merged PRs (30d)
2

Description

**Upd with a short summary of the whole thread:** Not fixed. See a temporary [workaround](https://github.com/microsoft/reverse-proxy/issues/2721#issuecomment-2609855106).

## Description

We are experiencing **"Stream was already consumed"** issue, which has been explained previously: https://github.com/microsoft/reverse-proxy/issues/1683

> With HTTP/2, for example, we could have already started sending the request body to the server when the server responds with a GOAWAY. The server can tell us that it never processed the request, so HttpClient knows it can try to retry without side effects to the server application.

First, the request is failing with the following error message:

```plaintext
[HandlerMessage] poolId: 50533821, workerId: 0, requestId: 0, memberName: SendWithVersionDetectionAndRetryAsync, message: Retry attempt 1 after connection failure. Connection exception: System.Net.Http.HttpRequestException: The request was aborted.
---> System.Net.Http.HttpProtocolException: The HTTP/2 server closed the connection. HTTP/2 error code 'NO_ERROR' (0x0). (HttpProtocolError)
--- End of inner exception stack trace ---
at System.Net.Http.Http2Connection.ThrowRetry(String message, Exception innerException)
at System.Net.Http.Http2Connection.Http2Stream.TryEnsureHeaders()
at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken)
at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
```

The retry attempt fails with this error:

```plaintext
[HandlerMessage] poolId: 50533821, workerId: 19317591, requestId: 0, memberName: SendAsync, message: Sending request content failed: System.InvalidOperationException: Stream was already consumed.
at Yarp.ReverseProxy.Forwarder.StreamCopyHttpContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
at System.Net.Http.Http2Connection.Http2Stream.SendRequestBodyAsync(CancellationToken cancellationToken)
at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
```

## Problem
We are already doing buffering of request body for logging purposes using [EnableBuffering](https://learn.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.http.httprequestrewindextensions.enablebuffering?view=aspnetcore-9.0), but it is not being respected by `StreamCopyHttpContent`

1. Does it make sense to fix `StreamCopyHttpContent.SerializeToStreamAsync` behavior, so it won't throw "Stream was already consumed" exception if the `stream.CanSeek` (or has a type like [FileBufferingReadStream](https://learn.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.webutilities.filebufferingreadstream?view=aspnetcore-9.0)) ?

2. What's the official workaround to "buffering is needed to be able to retry failed requests" problem? How to by default enable it for all requests to downstream services?

Our current fix is implemented as a transform where we convert content to `StreamContent` that supports buffering. Is this approach reasonable, or is there a better way to address the issue?
```csharp
public void Apply(TransformBuilderContext context)
{
context.AddRequestTransform(transformContext =>
{
if (transformContext.ProxyRequest.Content != null)
{
var content = new StreamContent(transformContext.HttpContext.Request.Body);
foreach (var header in transformContext.ProxyRequest.Content.Headers)
{
content.Headers.TryAddWithoutValidation(header.Key, header.Value.ToArray());
}

transformContext.ProxyRequest.Content = content;
}

return ValueTask.CompletedTask;
});
}
```

## Expected behavior
When buffering the request body using `EnableBuffering()`, proxying the request to downstream services should not result in a "Stream was already consumed" exception.

Similar issues:
* https://github.com/microsoft/reverse-proxy/issues/2022
* https://github.com/microsoft/reverse-proxy/issues/1683

Contributor guide

Open the contributing guide

Research direction

Start at StreamCopyHttpContent.SerializeToStreamAsync and reproduce the retry path described in the exception, with EnableBuffering applied to the request body. Compare it with the supplied StreamContent transform and verify that a retried buffered request completes without a "Stream was already consumed" exception; document the supported workaround if no general fix is established.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.