HttpRequestException after update to .NET 8

After updating to .NET 8, we see that we get a new HttpRequestException on Grpc calls to Bigtable, not sure if the issue is in Grpc.net or in google Bigtable. This issue only exists after upgrading to .NET 8.

It seems the error is intermittent, and it looks like it happens during high request periods of our application.

### What version of gRPC and what language are you using?
Grpc: 2.46.6
Grpc.Tools: 2.64.0
Google.Protobuf: 3.26.1
Google.Protobuf.Tools: 3.26.1
Google.Cloud.Bigtable.V2: 3.13.0

### What operating system (Linux, Windows,...) and version?
Docker Ubuntu aspnet image

### What runtime / compiler are you using (e.g. .NET Core SDK version `dotnet --info`)
Host:
Version: 8.0.6
Architecture: x64
Commit: 3b8b000a0e
RID: linux-x64

.NET SDKs installed:
No SDKs were found.

.NET runtimes installed:
Microsoft.AspNetCore.App 8.0.6 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 8.0.6 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
None

Environment variables:
Not set

global.json file:
Not found

Learn more:
https://aka.ms/dotnet/info

Download .NET:
https://aka.ms/dotnet/download

### What did you do?
Issue seem to only exists in kubernetes cluster, and seem to be intermittent. Not able to reproduce on Windows or Linux Mint while debugging. Have also tried to build Docker image on top of Focal with the same packages as .NET 6 runs on to check if it's an issue with updated linux packages as libssl3 instead of libssl1.1. Have tested

### What did you expect to see?
No new issues with GRPC after upgrading to .NET 8

### What did you see instead?
Exception as noted below

```
Error message:
Status(StatusCode="Internal", Detail="Error starting gRPC call. HttpRequestException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError) HttpProtocolException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)", DebugException="System.Net.Http.HttpRequestException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)") The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError) The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)

Stack trace:
Grpc.Core.RpcException:
at Grpc.Net.Client.Internal.HttpContentClientStreamReader`2+d__18.MoveNext (Grpc.Net.Client, Version=2.0.0.0, Culture=neutral, PublicKeyToken=d754f35622e28bad)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Google.Api.Gax.Grpc.Gcp.GcpClientResponseStream`2+d__7.MoveNext (Google.Api.Gax.Grpc, Version=4.8.0.0, Culture=neutral, PublicKeyToken=3ec5ea7f18953e47)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Google.Cloud.Bigtable.V2.RowAsyncEnumerator+d__23.MoveNext (Google.Cloud.Bigtable.V2, Version=3.10.0.0, Culture=neutral, PublicKeyToken=185c282632e132a0)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Linq.AsyncEnumerable+SelectEnumerableAsyncIterator`2+d__7.MoveNext (System.Linq.Async, Version=6.0.0.0, Culture=neutral, PublicKeyToken=94bc3704cddfc263: /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/Select.cs:221)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Linq.AsyncIteratorBase`1+d__8.MoveNext (System.Linq.Async, Version=6.0.0.0, Culture=neutral, PublicKeyToken=94bc3704cddfc263: /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:70)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Linq.AsyncIteratorBase`1+d__8.MoveNext (System.Linq.Async, Version=6.0.0.0, Culture=neutral, PublicKeyToken=94bc3704cddfc263: /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:75)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Linq.AsyncEnumerable+WhereEnumerableAsyncIterator`1+d__8.MoveNext (System.Linq.Async, Version=6.0.0.0, Culture=neutral, PublicKeyToken=94bc3704cddfc263: /_/Ix.NET/Source/System.Linq.Async/System/Linq/Operators/Where.cs:233)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Linq.AsyncIteratorBase`1+d__8.MoveNext (System.Linq.Async, Version=6.0.0.0, Culture=neutral, PublicKeyToken=94bc3704cddfc263: /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:70)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Linq.AsyncIteratorBase`1+d__8.MoveNext (System.Linq.Async, Version=6.0.0.0, Culture=neutral, PublicKeyToken=94bc3704cddfc263: /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncIterator.cs:75)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Collections.Generic.AsyncEnumerableHelpers+d__1`1.MoveNext (System.Linq.Async, Version=6.0.0.0, Culture=neutral, PublicKeyToken=94bc3704cddfc263: /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncEnumerableHelpers.cs:49)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Collections.Generic.AsyncEnumerableHelpers+d__1`1.MoveNext (System.Linq.Async, Version=6.0.0.0, Culture=neutral, PublicKeyToken=94bc3704cddfc263: /_/Ix.NET/Source/System.Linq.Async/System/Linq/AsyncEnumerableHelpers.cs:91)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Threading.Tasks.ValueTask`1.get_Result (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Collections.Generic.AsyncEnumerableHelpers+d__0`1.MoveNext (System.Linq.Async, Version=6.0.0.0, Culture=neutral, PublicKeyToken=94bc3704cddfc263)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Infrastructure.Repositories.ValueRepository+d__13.MoveNext (Infrastructure, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: /app/src/Infrastructure/Repositories/ValueRepository.cs:257)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Infrastructure.Repositories.ValuesRepository+d__7.MoveNext (Infrastructure, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: /app/src/Infrastructure/Repositories/ValuesRepository.cs:130)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Infrastructure.Repositories.ValuesRepository+d__6.MoveNext (Infrastructure, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: /app/src/Infrastructure/ValuesRepository.cs:44)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Core.Services.ValuesService+d__5.MoveNext (Core, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: /app/src/Core/Services/ValuesService.cs:54)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Api.Services.Calculation.CalculationHandler+d__3.MoveNext (Api, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: /app/src/Api/Services/Calculation/CalculationHandler.cs:50)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Api.Services.Calculation.Internal.AbstractCalculationHandler+d__22.MoveNext (Api, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: /app/src/Api/Services/Calculation/Internal/AbstractCalculationHandler.cs:57)
Inner exception System.Net.Http.HttpRequestException handled at Grpc.Net.Client.Internal.HttpContentClientStreamReader`2+d__18.MoveNext:
at System.Net.Http.Http2Connection+d__111.MoveNext (System.Net.Http, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.HttpConnectionPool+d__89.MoveNext (System.Net.Http, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.DiagnosticsHandler+d__10.MoveNext (System.Net.Http, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.RedirectHandler+d__4.MoveNext (System.Net.Http, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Grpc.Net.Client.Balancer.Internal.BalancerHttpHandler+d__11.MoveNext (Grpc.Net.Client, Version=2.0.0.0, Culture=neutral, PublicKeyToken=d754f35622e28bad)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Grpc.Net.Client.Internal.GrpcCall`2+d__82.MoveNext (Grpc.Net.Client, Version=2.0.0.0, Culture=neutral, PublicKeyToken=d754f35622e28bad)
Inner exception System.Net.Http.HttpProtocolException handled at System.Net.Http.Http2Connection+d__111.MoveNext:
at System.Net.Http.Http2Connection.ThrowRequestAborted (System.Net.Http, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Net.Http.Http2Connection+Http2Stream.TryEnsureHeaders (System.Net.Http, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Net.Http.Http2Connection+Http2Stream+d__74.MoveNext (System.Net.Http, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.Http2Connection+d__111.MoveNext (System.Net.Http, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
```

The failure surfaces in Grpc.Net.Client.Internal.HttpContentClientStreamReader during the Bigtable stream; start by tracing the call from Infrastructure/Repositories/ValueRepository.cs:257 and comparing the .NET 6 and .NET 8 Docker/Kubernetes environments. Done means producing a reliable reproduction or isolating whether the HTTP/2 PROTOCOL_ERROR comes from Grpc.Net.Client, Bigtable, or the deployment environment, with supporting diagnostics.

Written by the indexing model from the issue text.