gRPC client stability issues on high demand

Our apps are running on .NET 8 C# and Azure app service. In every user http request, our client app makes grpc call to the identity server to check permissions and other stuff, on high volume, the grpc client starts throwing the following exception : -
**Call failed with gRPC error status. Status code: 'Unknown', Message: 'Bad gRPC response. HTTP status code: 500'.**
There are no other details in the exception,

Our client is configured as the following: -

services.AddGrpcClient(o =>
{
o.Address = new Uri(Configuration["identity:MiddelwarePortalURL"]);

}).ConfigurePrimaryHttpMessageHandler(() =>
{
var handler = new SocketsHttpHandler();
handler.EnableMultipleHttp2Connections = true;
return handler;
}).ConfigureChannel((e) =>
{
var loggerFactory = LoggerFactory.Create(logging =>
{
logging.AddConsole();
logging.SetMinimumLevel(LogLevel.Debug);
});
e.LoggerFactory = loggerFactory;
})

Client logs

2024-06-04T18:45:39.2106962Z Successfully picked subchannel id '5-1' with address xxxx.azurewebsites.net:443. Transport status: ActiveStream
2024-06-04T18:45:39.2107162Z info: Grpc.Net.Client.Internal.GrpcCall[3]
2024-06-04T18:45:39.2107357Z Call failed with gRPC error status. Status code: 'Unknown', Message: 'Bad gRPC response. HTTP status code: 500'.
2024-06-04T18:45:39.2107561Z dbug: Grpc.Net.Client.Internal.GrpcCall[4]
2024-06-04T18:45:39.2107726Z Finished gRPC call.
2024-06-04T18:45:39.2107915Z info: Grpc.Net.Client.Internal.GrpcCall[3]
2024-06-04T18:45:39.2108092Z Call failed with gRPC error status. Status code: 'Unknown', Message: 'Bad gRPC response. HTTP status code: 500'.
2024-06-04T18:45:39.2108292Z dbug: Grpc.Net.Client.Internal.GrpcCall[4]
2024-06-04T18:45:39.2108478Z Finished gRPC call.2024-06-04T18:45:39.2108658Z info: Grpc.Net.Client.Internal.GrpcCall[3]
2024-06-04T18:45:39.2108846Z Call failed with gRPC error status. Status code: 'Unknown', Message: 'Bad gRPC response. HTTP status code: 500'.
2024-06-04T18:45:39.2109034Z dbug: Grpc.Net.Client.Internal.GrpcCall[4]
2024-06-04T18:45:39.2109230Z Finished gRPC call.

Testing code

```

List> l = new List>();
for(int i = 0; i < 1000; i++)
{
l.Add(_client.GetPermissionsAsync(new GetPermissionsRequest { Id = Profile.Id, Privilege = (int)method, Actor = actor }, new Metadata() { { "Authorization", $"Bearer {token}" } }));
}

var r = await Task.WhenAll(l.Select(c => c.ResponseHeadersAsync));
```

The issue names no repository files or tests. Start by reproducing the 1,000 concurrent GetPermissionsAsync calls on .NET 8 in Azure App Service, then inspect client and server logs around the HTTP 500 responses; done requires identifying the failure cause and documenting or verifying a fix.

Written by the indexing model from the issue text.