HttpLoggingMiddleware breaks gRPC service response

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Describe the bug

If the `HttpLoggingMiddleware` gets used in aspnet core 6 hosting Rest API and gRPC service
then the gRPC response message gets transparently removed.

The verbose logging of `Grpc.AspNetCore.Server.ServerCallHandler` logs following:
```
...
Sending message.
Serialized 'MyService.Protos.MyResponse' to 28 byte message.
Message sent.
Request deadline stopped.
Executed endpoint 'gRPC - /MyService/MyMethod'
```

The verbose logging of `Grpc.Net.Client.Internal.GrpcCall` logs following:
```
...
Reading message.
No message returned.
Message not returned from unary or client streaming call.
Call failed with gRPC error status. Status code: '"Internal"', Message: 'Failed to deserialize response message.'.
Finished gRPC call.
...
An unhandled exception has occurred while executing the request.
Grpc.Core.RpcException: Status(StatusCode="Internal", Detail="Failed to deserialize response message.")
```

Only the outgoing response message gets somehow removed by `HttpLoggingMiddleware`
Disabling it resolves the issue instantly

### Expected Behavior

`HttpLoggingMiddleware` should only log normal http requests and/or give more options to enable or filter gRPC logging

### Steps To Reproduce

service configuration:
```
public void ConfigureServices(IServiceCollection services)
{
//...
services.AddGrpc(opt =>
{
});

services.AddHttpLogging(logging =>
{
logging.LoggingFields = HttpLoggingFields.All;
logging.RequestBodyLogLimit = 4096;
logging.ResponseBodyLogLimit = 4096;
});
}
```

Configuration
```
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
if (env.IsDevelopment())
{
IdentityModelEventSource.ShowPII = true;

app.UseHttpLogging();

app.UseDeveloperExceptionPage();
app.UseSwagger();
app.UseSwaggerUI(c => c.SwaggerEndpoint("/swagger/v1/swagger.json", "MyService API v1"));
}
//...

app.UseEndpoints(endpoints =>
{
//...

endpoints.MapGrpcService()
.RequireHost("*:82", "*:443");

endpoints.MapControllers();
});
}
```

### Exceptions (if any)

On the gRPC Client side, inside an rest api
```
An unhandled exception has occurred while executing the request.
Grpc.Core.RpcException: Status(StatusCode="Internal", Detail="Failed to deserialize response message.")
at MyService.WebApi.Controllers.MyController.PostMethod(MyRequest request, MyClient client, CancellationToken cancellationToken) in C\MyService\src\MyService.WebApi\Controllers\MyController.cs:line 70
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Logged|17_1(ResourceInvoker invoker)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Logged|17_1(ResourceInvoker invoker)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)
```

### .NET Version

6.0.100

### Anything else?

_No response_