MapFallback catch all requests to API controller when ConsumeAttribute added

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Describe the bug

Hello.
In .net 6 behavior of MapFallback broke when ConsumesAttribute was added. MapFallback catch all requests to API controller.
In .net 6 index.html file always returns on requests to API controller but if I remove endpoints.MapFallbackToFile or delete ConsumesAttribute filter then everything works as I expected.
**In .net 5 the example works correctly.**

ConsumesAttribute was added for the correct generation of swagger.json by Swashbuckle in order not to add it to every action or controller.

### Expected Behavior

Requests to controllers must not be catched by MapFallback.

### Steps To Reproduce

```c#
public class Startup
{
public void ConfigureServices(IServiceCollection services)
{
services.AddControllers(options => options.Filters.Add(new ConsumesAttribute("application/json")));
}
public void Configure(IApplicationBuilder app)
{
app.UseRouting();
app.UseEndpoints(endpoints =>
{
endpoints.MapControllers();
endpoints.MapFallbackToFile("index.html");
});
}
}
```

### Exceptions (if any)

_No response_

### .NET Version

6.0.200

### Anything else?

_No response_