Endpoint routing can not invoke a static file middleware

### Describe the bug
The static file middleware does not allow any form of serving of files when used in combination with endpoint routing. Maybe I am missing something here, but this prevents me from doing stuff like this:

Notice I am essentially trying to add auth to my Swagger UI without writing a custom middleware.

```
app.UseEndpoints(endpoints =>
{
endpoints.MapControllers();

var swaggerUiDelegate = endpoints.CreateApplicationBuilder().UseSwaggerUI(options =>
{
// Set documentation route to /docs
options.RoutePrefix = "docs";
})
.Build();

endpoints.Map("docs/{*wildcard}", swaggerUiDelegate).RequireAuthorization("Swagger");
});
```

### To Reproduce
The above code should work if you essentially just add Swashbuckle to your project and attempt to add auth to your Swagger UI.

### Exceptions (if any)
InvalidOperationException: The request reached the end of the pipeline without executing the endpoint: 'docs/{*wildcard}'. Please register the EndpointMiddleware using 'IApplicationBuilder.UseEndpoints(...)' if using routing.

### Further technical details
- ASP.NET Core version = 3.1
- Include the output of `dotnet --info`
```
.NET Core SDK (reflecting any global.json):
Version: 3.1.201
Commit: b1768b4ae7

Runtime Environment:
OS Name: Mac OS X
OS Version: 10.15
OS Platform: Darwin
RID: osx.10.15-x64
Base Path: /usr/local/share/dotnet/sdk/3.1.201/

Host (useful for support):
Version: 5.0.0-preview.7.20364.11
Commit: 53976d38b1

.NET SDKs installed:
2.1.805 [/usr/local/share/dotnet/sdk]
3.1.201 [/usr/local/share/dotnet/sdk]
3.1.302 [/usr/local/share/dotnet/sdk]
5.0.100-preview.7.20366.6 [/usr/local/share/dotnet/sdk]

.NET runtimes installed:
Microsoft.AspNetCore.All 2.1.17 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.17 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.3 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.6 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.0-preview.7.20365.19 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.17 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.20 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.3 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.6 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.0-preview.7.20364.11 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET runtimes or SDKs:
https://aka.ms/dotnet-download
```
- The IDE (VS / VS Code/ VS4Mac) you're running on, and it's version
Visual Studio 2019 for Mac