Combine MapWhen with UseAuthorization and many Endpoints is slow

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Describe the bug

We have several hundreds of endpoints and discovered that our startup time of our application is terrible slow.
After Investigation into this situation we enclosed the problem that the call to "UseAuthorization" makes the WebApplication configuration process slow.

See here the sample code to reproduce the issue:
https://gist.github.com/softwaretirol/a3169c404226fbe5193c70a721ec8cc5

With UseAuthorization() the console gives us a duration of "00:00:02.94".
Without UseAuthorization() the console gives us a duration of "00:00:00.023"

### Expected Behavior

Same performance with UseAuthorization as without UseAuthorization

### Steps To Reproduce

_No response_

### Exceptions (if any)

_No response_

### .NET Version

8.0.100

### Anything else?

_No response_