Routing performance improvements

We've seen some issues with large DFAs as a result of some patterns that involve a large amount of variable prefixes. We've done work in 6.0 to address this via automatically trimming the DFA we generate by leveraging existing route constraints as well as the existing structure in complex segments.

However, all these techniques don't guarantee smaller DFAs (although in most common cases they will) and require explicit actions from the user to identify and address this issue (adding constraints to the app). We believe there are other optimizations we can do in this space that will have an impact in the size of the DFA without involving explicit user action.

This is critical to provide a good experience for large applications as well as proxy servers like YARP as it contributes to reduce the memory footprint required to run those apps.

In addition to this, there are other things we want to consider with regards to performance on a per request basis. Our MapFallback APIs greatly negatively impact matching on other scenarios as they force a dynamic endpoint on to all the endpoint candidate lists of the app.
* For Blazor it will be ideal a "tighter" integration with ASP.NET Core routing to clearly identify the set of routes the app uses.
* For other SPA frameworks to provide the ability to read the list of client routes generated at build time from a file to narrow down the set of patterns involved.

Another initiative to pursue in this area is the ability to have more powerful matching primitives on a segment. Proxying scenarios involve matching a segment but not capturing the value, which reduces allocations per request. We do custom processing for complex segments where we perform our own "pseudo-regex" matching. We are also interested on some potential new "regex segment" as a more powerful expression of complex segments.

You can currently use regex constraints in parameters, however that leads to an awkward syntax and doesn't allow for capturing subsets of the regex expressions. These come in to play in proxying scenarios and other servers offer similar functionality. This might allows us to do two things:
* A more powerful version of complex segments, where the parts that aren't parameter don't have to be literals.
* A way to optimize existing complex segments by transforming them into "regex" segments.
* A way to offer non-capturing segments/parameters via non-capturing groups in the regex.
* A way to optimize all these patterns at compile time if we are able to work with the runtime folks to generate the regex code at compile time using the experimental regex source generator.

https://github.com/dotnet/aspnetcore/issues/33749
https://github.com/dotnet/aspnetcore/issues/27996
https://github.com/dotnet/aspnetcore/issues/27840