F# Compiler & Tools performance discussion

Hi dotnet/performance folk :)

The F# compiler and tools ship as part of the .NET SDK, but we don't have a systematic, reproducible, reliable, longitudinal approach to performance and scalability testing for those tools. We'd like to understand what we should do about this.

Basically we'd love your advice on how we should be thinking about this, what we should be doing, and how we should be going about it.

People on our side are @vzarytovskii (team lead), @KathleenDollard (PM), @KevinRansom, @brettfo and others. @dsyme, @TIHan and others are v-team contributors.

cc @danmoseley @adamsitnik @DrewScoggins @LoopedBard3. Also @davkean since he's historically been a good source of advice on perf issues and may know the Roslyn team performance methodology.

I've written some initial notes below, thanks :) Overall it feels like these requirements must be similar in nature to many "upstack" components like Roslyn, ASP.NET and so on.

#### Areas of high concern:

* Our analysis is that it is **the performance of the compiler and tools themselves** that is of most immediate concern to .NET customers.

* This includes the performance of the FSharp.Compiler.Service component (F# equivalent of Roslyn) under developer-time scenarios, which we would model via bespoke perf benchmarking code testing code capturing common usage scenarios

* Scaling of the developer tools is of particular concern. We do not, for example, regularly test the tools with '00s of projects, or have automated testing for knowing how the tools scale w.r.t. larger inputs of different kinds.

#### Areas of currently lower concern:

* The quality of IL code generated by the F# compiler is not currently the high concern for F# customers or the dev team - while it can be improved, it's not likely to regress and we have many "IL baseline" tests that pin down the existing performance quality.

* The performance of the .NET CLR with respect to the code generated by the F# compiler is not of current high concern. With the exception of tailcalls, we feel existing .NET CLR perf testing for C# code is adequate. For tailcalls, we should test there are existing perf tests in place

* The performance of the FSharp.Core library is not currently the high concern for F# customers of the dev team. Incremental improvements are made by dev team and community, but regressions are very rare, and most implementation code is readily analysable for performance characteristics in code review.

* The interaction between F# and PGO and other .NET perf tooling is not currently of high concern. It is important, and should be validated, but we do not feel it's an area susceptible to regression

#### Approximate needs

For the compiler and tools, our rough needs are as follows:

What:

* Create and add benchmarks that exercise the F# tooling via the `dotnet` SDK command line, which is highly stable.
* Create and add benchmarks that exercise the F# tooling via bespoke unit benchmarking of "FSharp.Compiler.Service" (FCS). (Note the FCS API may change a little over time (about every 6 months), which can make things a little tricker, though we can find ways to workaround that.)

Execution:

* Run regularly on SDK releases and builds
* Run regularly where the SDK draws F# compiler tooling from the `main` and `release` branches on dotnet/fsharp
* Manually request a run where the SDK draws F# compiler tooling from PRs in dotnet/fsharp

Variation:

Where possible, "everything else" besides the F# tooling should be kept constant in these scenarios, or at least we should be able to identify change in F# tooling performance independently to change in .NET CLR performance, and know exactly which .NET CLR is used at each step.

Reporting:

* Automated production of graphs showing relative gains/losses, tracked back to commits in dotnet/fsharp