dotnet / dotnet/BenchmarkDotNet
Source Generators based Toolchain
- Dominant language
- C#
- Stars
- 11.5k
- Forks
- 1.1k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 11
Description
From the .NET Team perspective we quite often hit some build-related issues that require us to change BDN source code and update to latest version. This is especially painful with the WASM toolchain which needs to catch up with a very quickly envolving WASM project build requirements. (cc @naricc @radekdoulik @Lxiamail) It's also a problem for early .NET adopters like ASP.NET Team (cc @halter73 @sebastienros @davidfowl) and projects that use a lot of custom MSBuild settings like ML.NET (@eerhardt).
BDN also lacks the possibility to build the benchmarks on one machine, copy it to a different machine that has no .NET SDK installed and run them there (cc @billwert @drewscoggins @ig-sinicyn).
I'be been thinking about it for a while and I believe that we could solve both problems by implementing a toolchain that takes advantage of C# Source Generators. The idea I have is following:
1. Use source generators to generate the boilerplate code (in theory all custom MSBuild setting works OOTB)
2. The host process starts new benchmarks by using it's own exe path, but with a different set of arguments that are recognized and translated to "find selected benchmark in the assembly and run it".
3. Users should be allowed to specify the runner exe path and args without modifying BDN. Example: V8 for WASM, CoreRun for dotnet/runtime builds.
It would defnitely not become the default toolchain (at least for now) because of the following limitations:
- no F# support (me and @AndreyAkinshin care about F#)
- no ability to run benchmarks for a different runtime (no `--runtimes` support)
I am also not sure what would be the performance of C# Source Generators in case of dotnet/performance Microbenchmarks.csproj that contains 3.5k benchmarks (the boilerplate generated with the existing toolchains in dotnet/performance is more than one million lines of source code and Roslyn fails with OOTM for x86 when we try to run all the benchmarks).
cc @jeffhandley @danmoseley
Contributor guide
Assessment
This issue has not been assessed yet.