Reliable FCS benchmarks which run on every release
- Dominant language
- F#
- Stars
- 4.3k
- Forks
- 876
- Avg merge
- 4d 22h
- Merged PRs (30d)
- 144
Description
We need to come up with reliable FCS benchmarks, and run them as part of `dotnet/performance`, ideally every release (on every signed build for current release branch).
Some ideas for scenarios we want to test:
- VS related (may be better solved with telemetry, rather than benchmarks):
- Cold:
- Time to colouring and breadcrumbs in the newly built projects of different sizes.
- Time to intellisense in an arbitrary document in a newly build project of different sizes.
- Time to build, time to rebuild.
- Time to produce diagnostics on error (type errors, syntax errors, etc).
- Time to full typecheck of project graphs of both connected and independent.
- Hot (re-check):
- Same as above but in a prebuilt project of various sizes.
- General (FCS):
- Static tests:
- Typechecking of projects of various sizes.
- Typechecking of project graphs of both connected and independent.
- Runtime tests (pretty much testing our codegen).
- Time to main, we need to include different "configs" - ngen'd/crossgen'd, w/ and w/o mibc.
Some metrics we are interested in: time spent in scenarios, allocations (how much do we promote, is there anything ending up in LOH, etc), time spent in GC, etc.
More info and context: https://github.com/dotnet/fsharp/discussions/12526 & https://github.com/dotnet/fsharp/pull/13419
Contributor guide
Assessment
This issue has not been assessed yet.