dotnet / dotnet/fsharp

Reliable FCS benchmarks which run on every release

Open
#13,606 6 comments 1 reaction 0 assignees View on GitHub
Theme-Performance
Dominant language
F#
Stars
4.3k
Forks
876
Avg merge
4d 22h
Merged PRs (30d)
144

Description

We need to come up with reliable FCS benchmarks, and run them as part of `dotnet/performance`, ideally every release (on every signed build for current release branch).

Some ideas for scenarios we want to test:
- VS related (may be better solved with telemetry, rather than benchmarks):
- Cold:
- Time to colouring and breadcrumbs in the newly built projects of different sizes.
- Time to intellisense in an arbitrary document in a newly build project of different sizes.
- Time to build, time to rebuild.
- Time to produce diagnostics on error (type errors, syntax errors, etc).
- Time to full typecheck of project graphs of both connected and independent.
- Hot (re-check):
- Same as above but in a prebuilt project of various sizes.
- General (FCS):
- Static tests:
- Typechecking of projects of various sizes.
- Typechecking of project graphs of both connected and independent.
- Runtime tests (pretty much testing our codegen).
- Time to main, we need to include different "configs" - ngen'd/crossgen'd, w/ and w/o mibc.

Some metrics we are interested in: time spent in scenarios, allocations (how much do we promote, is there anything ending up in LOH, etc), time spent in GC, etc.

More info and context: https://github.com/dotnet/fsharp/discussions/12526 & https://github.com/dotnet/fsharp/pull/13419

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.