dotnet / dotnet/msbuild

Supporting MSBuild hosts that want to be AOT- or Trim-compiled

Open
#12,141 1 comment 6 reactions 1 assignee Claimed by @baronfel View on GitHub
Area: Evaluation
Dominant language
C#
Stars
5.5k
Forks
1.5k
Avg merge
1d 8h
Merged PRs (30d)
141

Description

### Summary

MSBuild can be hosted/used in a variety of scenarios:
* `msbuild.exe` command-line tool
* `dotnet` and it's variety of build- and evaluation-using commands
* API-based hosts that do evaluation - this is often tooling that wants to make use of data contained _in_ the project
* API-based hosts that do execution (including "design-time builds")

Today, all API-based users of MSBuild are prevented from exploring AOT or Trimmed deployment modes because the MSBuild libraries use trimming-unfriendly patterns in several places. While it is not required for MSBuild to entirely move to statically-analyzable mechanisms for some of these patterns, it _is_ required that we provide enough extensibility points that a motivated host like `dotnet` could provide statically-analyzable implementations where required.

### Background and Motivation

The main goal of this epic is to
* define the MSBuild features/functionality that we would like to enable for AOT/Trimming scenarios
* catalogue the blockers to each of those features
* establish a prioritization for addressing each of those blockers, weighted towards the needs of `dotnet` specifically
* do all of this with an eye specifically towards enabling AOT of the `dotnet` CLI. Any other clients that would like to AOT are nice, but not directly the aim.

We want to do this work so that the `dotnet` CLI can drastically reduce the overhead of invoking its commands. Many SDK commands do either evaluations or full builds, and while for the full builds we can eject to calling MSBuild out-of proc, there is no such easy pathway to handling evaluations inside an AOT app.

### Proposed Feature

## Use Cases to enable for AOT

MSBuild is by design a _very_ extensible system. It consists of two primary phases:

* Evaluation, where Projects are read/created and their Properties, Items, and Targets are discovered
* Execution, where various Targets on those Projects are run

**Execution** specifically is highly-dynamic - it can occur on many different Nodes, and often requires loading Tasks from assemblies that may not be known when the application is compiled (for example, if they are registered and provided by NuGet packages). It is infeasible for execution of Tasks to occur in an AOT'd application as a result.

Therefore we will focus on the blockers to performing **Evaluation** in an AOT'd host. Evaluation has five main areas that are reflection-unfriendly today:

* Certain MSBuild-provided implicit values/properties - often related to versioning of MSBuild itself.
* SdkResolver discovery and invocation as part of SdkResolution, very early in the Evaluation phase
* Property Function invocation, which happens typically by reflection at various points during Evaluation
* Adding dynamically-provided loggers to an evaluation
* Adding buildcheck checks discovered during and evaluation

### MSBuild Implicit values

In my mind this is relatively straightforward. Cases like [`ProjectCollection.Version`](https://github.com/dotnet/msbuild/blob/f9070440b6f9a59b33395dfa8927057da9e2b1be/src/Build/Definition/ProjectCollection.cs#L448-L498) where we load metadata from the MSBuild assembly need an AOT-friendly mechanism. We can't just rely on, e.g. the version of the `dotnet.exe` instead of `MSBuild.dll` because those products do not version on the same schedule, and we need to keep to MSBuild's versioning scheme because that value is used for comparisons in build logic.

### SDK Resolver discovery and invocation

MSBuild Sdk Resolution is currently behind an [internal ISdkResolverService interface](https://github.com/dotnet/msbuild/blob/d131702c0a3082e192335d28454980b245d109ea/src/Build/BackEnd/Components/SdkResolution/ISdkResolverService.cs#L16). This interface is registered in the IBuildComponentHost and then transparently used by the rest of the build when necessary.

The default implementation of this interface reads manifest files to load assemblies dynamically at runtime, which would not be possible for AOT'd/Trimmed applications.

If this interface were public, it would then be possible for AOT/Trimmed hosts (that have the exhaustive set of SDKResolvers that they need) to create a static version that uses more understandable techniques like name matching + explicit dispatching, and use it for any evaluations they need.

### Property Function invocation

When MSBuild well-known, intrinsic or property functions are invoked, they go through a few pathways:
* a [type-and-method-name based lookup](https://github.com/dotnet/msbuild/blob/8441cf4e7b980732471b78491b0fe1e6f512df34/src/Build/Evaluation/Expander/WellKnownFunctions.cs#L781)
* a [series of reflection-based invocations](https://github.com/dotnet/msbuild/blob/04e602dcfb6a39c9c7fb402d77d8ca49cbf76f04/src/Build/Evaluation/Expander.cs#L4031-L4055)
* [NuGet](https://github.com/dotnet/msbuild/blob/a4ef175ef7015e3a072fa84ca37c1af7d54e34a7/src/Build/Utilities/NuGetFrameworkWrapper.cs#L11)-and maybe [BuildEnvironment](https://github.com/dotnet/msbuild/blob/0e2431add8a483acfd985c6fdd352ebfcade6755/src/Shared/BuildEnvironmentHelper.cs) initialization that can use reflection or COM, etc.

Anything in this path that uses reflection is potentially fraught. We'd need a way for hosts to provide
* explicit NuGet calls in lieu of the wrapper
* shims for BuildEnvironment, parts of this could be static?
* In the most extreme case an entire replacement for the entire Intrinsic and Well-known functions lookup mechanism

### Managing Loggers

These AOT hosts will be able to create and manage Loggers for their evaluations, but they _will not_ be able to create any Loggers that are not part of their Trimming closure.

### Managing BuildChecks

BuildChecks are initialized when found/discovered during evaluation, and this happens by way of the `RegisterBuildCheck` Intrinsic Function. The default implementation then does reflection-based loading of those assemblies. AOT'd applications would need to replace this component, or likely disable it for their evaluations.

## Other resources

I have a horrible, hacky proof of concept of this on the [aot-mode](https://github.com/dotnet/msbuild/compare/main...aot-mode) branch, which I used together with my fork of the SDK's [aot-tinkering](https://github.com/baronfel/sdk/tree/aot-tinkering) branch to prove that with enough munging AOT'd evaluations are _possible_. With all of this cutting away, I was able to do an evaluation of a simple SDK-style console app in about 100ms of total process time (incl. startup, teardown, etc).

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.