dotnet / dotnet/msbuild

Question: Shared state file to improve RAR task performance

Open
#6,911 3 comments 0 reactions 0 assignees View on GitHub
Area: Performance triaged
Dominant language
C#
Stars
5.5k
Forks
1.5k
Avg merge
1d 13h
Merged PRs (30d)
133

Description

This is not a performance issue rather it is a question on how to use the MS Build API to solve a performance issue I have.

### Issue Description
My build has 7000~ directories in a packages folder. The total time for RAR across the projects in the build is 30 minutes.
I currently inject many search paths via SearchPaths into the RAR task for compatibility reasons within my build.

I desire to have a faster build so optimizing RAR felt like a good place to start.
A solution I've thought about for a long time is to have a single state file for all of the projects as there is so much duplicated information that each RAR task discovers as it is a bit of a gold fish on a clean CI build.

I noticed in a recent drop of MSBuild that some new RAR properties had shown up with the words "precompute" and "statefile" which sounds very exciting.

A sample was constructed to have a single RAR instance walk the packages directory and hopefully crack open every single DLL in the folder and write that info the cache file which I can then provide to each project.

I've _assumed_ I am able to use **AssemblyInformationCacheOutputPath** and then feed that back into **AssemblyInformationCachePaths**.

```
public class PrecomputeReferenceCache : Task {

public string[] ReferencePath { get; set; }

public override bool Execute() {
System.Diagnostics.Debugger.Launch();

// 7000 items
var directories = Directory.GetDirectories(ReferencePath[0], "*", SearchOption.AllDirectories);

var sw = Stopwatch.StartNew();
ResolveAssemblyReference rar = new ResolveAssemblyReference() {
BuildEngine = this.BuildEngine,
SearchPaths = directories,
Assemblies = new ITaskItem[] {
new TaskItem("WebGrease, Version=1.6.5135.21930, Culture=neutral, PublicKeyToken=31bf3856ad364e35")
},
AssemblyInformationCacheOutputPath = @"C:\temp\rar.cache2",
StateFile = @"C:\temp\rar.cache",
};

rar.Execute();
sw.Stop();
var elapsed1 = sw.ElapsedMilliseconds;

sw = Stopwatch.StartNew();
ResolveAssemblyReference rar2 = new ResolveAssemblyReference()
{
BuildEngine = this.BuildEngine,
SearchPaths = directories,
Assemblies = new ITaskItem[] {
new TaskItem("WebGrease, Version=1.6.5135.21930, Culture=neutral, PublicKeyToken=31bf3856ad364e35")
},
AssemblyInformationCachePaths = new ITaskItem[]
{
new TaskItem(@"C:\temp\rar.cache2")
},
StateFile = @"C:\temp\rar.cache"

};
rar2.Execute();
sw.Stop();
var elapsed2 = sw.ElapsedMilliseconds;

return !Log.HasLoggedErrors;
}
```

elapsed1: 4528
elapsed2: 5773

The second RAR instance takes longer than the first instance and still grovels into each folder in SearchPaths.

![image](https://user-images.githubusercontent.com/1083622/136095202-f70f8068-4944-4bc4-9c9d-0608e71db67c.png)

The output files from the first RAR instance have some size to them
![image](https://user-images.githubusercontent.com/1083622/136095018-7f1c60d7-3fb0-43ee-968b-f4cf4548e3a2.png)

I was hoping that upToDateLocalFileStateCache would contain all of the metadata for each assembly so I can reuse this information for each project and avoid the repeated analysis cost.

![image](https://user-images.githubusercontent.com/1083622/136095757-1523ae23-663e-4eb3-adec-62b356fdd8ad.png)

**What I've learned**
It appears RAR will only store metadata for a DLL if it is present in the Assemblies list.
In my example I'm looking for WebGrease which only appears in a single folder of the 7000.

If I add 7000 unique entries to the Assemblies list will I get the behaviour I desire?

If I cannot get the precomputed state file to work I could look at removing as many SearchPaths as possible to speed up RAR.

### Versions & Configurations
```
Microsoft (R) Build Engine version 16.11.0+0538acc04 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

16.11.0.36601
```

### Regression?
No

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with ResolveAssemblyReference and the AssemblyInformationCacheOutputPath, AssemblyInformationCachePaths, and StateFile properties shown in the sample task. Compare the first and second RAR invocations, including the continued SearchPaths traversal, to determine whether the cache is intended to cover this use case. Done means a documented, verified answer about sharing precomputed assembly state or a clearly scoped change request.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
build-system
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.