dotnet / dotnet/msbuild

Contention of Project Evaluation in parallel builds

Open
#7,625 7 comments 1 reaction 0 assignees View on GitHub
Area: Performance backlog Partner request triaged
Dominant language
C#
Stars
5.5k
Forks
1.5k
Avg merge
1d 8h
Merged PRs (30d)
141

Description

### Issue Description
Project Evaluation in parallel builds have contention causing evaluations of 20-30ms to take over 1000ms.

### Steps to Reproduce
Create a solution with lots of small projects, enough to saturate your CPU. I used 4 times CPU threads worth of projects. The contents of each projects is not relevant as I used "Clean" target to do the least amount work. I used nearly identical projects to remove variables. Projects don't have P2P to maximize throughput. Nodereuse:false in all cases.

Case 1:
msbuild /t:clean /bl /v:q

Case 2:
msbuild /t:clean /bl /v:q /m

Used binlog to record results and set verbose to quiet to avoid console print out noise. Observe the Project Evaluation times of all projects.

### Data & Analysis
This image is the trace of a single node build (case 1). Observer that each evaluation time took a few 20-30ms, except for the initial project.
![image](https://user-images.githubusercontent.com/19828377/168402862-275699b6-12ff-418e-805e-c0871f274b5d.png)

This image is the trace of a multi node build (case 2) Observer that first evaluation took the same time in case 1, once parallel nodes started, the time of first evaluation takes seconds. Following subsequent project, their evaluation are faster. Notice node 1 is also having slowdown.
![image](https://user-images.githubusercontent.com/19828377/168402848-e84eaca7-c66a-4e21-a491-09d99d07b124.png)

### Theory
I theorize there is single threaded file cache service that handles file IO. The file cache probably serializes the data back to the nodes while holding onto the lock, thus blocking other nodes from using it. Node 0 is affected by the contention, so that disproves the "new" node cost.
Alternative is an evaluation cache where the lock is on the entire evaluation duration.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.