Contention of Project Evaluation in parallel builds
- Dominant language
- C#
- Stars
- 5.5k
- Forks
- 1.5k
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 141
Description
### Issue Description
Project Evaluation in parallel builds have contention causing evaluations of 20-30ms to take over 1000ms.
### Steps to Reproduce
Create a solution with lots of small projects, enough to saturate your CPU. I used 4 times CPU threads worth of projects. The contents of each projects is not relevant as I used "Clean" target to do the least amount work. I used nearly identical projects to remove variables. Projects don't have P2P to maximize throughput. Nodereuse:false in all cases.
Case 1:
msbuild /t:clean /bl /v:q
Case 2:
msbuild /t:clean /bl /v:q /m
Used binlog to record results and set verbose to quiet to avoid console print out noise. Observe the Project Evaluation times of all projects.
### Data & Analysis
This image is the trace of a single node build (case 1). Observer that each evaluation time took a few 20-30ms, except for the initial project.

This image is the trace of a multi node build (case 2) Observer that first evaluation took the same time in case 1, once parallel nodes started, the time of first evaluation takes seconds. Following subsequent project, their evaluation are faster. Notice node 1 is also having slowdown.

### Theory
I theorize there is single threaded file cache service that handles file IO. The file cache probably serializes the data back to the nodes while holding onto the lock, thus blocking other nodes from using it. Node 0 is affected by the contention, so that disproves the "new" node cost.
Alternative is an evaluation cache where the lock is on the entire evaluation duration.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.