dotnet / dotnet/msbuild

Investigate possible build perf improvements in Copy task

Open
#11,374 2 comments 0 reactions 1 assignee Claimed by @AR-May View on GitHub
Area: Performance triaged
Dominant language
C#
Stars
5.5k
Forks
1.5k
Avg merge
1d 8h
Merged PRs (30d)
141

Description

### Context

Reported by @davkean:

> In 40% of builds, I see something that Eric was looking at; [synchronous calls to Copy taking out thread pool threads](https://prism.vsdata.io/failure/?query=tags%3Dttperfscenarios_solutionbuild%20bc!%3Dgc%20bc!%3Dhardfault%20threadtype%3D.NETThreadPool&eventType=blockedtime&failureType=dualdirection&failureHash=3cf89148-5518-7128-36c4-d660b78553b0). The CLR will take 500ms before it adds another thread to replace each one of these (so 5 seconds to replace 10 threads), so this is likely causing thread pool starvation and a non-optimal use of CPU usage while that occurs.

Due to the 'multi-process' nature of msbuild the threatpool starvation might not be such a big concern yet, but blocking the ability to run other waiting build work is of a big concern.

Additionaly there might be additional task in a same bucket - e.g. [`CopyRefAssembly`](https://github.com/dotnet/roslyn/blob/main/src/Compilers/Core/MSBuildTask/CopyRefAssembly.cs) from roslyn

### Suggested approach

- Attempt to perform couple experiments with implanting `TaskHost.Yield` into the Copy task (similarly to what Zip/Unzip does today) and evaluate the impact on the build
- Attempt to see if we can improve the current paralelized copy algorithm (possbily in connection with above)

### Goal

`Copy` is notoriously among the top time consumers within the builds

Image

being able to use the consummed time for additional build tasks might help improve the CPU utilization and throughput of the build

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.