dotnet build slow when using GeneratePathProperty="true" and $(Pkg...) variables
- Dominant language
- C#
- Stars
- 5.5k
- Forks
- 1.5k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 133
Description
### Issue Description
`dotnet restore` and `dotnet build` take a long time to build a solution. The solution only has one project and uses `GeneratePathProperty="true"` and a `$(PkgPackage_Name)` variable with wildcard paths.
I have a net6.0 console project that have some NuGet dependencies on external tools (`NUnit.ConsoleRunner` in this example). Our CI expects to find the console runner executable in a specific place, and I want to copy it using the .csproj project. This .csproj reproduces the issue. `dotnet restore` and `dotnet build` take about 4 min in my system with this example:
```xml
Exe
net6.0
enable
enable
```
### Steps to Reproduce
* Unzip this [sample project](https://github.com/dotnet/msbuild/files/8532638/NugetPerformanceIssuesCore.zip)
* Run `dotnet restore`
* Run `dotnet build`
Both commands take about 4 min on my Windows laptop. If I remove the `**` from the `` tag, or if I hardwire the path instead of using the variable `$(PkgNUnit_ConsoleRunner)` it takes a couple of seconds to run as expected.
### Data
These are the execution times when not using `**`:
```
~\source\repos\NugetPerformanceIssuesCore
λ dotnet restore
Determining projects to restore...
Restored C:\Users\MyUser\source\repos\NugetPerformanceIssuesCore\NugetPerformanceIssuesCore\NugetPerformanceIssuesCore.csproj (in 267 ms).
~\source\repos\NugetPerformanceIssuesCore
λ dotnet build
Microsoft (R) Build Engine version 17.1.1+a02f73656 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.
Determining projects to restore...
All projects are up-to-date for restore.
NugetPerformanceIssuesCore -> C:\Users\MyUser\source\repos\NugetPerformanceIssuesCore\NugetPerformanceIssuesCore\bin\Debug\net6.0\NugetPerformanceIssuesCore.dll
Build succeeded.
0 Warning(s)
0 Error(s)
Time Elapsed 00:00:03.65
```
And these when `**` is used to copy recursivelly. Please note that the "Determining projects to restore..." step is not timed by the dotnet command, but in my tests it took almost the same as `dotnet build`.
```
~\source\repos\NugetPerformanceIssuesCore
λ dotnet restore
Determining projects to restore...
Restored C:\Users\MyUser\source\repos\NugetPerformanceIssuesCore\NugetPerformanceIssuesCore\NugetPerformanceIssuesCore.csproj (in 335 ms).
~\source\repos\NugetPerformanceIssuesCore
λ dotnet build
Microsoft (R) Build Engine version 17.1.1+a02f73656 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.
Determining projects to restore...
All projects are up-to-date for restore.
NugetPerformanceIssuesCore -> C:\Users\MyUser\source\repos\NugetPerformanceIssuesCore\NugetPerformanceIssuesCore\bin\Debug\net6.0\NugetPerformanceIssuesCore.dll
Build succeeded.
0 Warning(s)
0 Error(s)
Time Elapsed 00:03:45.57
```
### Analysis
I ran both commands while monitoring with [Procmon](https://docs.microsoft.com/en-us/sysinternals/downloads/procmon) and `dotnet` seems to be traversing the whole filesystem, as if `$(PkgNUnit_ConsoleRunner)\**\*.*` was resolved as `\**\*.*`. This does not happen with `$(PkgNUnit_ConsoleRunner)\*.*` or with `C:\Users\MyUser\.nuget\packages\nunit.consolerunner\3.15.0\**\*.*`, but of course both solutions are not desirable.
### Versions & Configurations
```
OS: Windows
Architecture: x64
MSBuild: Microsoft (R) Build Engine version 17.1.1+a02f73656 for .NET (It can be reproduced with 17.0 too)
```
I have reproduced the issue with net6.0 and net472 projects (as long as you use ``).
```
λ dotnet --info
.NET SDK (reflecting any global.json):
Version: 6.0.202
Commit: f8a55617d2
Runtime Environment:
OS Name: Windows
OS Version: 10.0.19043
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\6.0.202\
Host (useful for support):
Version: 6.0.4
Commit: be98e88c76
.NET SDKs installed:
2.1.202 [C:\Program Files\dotnet\sdk]
2.1.401 [C:\Program Files\dotnet\sdk]
2.1.402 [C:\Program Files\dotnet\sdk]
2.1.403 [C:\Program Files\dotnet\sdk]
2.1.526 [C:\Program Files\dotnet\sdk]
2.1.602 [C:\Program Files\dotnet\sdk]
2.1.700 [C:\Program Files\dotnet\sdk]
2.2.105 [C:\Program Files\dotnet\sdk]
2.2.202 [C:\Program Files\dotnet\sdk]
2.2.300 [C:\Program Files\dotnet\sdk]
3.1.412 [C:\Program Files\dotnet\sdk]
5.0.100 [C:\Program Files\dotnet\sdk]
5.0.401 [C:\Program Files\dotnet\sdk]
5.0.404 [C:\Program Files\dotnet\sdk]
6.0.100 [C:\Program Files\dotnet\sdk]
6.0.202 [C:\Program Files\dotnet\sdk]
.NET runtimes installed:
Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.24 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.24 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.18 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.22 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.10 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.13 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 6.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 6.0.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.3-servicing-26724-03 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.24 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.2.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.18 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.22 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.23 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.10 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.14 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 6.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 3.1.9 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 3.1.18 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 3.1.22 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 3.1.23 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 5.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 5.0.10 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 5.0.13 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 5.0.14 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 6.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 6.0.3 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 6.0.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
To install additional .NET runtimes or SDKs:
https://aka.ms/dotnet-download
```
### Regression?
I do not know.
Contributor guide
No contributing guide indexed for this repository
Research direction
Unzip the linked NugetPerformanceIssuesCore sample and run dotnet restore and dotnet build with the recursive $(PkgNUnit_ConsoleRunner)\**\*.* entry in its .csproj. Compare it with the wildcard and hardcoded-path variants, then trace how GeneratePathProperty and the package variable are evaluated. Done means recursive copying still works without traversing the whole drive or taking minutes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- build-system, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100