dotnet / dotnet/roslyn

Honor BuildInParallel

Open
#82,173 7 comments 0 reactions 0 assignees View on GitHub
Area-Compilers help wanted
Dominant language
C#
Stars
20.7k
Forks
4.3k
PR merge metrics
PR metrics pending

Description

Commit 58047eaec542dec6081cd0393dc710f3f5d739be

My old system is very resource-constrained so I normally need to compile with the equivalent of `--jobs 1 --load 2.0`.

When building Roslyn, I had `htop` open because my system hung in a previous attempt (needed magic+sysrq+f (oom)), and saw all 4 cores were maxed and memory quickly climbed for the `/opt/dotnet/dotnet exec /opt/dotnet/sdk/9.0.306/Roslyn/bincore/VBCSCompiler.dll -pipename:` process.

Yes, "9.0". I made some small alterations to try to build with what I have (including rolling-back the arcade version). I've successfully done this in the past with omnisharp. It just takes time for me to figure out. But the "parallel" aspect is unrelated.

I looked anywhere I could for `BuildInParallel` that wasn't explicitly set to `false`, and forced it (eg `src/Features/CSharpTest/Microsoft.CodeAnalysis.CSharp.Features.UnitTests.csproj` and `src/LanguageServer/Microsoft.CodeAnalysis.LanguageServer/PackAllRids.targets`).

But even then, my command:
```
$ dotnet build \
--no-restore \
-verbosity:diag \
-maxCpuCount:1 \
-property:RestoreBuildInParallel=false \
-property:BuildInParallel=false \
-property:Configuration=Release \
-property:Platform="Any CPU" \
-property:TargetFramework=net9.0 \
-property:TargetRuntime=linux-x64 \
-property:DotNetBuildSourceOnly=true \
-property:DotNetBuildFromVMR=true \
-property:TargetRid=linux-x64 \
-property:RepoRoot=${PWD}/ \
-property:TreatWarningsAsErrors=false \
Compilers.slnf \
2>&1 | tee build.log
```
still resulted in:
```
<...>
//roslyn/src/NuGet/Microsoft.CodeAnalysis.BuildClient.Package/Microsoft.CodeAnalysis.BuildClient.Package.csproj
AdditionalProperties=Configuration=Release; Platform=AnyCPU
Configuration=Release
Platform=AnyCPU
SkipNonexistentProjects=False
ToolsVersion= (TaskId:3)
Task Parameter:BuildInParallel=True (TaskId:3)
Global Properties: (TaskId:3)
BuildingSolutionFile=true (TaskId:3)
CurrentSolutionConfigurationContents=
Release|AnyCPU
<...>
```
Note `BuildingSolutionFile=true` for that task.

While the system resources were climbing for the process, I froze it to see the last stuff outputted was:
```
<...other *.cs exceeding beyond the scrollback buffer>
//src/Compilers/CSharp/CSharpAnalyzerDriver/CSharpDeclarationComputer.cs "//artifacts/obj/Microsoft.CodeAnalysis.CSharp/Release/net9.0/.NETCoreApp,Version=v9.0.AssemblyAttributes.cs" //artifacts/obj/Microsoft.CodeAnalysis.CSharp/Release/net9.0/Microsoft.CodeAnalysis.CSharp.InternalsVisibleTo.cs //artifacts/obj/Microsoft.CodeAnalysis.CSharp/Release/net9.0/Microsoft.CodeAnalysis.CSharp.CSharpResources.cs //artifacts/obj/Microsoft.CodeAnalysis.CSharp/Release/net9.0/Microsoft.CodeAnalysis.CSharp.AssemblyInfo.cs /warnaserror+:NU1605,SYSLIB0011 /warnaserror-:IDE0055 (TaskId:307)
Assembly loaded during TaskRun (Microsoft.CodeAnalysis.BuildTasks.Csc): System.Security.Principal.Windows, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a (location: /opt/dotnet/shared/Microsoft.NETCore.App/9.0.10/System.Security.Principal.Windows.dll, MVID: d7ae0db5-c7a2-4ffb-94b9-b59a3271f1fc, AssemblyLoadContext: Default) (TaskId:307)
Assembly loaded during TaskRun (Microsoft.CodeAnalysis.BuildTasks.Csc): System.Security.Claims, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a (location: /opt/dotnet/shared/Microsoft.NETCore.App/9.0.10/System.Security.Claims.dll, MVID: 57921f00-3884-44e5-a77c-29ab174756fa, AssemblyLoadContext: Default) (TaskId:307)
Assembly loaded during TaskRun (Microsoft.CodeAnalysis.BuildTasks.Csc): System.IO.Pipes, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a (location: /opt/dotnet/shared/Microsoft.NETCore.App/9.0.10/System.IO.Pipes.dll, MVID: b7b52840-70ad-48cd-b397-7a554b891b4f, AssemblyLoadContext: Default) (TaskId:307)
Assembly loaded during TaskRun (Microsoft.CodeAnalysis.BuildTasks.Csc): System.Diagnostics.StackTrace, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a (location: /opt/dotnet/shared/Microsoft.NETCore.App/9.0.10/System.Diagnostics.StackTrace.dll, MVID: 6bcf773e-a3c7-4849-924e-d43e57ae6a0e, AssemblyLoadContext: Default) (TaskId:307)
```

So is the issue with `VBCSCompiler.dll` not receiving any `parallel` option?
A quick `ildasm` glance didn't show any such capability.

Contributor guide

Open the contributing guide

Research direction

Start with the diagnostic output from Compilers.slnf and the BuildInParallel settings in src/Features/CSharpTest/Microsoft.CodeAnalysis.CSharp.Features.UnitTests.csproj and src/LanguageServer/Microsoft.CodeAnalysis.LanguageServer/PackAllRids.targets. Trace how the VBCSCompiler.dll process and solution task handle --maxCpuCount and BuildInParallel. Done means identifying where parallelism is controlled and adding a focused test or documenting the observed behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.