SDK local build is extremely flaky

Running `Build.cmd` locally is very flaky. It can fail with very different errors each time, and takes so long until I get a green build. Example of errors (they don't happen together in the same build):

```
Msbuild.Tests.Utilities net11.0 failed with 1 error(s) (18.4s)
C:\Users\ygerges\Desktop\sdk\test\Common\Program.cs(41,35): error CS0234: The type or namespace name 'ConsoleClient' does not exist in the namespace 'Xunit' (are you missing an assembly reference?)```
```

```
Microsoft.DotNet.HotReload.Client.Tests net472 failed with 1 error(s) (24.6s)
C:\Users\ygerges\Desktop\sdk\.dotnet\sdk\11.0.100-alpha.1.26064.118\Microsoft.Common.CurrentVersion.targets(5496,5): error MSB3030: Could not copy the file "C:\Users\ygerges\Desktop\sdk\artifacts\obj\Microsoft.DotNet.HotReload.Client.Tests\Debug\Microsoft.DotNet.HotReload.Client.Tests.dll.config" because it was not found.```