dotnet / dotnet/BenchmarkDotNet
A benchmark job with a Native AOT tool chain fails on package restore when package source mapping is used
- Dominant language
- C#
- Stars
- 11.5k
- Forks
- 1.1k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 11
Description
(BenchmarkDotNet v0.14 on Windows 11.)
I have a solution that uses [NuGet package source mapping](https://learn.microsoft.com/en-us/nuget/consume-packages/package-source-mapping). There's a benchmark project with a `Program.cs` that looks like this:
```csharp
public static class Program
{
public static void Main(string[] args)
{
var config = GetConfig();
BenchmarkSwitcher
.FromAssembly(Assembly.GetExecutingAssembly())
.Run(args, config);
}
private static ManualConfig GetConfig()
{
var job = Job.Default;
var config = ManualConfig.Create(DefaultConfig.Instance)
.AddJob(
job.WithToolchain(CsProjCoreToolchain.NetCoreApp80).AsBaseline(),
Job.Default.WithRuntime(NativeAotRuntime.Net80),
job.WithToolchain(CsProjCoreToolchain.NetCoreApp60)
)
.AddDiagnoser(MemoryDiagnoser.Default);
if (Environment.OSVersion.Platform == PlatformID.Win32NT)
{
config.AddJob(job.WithToolchain(CsProjClassicNetToolchain.Net481));
}
config.SummaryStyle = SummaryStyle.Default
.WithRatioStyle(RatioStyle.Percentage);
config.AddValidator(JitOptimizationsValidator.FailOnError); // Fail when any of the referenced assemblies are not optimized
config.WithOrderer(new DefaultOrderer(SummaryOrderPolicy.FastestToSlowest));
return config;
}
}
```
So I'm running benchmarks against .NET 8 as my baseline, .NET 8 with Native AOT, .NET 6 and .NET 4.8.1 when on Windows. When running this using `dotnet build -c Release && dotnet run -c Release -f net8.0` the console shows this for the "Native AOT on .NET 8" job:
```
// Build Error: Standard output:
Standard error:
Determining projects to restore...
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\AtleX.HaveIBeenPwned.Benchmarks.csproj : error NU1100: Unable to resolve 'Microsoft.NETCore.App.Runtime.win-x64 (= 6.0.30)' for 'net6.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed. [C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj]
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj : error NU1100: Unable to resolve 'Microsoft.DotNet.ILCompiler (>= 8.0.5)' for 'net8.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed.
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned\AtleX.HaveIBeenPwned.csproj : error NU1100: Unable to resolve 'Microsoft.NETCore.App.Runtime.win-x64 (= 6.0.30)' for 'net6.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed. [C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj]
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned\AtleX.HaveIBeenPwned.csproj : error NU1100: Unable to resolve 'Microsoft.WindowsDesktop.App.Runtime.win-x64 (= 6.0.30)' for 'net6.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed. [C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj]
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned\AtleX.HaveIBeenPwned.csproj : error NU1100: Unable to resolve 'Microsoft.AspNetCore.App.Runtime.win-x64 (= 6.0.30)' for 'net6.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed. [C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj]
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned\AtleX.HaveIBeenPwned.csproj : error NU1100: Unable to resolve 'Microsoft.NETCore.App.Runtime.win-x64 (= 8.0.5)' for 'net8.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed. [C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj]
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned\AtleX.HaveIBeenPwned.csproj : error NU1100: Unable to resolve 'Microsoft.WindowsDesktop.App.Runtime.win-x64 (= 8.0.5)' for 'net8.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed. [C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj]
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned\AtleX.HaveIBeenPwned.csproj : error NU1100: Unable to resolve 'Microsoft.AspNetCore.App.Runtime.win-x64 (= 8.0.5)' for 'net8.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed. [C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj]
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj : error NU1100: Unable to resolve 'Microsoft.DotNet.ILCompiler (>= 8.0.5)' for 'net8.0/win-x64'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed.
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\AtleX.HaveIBeenPwned.Benchmarks.csproj : error NU1100: Unable to resolve 'Microsoft.WindowsDesktop.App.Runtime.win-x64 (= 6.0.30)' for 'net6.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed. [C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj]
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj : error NU1100: Unable to resolve 'Microsoft.NETCore.App.Runtime.win-x64 (= 8.0.5)' for 'net8.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed.
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj : error NU1100: Unable to resolve 'Microsoft.WindowsDesktop.App.Runtime.win-x64 (= 8.0.5)' for 'net8.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed.
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\AtleX.HaveIBeenPwned.Benchmarks.csproj : error NU1100: Unable to resolve 'Microsoft.AspNetCore.App.Runtime.win-x64 (= 6.0.30)' for 'net6.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed. [C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj]
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj : error NU1100: Unable to resolve 'Microsoft.AspNetCore.App.Runtime.win-x64 (= 8.0.5)' for 'net8.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed.
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj : error NU1100: Unable to resolve 'runtime.win-x64.Microsoft.DotNet.ILCompiler (= 8.0.5)' for 'net8.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed.
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\AtleX.HaveIBeenPwned.Benchmarks.csproj : error NU1100: Unable to resolve 'Microsoft.NETCore.App.Runtime.win-x64 (= 8.0.5)' for 'net8.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed. [C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj]
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\AtleX.HaveIBeenPwned.Benchmarks.csproj : error NU1100: Unable to resolve 'Microsoft.WindowsDesktop.App.Runtime.win-x64 (= 8.0.5)' for 'net8.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed. [C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj]
C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\AtleX.HaveIBeenPwned.Benchmarks.csproj : error NU1100: Unable to resolve 'Microsoft.AspNetCore.App.Runtime.win-x64 (= 8.0.5)' for 'net8.0'. PackageSourceMapping is enabled, the following source(s) were not considered: nativeAotNuGetFeed. [C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj]
Failed to restore C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned\AtleX.HaveIBeenPwned.csproj (in 188 ms).
Failed to restore C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6\BenchmarkDotNet.Autogenerated.csproj (in 188 ms).
Failed to restore C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\AtleX.HaveIBeenPwned.Benchmarks.csproj (in 205 ms).
// BenchmarkDotNet has failed to build the auto-generated boilerplate code.
// It can be found in C:\Git\AtleX.HaveIBeenPwned\src\AtleX.HaveIBeenPwned.Benchmarks\bin\Release\net8.0\581d3e4e-bc43-4985-a202-3b1b870531e6
// Please follow the troubleshooting guide: https://benchmarkdotnet.org/articles/guides/troubleshooting.html
```
This also happens when there's only one job with the `NativeAotRuntime.Net80` tool chain and not the others.
I can solve this by changing my `nuget.config` and add a package source mapping with that `nativeAotNuGetFeed` name:
```xml
```
This allows the benchmarks to run successfully. And after it has run once on a machine, I can remove the `packageSource` mapping for `nativeAotNuGetFeed` again and it will continue to run. However, it will fail again on any machine that hasn't run the Native AOT benchmarks before.
For reproduction the code I ran is on Github: https://github.com/akamsteeg/AtleX.HaveIBeenPwned/commit/c6c6cd483a58044db104b94ca280db8612b20f03. Running it with `dotnet build -c Release && dotnet run -c Release -f net8.0` and picking any benchmark should reproduce this. ⚠️ Remember that the benchmark continues to succeed on the same machine once it has managed to restore the packages once. Probably clearing package caches will help, but I haven't tried that.
Contributor guide
Assessment
This issue has not been assessed yet.