dotnet / dotnet/BenchmarkDotNet

Enabling AllowVeryLargeObjects has no effect on Framework benchmark process with Core host process

Open
#1,519 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
11.5k
Forks
1.1k
Avg merge
4d 11h
Merged PRs (30d)
11

Description

I am using v0.12.1 in a console project to run benchmarks on both .NET Core and Framework. I have configured the `.csproj` as follows...
```xml
netcoreapp3.1;net48
```
...and then launch it with...

dotnet run --configuration release --framework netcoreapp3.1 benchmark

This works fine. (That trailing `benchmark` parameter is handled by my code to then call `BenchmarkDotNet.Running.BenchmarkRunner.Run();`)

After increasing a `[Params()]` value that controls the `Capacity` of a `List<>`, I started getting failures in the Framework benchmarks...
> OutOfMemoryException!
> BenchmarkDotNet continues to run additional iterations until desired accuracy level is achieved. It's possible only if the benchmark method doesn't have any side-effects.
> If your benchmark allocates memory and keeps it alive, you are creating a memory leak.
> You should redesign your benchmark and remove the side-effects. You can use `OperationsPerInvoke`, `IterationSetup` and `IterationCleanup` to do that.
>
>

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.OutOfMemoryException: Array dimensions exceeded supported range.

> at System.Collections.Generic.List`1..ctor(Int32 capacity)

To remedy this, I changed my configuration from...
```c#
[SimpleJob(RuntimeMoniker.NetCoreApp31)]
[SimpleJob(RuntimeMoniker.Net48)]
public class Benchmarks
{
```
...to...
```c#
Job baseJob = Job.Dry;
IConfig config = DefaultConfig.Instance
.AddJob(baseJob.WithRuntime(ClrRuntime.Net48).WithGcAllowVeryLargeObjects(true))
.AddJob(baseJob.WithRuntime(CoreRuntime.Core31));

BenchmarkDotNet.Running.BenchmarkRunner.Run(config);
```
What I found, though, is that the Framework benchmarks all still failed with that same error when passing `--framework netcoreapp3.1` to `dotnet run`. Immediately before the error the log even shows that `AllowVeryLargeObjects` is enabled...

> // BeforeAnythingElse
>
> // Benchmark Process Environment Information:
> // Runtime=.NET Framework 4.8 (4.8.4200.0), X64 RyuJIT
> // GC=Concurrent Workstation
> // Job: Dry(AllowVeryLargeObjects=True, IterationCount=1, LaunchCount=1, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=1)

...but only after changing to `--framework net48` would the benchmarks succeed.

My question is, is this expected behavior and, if so, can you help me understand why that is? I've read [Toolchains](https://benchmarkdotnet.org/articles/configs/toolchains.html), among other articles, but I'm still not seeing how the runtime of the **host process** would affect a runtime-specific **benchmark process** in this way.

I should add that my program also takes a `test` command line parameter, and my project has an `App.config` file configured so that code path, too, can successfully create very large arrays. With that file in place and running BenchmarkDotNet with `--framework netcoreapp3.1` and `KeepBenchmarkFiles(true)` added to `config` above, I see that same `` directive does get propagated to...

- `bin\release\net48\MyProject.exe.config`,
- `bin\release\netcoreapp3.1\Dry\bin\Release\net48\Dry.exe.config`,
- `bin\release\netcoreapp3.1\Dry\bin\Release\netcoreapp3.1\MyProject.dll.config`, and
- `bin\release\netcoreapp3.1\MyProject.dll.config`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.