dotnet / dotnet/performance

"Out of memory" error when running GC benchmarks

Open
#3,382 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
F#
Stars
773
Forks
301
Avg merge
6d 9h
Merged PRs (30d)
15

Description

I'm getting an `Out of memory` error when I try to run the HighMemory Server scenario with .NET 6.0 using GC.Infrastructure.exe.

Here is the .yaml file I'm using:

HighMemory.yaml

```
runs:

server:
override_parameters:
tlgb: 2
sohsi: 50

workstation:
override_parameters:
tlgb: 2
sohsi: 50
environment_variables:
COMPlus_GCServer: 0

# Top level microbenchmark configuration.
gcperfsim_configurations:
parameters:
tc: 36
tagb: 540
tlgb: 2
lohar: 0
pohar: 0
sohsr: 100-4000
lohsr: 102400-204800
pohsr: 100-204800
sohsi: 0
lohsi: 0
pohsi: 0
sohpi: 0
lohpi: 0
sohfi: 0
lohfi: 0
pohfi: 0
allocType: reference
testKind: time
gcperfsim_path: D:\ac\20230927\performance\artifacts\bin\GCPerfSim\Release\net6.0\GCPerfSim.dll

coreruns:
baseline:
path: D:\ac\20230927\runtime-6.0\artifacts\tests\coreclr\windows.x64.Release\Tests\Core_Root\corerun.exe
environment_variables:
COMPlus_GCName: clrgc.dll
run:
path: D:\ac\20230927\runtime-8.0\artifacts\tests\coreclr\windows.x64.Release\Tests\Core_Root\corerun.exe
environment_variables:
COMPlus_GCName: clrgc.dll

environment:
environment_variables:
COMPlus_GCServer: 1
COMPlus_GCHeapCount: 18
COMPlus_GCName: clrgc.dll
COMPlus_GCHeapHardLimit: "0x100000000"
COMPlus_GCTotalPhysicalMemory: "0x100000000"

default_max_seconds: 300
iterations: 3

# Configurations that involve capturing a trace.
trace_configurations:
type: gc # Choices: gc, verbose, cpu, threadtime, none

output:
path: D:\ac\20230927\GCPerfSim\HighMemory_NormalServer_debug
columns:
- Count
- total allocated (mb)
- total pause time (msec)
- PctTimePausedInGC
- FirstToLastGCSeconds
- HeapSizeAfter_Mean
- HeapSizeBeforeMB_Mean
- PauseDurationMSec_95PWhereIsGen0
- PauseDurationMSec_95PWhereIsGen1
- PauseDurationMSec_95PWhereIsBackground
- PauseDurationMSec_95PWhereIsBlockingGen2
- CountIsBlockingGen2
- HeapCount
- TotalNumberGCs
- TotalAllocatedMB
- Speed
- PauseDurationMSec_MeanWhereIsEphemeral
- PauseDurationMSec_MeanWhereIsBackground
- PauseDurationMSec_MeanWhereIsBlockingGen2
- PauseDurationSeconds_SumWhereIsGen1
- PauseDurationSeconds_Sum
- CountIsGen1
- ExecutionTimeMSec
percentage_disk_remaining_to_stop_per_run: 0
all_columns:
- Count
- total allocated (mb)
- total pause time (msec)
- PctTimePausedInGC
- FirstToLastGCSeconds
- HeapSizeAfter_Mean
- HeapSizeBeforeMB_Mean
- PauseDurationMSec_95PWhereIsGen0
- PauseDurationMSec_95PWhereIsGen1
- PauseDurationMSec_95PWhereIsBackground
- PauseDurationMSec_95PWhereIsBlockingGen2
- CountIsBlockingGen2
- HeapCount
- TotalNumberGCs
- TotalAllocatedMB
- Speed
- PauseDurationMSec_MeanWhereIsEphemeral
- PauseDurationMSec_MeanWhereIsBackground
- PauseDurationMSec_MeanWhereIsBlockingGen2
- PauseDurationSeconds_SumWhereIsGen1
- PauseDurationSeconds_Sum
- CountIsGen1
- ExecutionTimeMSec
- Count
- PctTimePausedInGC
- FirstToLastGCSeconds
- HeapSizeAfter_Mean
- HeapSizeBeforeMB_Mean
- PauseDurationMSec_95PWhereIsGen0
- PauseDurationMSec_95PWhereIsGen1
- PauseDurationMSec_95PWhereIsBackground
- PauseDurationMSec_95PWhereIsBlockingGen2
- CountIsBlockingGen2
- HeapCount
- TotalNumberGCs
- TotalAllocatedMB
- Speed
- PauseDurationMSec_MeanWhereIsEphemeral
- PauseDurationSeconds_SumWhereIsGen1
- PauseDurationSeconds_Sum
- CountIsGen1
- ExecutionTimeMSec
formats:
- markdown
- json
name: HighMemory_NormalServer
trace_configurations:
type: gc
```

I'm running GC.Infrastructure from the build directory:

```
D:\ac\20230927\performance\artifacts\bin\GC.Infrastructure\Release\net7.0>.\GC.Infrastructure.exe gcperfsim --configuration D:\ac\performance\src\benchmarks\gc\GC.Infrastructure\Configurations\GCPerfSim\HighMemory.yaml
```

This does not happen every time, but the majority of the time I run that command, I get the OoM error:

```
(10/4/2023 2:20:09 PM) Running HighMemory_NormalServer: baseline for server - Iteration: 0
HighMemory_NormalServer: baseline for server failed with:
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
```

I attempted to troubleshoot/debug this, but I'm unable to reproduce the OoM if I run GCPerfSim directly:

```
PS D:\ac\20230927\runtime-6.0\artifacts\tests\coreclr\windows.x64.Release\Tests\Core_Root> .\corerun.exe D:\ac\20230927\performance\artifacts\bin\GCPerfSim\Release\net6.0\GCPerfSim.dll -tc 36 -tagb 540 -tlgb 2 -lohar 0 -pohar 0 -sohsr 100-4000 -lohsr 102400-204800 -pohsr 100-204800 -sohsi 50 -lohsi 0 -pohsi 0 -sohpi 0 -lohpi 0 -sohfi 0 -lohfi 0 -pohfi 0 -allocType reference -testKind time
```

When running GCPerfSim directly with a .NET 6.0 corerun.exe, I am unable to reproduce the OoM error.

It seems like this could be an issue with GC.Infrastructure.exe or PerfView since I cannot reproduce the OoM with a standalone corerun.exe, but that's just a guess.

Please let me know if I can clarify anything above. Thanks!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.