"Out of memory" error when running GC benchmarks
- Dominant language
- F#
- Stars
- 773
- Forks
- 301
- Avg merge
- 6d 9h
- Merged PRs (30d)
- 15
Description
I'm getting an `Out of memory` error when I try to run the HighMemory Server scenario with .NET 6.0 using GC.Infrastructure.exe.
Here is the .yaml file I'm using:
HighMemory.yaml
```
runs:
server:
override_parameters:
tlgb: 2
sohsi: 50
workstation:
override_parameters:
tlgb: 2
sohsi: 50
environment_variables:
COMPlus_GCServer: 0
# Top level microbenchmark configuration.
gcperfsim_configurations:
parameters:
tc: 36
tagb: 540
tlgb: 2
lohar: 0
pohar: 0
sohsr: 100-4000
lohsr: 102400-204800
pohsr: 100-204800
sohsi: 0
lohsi: 0
pohsi: 0
sohpi: 0
lohpi: 0
sohfi: 0
lohfi: 0
pohfi: 0
allocType: reference
testKind: time
gcperfsim_path: D:\ac\20230927\performance\artifacts\bin\GCPerfSim\Release\net6.0\GCPerfSim.dll
coreruns:
baseline:
path: D:\ac\20230927\runtime-6.0\artifacts\tests\coreclr\windows.x64.Release\Tests\Core_Root\corerun.exe
environment_variables:
COMPlus_GCName: clrgc.dll
run:
path: D:\ac\20230927\runtime-8.0\artifacts\tests\coreclr\windows.x64.Release\Tests\Core_Root\corerun.exe
environment_variables:
COMPlus_GCName: clrgc.dll
environment:
environment_variables:
COMPlus_GCServer: 1
COMPlus_GCHeapCount: 18
COMPlus_GCName: clrgc.dll
COMPlus_GCHeapHardLimit: "0x100000000"
COMPlus_GCTotalPhysicalMemory: "0x100000000"
default_max_seconds: 300
iterations: 3
# Configurations that involve capturing a trace.
trace_configurations:
type: gc # Choices: gc, verbose, cpu, threadtime, none
output:
path: D:\ac\20230927\GCPerfSim\HighMemory_NormalServer_debug
columns:
- Count
- total allocated (mb)
- total pause time (msec)
- PctTimePausedInGC
- FirstToLastGCSeconds
- HeapSizeAfter_Mean
- HeapSizeBeforeMB_Mean
- PauseDurationMSec_95PWhereIsGen0
- PauseDurationMSec_95PWhereIsGen1
- PauseDurationMSec_95PWhereIsBackground
- PauseDurationMSec_95PWhereIsBlockingGen2
- CountIsBlockingGen2
- HeapCount
- TotalNumberGCs
- TotalAllocatedMB
- Speed
- PauseDurationMSec_MeanWhereIsEphemeral
- PauseDurationMSec_MeanWhereIsBackground
- PauseDurationMSec_MeanWhereIsBlockingGen2
- PauseDurationSeconds_SumWhereIsGen1
- PauseDurationSeconds_Sum
- CountIsGen1
- ExecutionTimeMSec
percentage_disk_remaining_to_stop_per_run: 0
all_columns:
- Count
- total allocated (mb)
- total pause time (msec)
- PctTimePausedInGC
- FirstToLastGCSeconds
- HeapSizeAfter_Mean
- HeapSizeBeforeMB_Mean
- PauseDurationMSec_95PWhereIsGen0
- PauseDurationMSec_95PWhereIsGen1
- PauseDurationMSec_95PWhereIsBackground
- PauseDurationMSec_95PWhereIsBlockingGen2
- CountIsBlockingGen2
- HeapCount
- TotalNumberGCs
- TotalAllocatedMB
- Speed
- PauseDurationMSec_MeanWhereIsEphemeral
- PauseDurationMSec_MeanWhereIsBackground
- PauseDurationMSec_MeanWhereIsBlockingGen2
- PauseDurationSeconds_SumWhereIsGen1
- PauseDurationSeconds_Sum
- CountIsGen1
- ExecutionTimeMSec
- Count
- PctTimePausedInGC
- FirstToLastGCSeconds
- HeapSizeAfter_Mean
- HeapSizeBeforeMB_Mean
- PauseDurationMSec_95PWhereIsGen0
- PauseDurationMSec_95PWhereIsGen1
- PauseDurationMSec_95PWhereIsBackground
- PauseDurationMSec_95PWhereIsBlockingGen2
- CountIsBlockingGen2
- HeapCount
- TotalNumberGCs
- TotalAllocatedMB
- Speed
- PauseDurationMSec_MeanWhereIsEphemeral
- PauseDurationSeconds_SumWhereIsGen1
- PauseDurationSeconds_Sum
- CountIsGen1
- ExecutionTimeMSec
formats:
- markdown
- json
name: HighMemory_NormalServer
trace_configurations:
type: gc
```
I'm running GC.Infrastructure from the build directory:
```
D:\ac\20230927\performance\artifacts\bin\GC.Infrastructure\Release\net7.0>.\GC.Infrastructure.exe gcperfsim --configuration D:\ac\performance\src\benchmarks\gc\GC.Infrastructure\Configurations\GCPerfSim\HighMemory.yaml
```
This does not happen every time, but the majority of the time I run that command, I get the OoM error:
```
(10/4/2023 2:20:09 PM) Running HighMemory_NormalServer: baseline for server - Iteration: 0
HighMemory_NormalServer: baseline for server failed with:
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
Out of memory.
```
I attempted to troubleshoot/debug this, but I'm unable to reproduce the OoM if I run GCPerfSim directly:
```
PS D:\ac\20230927\runtime-6.0\artifacts\tests\coreclr\windows.x64.Release\Tests\Core_Root> .\corerun.exe D:\ac\20230927\performance\artifacts\bin\GCPerfSim\Release\net6.0\GCPerfSim.dll -tc 36 -tagb 540 -tlgb 2 -lohar 0 -pohar 0 -sohsr 100-4000 -lohsr 102400-204800 -pohsr 100-204800 -sohsi 50 -lohsi 0 -pohsi 0 -sohpi 0 -lohpi 0 -sohfi 0 -lohfi 0 -pohfi 0 -allocType reference -testKind time
```
When running GCPerfSim directly with a .NET 6.0 corerun.exe, I am unable to reproduce the OoM error.
It seems like this could be an issue with GC.Infrastructure.exe or PerfView since I cannot reproduce the OoM with a standalone corerun.exe, but that's just a guess.
Please let me know if I can clarify anything above. Thanks!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.