dotnet / dotnet/BenchmarkDotNet

Stop gracefully when benchmark fails with an exception that can not be caught by a catch block

Open
#1,661 12 comments 0 reactions 1 assignee Claimed by @iAmK1 View on GitHub
bug good first issue up-for-grabs
Dominant language
C#
Stars
11.5k
Forks
1.1k
Avg merge
4d 11h
Merged PRs (30d)
11

Description

The original issue was reported in https://github.com/dotnet/performance/issues/1701:

```log
// Benchmark Process Environment Information:
// Runtime=.NET 6.0.0 (6.0.21.11801), X64 RyuJIT
// GC=Concurrent Workstation
// Job: Job-AQTFSE(PowerPlanMode=00000000-0000-0000-0000-000000000000, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1)

OverheadJitting 1: 1 op, 537000.00 ns, 537.0000 us/op
WorkloadJitting 1: 1 op, 6767100.00 ns, 6.7671 ms/op

OverheadJitting 2: 16 op, 731100.00 ns, 45.6938 us/op
WorkloadJitting 2: 16 op, 2527400.00 ns, 157.9625 us/op

WorkloadPilot 1: 16 op, 7800.00 ns, 487.5000 ns/op
// Benchmark Process 3336 has exited with code -1073741819
Unhandled exception. System.InvalidOperationException: Sequence contains no matching element
at System.Linq.ThrowHelper.ThrowNoMatchException()
at BenchmarkDotNet.Running.BenchmarkRunnerClean.Execute(ILogger logger, BenchmarkCase benchmarkCase, BenchmarkId benchmarkId, IToolchain toolchain, BuildResult buildResult, IResolver resolver)
at BenchmarkDotNet.Running.BenchmarkRunnerClean.RunCore(BenchmarkCase benchmarkCase, BenchmarkId benchmarkId, ILogger logger, IResolver resolver, BuildResult buildResult)
at BenchmarkDotNet.Running.BenchmarkRunnerClean.Run(BenchmarkRunInfo benchmarkRunInfo, Dictionary`2 buildResults, IResolver resolver, ILogger logger, List`1 artifactsToCleanup, String resultsFolderPath, String logFilePath, StartedClock& runChronometer)
at BenchmarkDotNet.Running.BenchmarkRunnerClean.Run(BenchmarkRunInfo[] benchmarkRunInfos)
at BenchmarkDotNet.Running.BenchmarkSwitcher.RunWithDirtyAssemblyResolveHelper(String[] args, IConfig config)
at BenchmarkDotNet.Running.BenchmarkSwitcher.Run(String[] args, IConfig config)
```

The benchmark process exited with `-1073741819` which translates to `0xc0000005` which is **Access Violation**.

BenchmarkDotNet should:

* detect failures for exceptions that can not be caught (most probably [this](https://github.com/dotnet/BenchmarkDotNet/blob/8c28c871ea0422a1489bda153880046cc90ee136/src/BenchmarkDotNet/Running/BenchmarkRunnerClean.cs#L408-L420) logic has to be modified)
* stop gracefully in such cases

Whoever is willing to work on that should most probably start by creating a benchmark that throws an exception that can not be caught by a catch block (`AccessViolation` would be the best), then create a test for it similar to [this one](https://github.com/dotnet/BenchmarkDotNet/blob/8c28c871ea0422a1489bda153880046cc90ee136/tests/BenchmarkDotNet.IntegrationTests/StandardErrorTests.cs#L26-L41) and fix `BenchmarkRunnerClean.Execute`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.