dotnet / dotnet/BenchmarkDotNet
Stop gracefully when benchmark fails with an exception that can not be caught by a catch block
- Dominant language
- C#
- Stars
- 11.5k
- Forks
- 1.1k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 11
Description
The original issue was reported in https://github.com/dotnet/performance/issues/1701:
```log
// Benchmark Process Environment Information:
// Runtime=.NET 6.0.0 (6.0.21.11801), X64 RyuJIT
// GC=Concurrent Workstation
// Job: Job-AQTFSE(PowerPlanMode=00000000-0000-0000-0000-000000000000, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1)
OverheadJitting 1: 1 op, 537000.00 ns, 537.0000 us/op
WorkloadJitting 1: 1 op, 6767100.00 ns, 6.7671 ms/op
OverheadJitting 2: 16 op, 731100.00 ns, 45.6938 us/op
WorkloadJitting 2: 16 op, 2527400.00 ns, 157.9625 us/op
WorkloadPilot 1: 16 op, 7800.00 ns, 487.5000 ns/op
// Benchmark Process 3336 has exited with code -1073741819
Unhandled exception. System.InvalidOperationException: Sequence contains no matching element
at System.Linq.ThrowHelper.ThrowNoMatchException()
at BenchmarkDotNet.Running.BenchmarkRunnerClean.Execute(ILogger logger, BenchmarkCase benchmarkCase, BenchmarkId benchmarkId, IToolchain toolchain, BuildResult buildResult, IResolver resolver)
at BenchmarkDotNet.Running.BenchmarkRunnerClean.RunCore(BenchmarkCase benchmarkCase, BenchmarkId benchmarkId, ILogger logger, IResolver resolver, BuildResult buildResult)
at BenchmarkDotNet.Running.BenchmarkRunnerClean.Run(BenchmarkRunInfo benchmarkRunInfo, Dictionary`2 buildResults, IResolver resolver, ILogger logger, List`1 artifactsToCleanup, String resultsFolderPath, String logFilePath, StartedClock& runChronometer)
at BenchmarkDotNet.Running.BenchmarkRunnerClean.Run(BenchmarkRunInfo[] benchmarkRunInfos)
at BenchmarkDotNet.Running.BenchmarkSwitcher.RunWithDirtyAssemblyResolveHelper(String[] args, IConfig config)
at BenchmarkDotNet.Running.BenchmarkSwitcher.Run(String[] args, IConfig config)
```
The benchmark process exited with `-1073741819` which translates to `0xc0000005` which is **Access Violation**.
BenchmarkDotNet should:
* detect failures for exceptions that can not be caught (most probably [this](https://github.com/dotnet/BenchmarkDotNet/blob/8c28c871ea0422a1489bda153880046cc90ee136/src/BenchmarkDotNet/Running/BenchmarkRunnerClean.cs#L408-L420) logic has to be modified)
* stop gracefully in such cases
Whoever is willing to work on that should most probably start by creating a benchmark that throws an exception that can not be caught by a catch block (`AccessViolation` would be the best), then create a test for it similar to [this one](https://github.com/dotnet/BenchmarkDotNet/blob/8c28c871ea0422a1489bda153880046cc90ee136/tests/BenchmarkDotNet.IntegrationTests/StandardErrorTests.cs#L26-L41) and fix `BenchmarkRunnerClean.Execute`
Contributor guide
Assessment
This issue has not been assessed yet.