dotnet / dotnet/performance

`Sequence contains no elements` Exception in `Reporter.cs` in F# benchmark

Open
#2,624 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
F#
Stars
773
Forks
301
Avg merge
6d 9h
Merged PRs (30d)
15

Description

There's an exception thrown in [`tools/Reporting/Reporting/Reporter.cs`](https://github.com/dotnet/performance/blob/11a3e1e86c6c09fcf76289959dab0ea68fd615c7/src/tools/Reporting/Reporting/Reporter.cs#L129) that fails the scenario benchmark for [F# Compiler Service](https://github.com/dotnet/performance/tree/main/src/scenarios/fsharpcompilerservice). It seemed like an intermittent issue initially but now it seems to be happening every time - on **Ubuntu.1804.Amd64.Open**

The issues is that `Counter.Results` is empty and therefore LINQ aggregations (`Min/Max/Average`) throw `System.InvalidOperationException: Sequence contains no elements`

- [Example CI run](https://dev.azure.com/dnceng-public/public/_build/results?buildId=38304&view=ms.vss-test-web.build-test-results-tab&runId=764332&resultId=100069&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab)
- [Full log](https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-performance-refs-heads-main-0146089f16284ee5a5/F%23%20Compiler%20Service%20Clean%20Build/1/console.8ef13c12.log?helixlogtype=result)

Full stack trace:
```
[2022/10/03 12:20:50][INFO] Parsing traces/F#_Compiler_Service_Clean_Build_startup.trace.zip
[2022/10/03 12:20:50][INFO] Unhandled exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
[2022/10/03 12:20:50][INFO] ---> System.InvalidOperationException: Sequence contains no elements
[2022/10/03 12:20:50][INFO] at System.Linq.ThrowHelper.ThrowNoElementsException()
[2022/10/03 12:20:50][INFO] at System.Linq.Enumerable.MaxFloat[T](IEnumerable`1 source)
[2022/10/03 12:20:50][INFO] at Reporting.Reporter.<>c.b__10_4(Counter c) in /__w/1/s/src/tools/Reporting/Reporting/Reporter.cs:line 129
[2022/10/03 12:20:50][INFO] at System.Linq.Enumerable.MaxInteger[TSource,TResult](IEnumerable`1 source, Func`2 selector)
[2022/10/03 12:20:50][INFO] at Reporting.Reporter.WriteResultTable() in /__w/1/s/src/tools/Reporting/Reporting/Reporter.cs:line 129
[2022/10/03 12:20:50][INFO] at ScenarioMeasurement.Startup.CreateTestReport(String scenarioName, IEnumerable`1 counters, String reportJsonPath, Logger logger) in /__w/1/s/src/tools/ScenarioMeasurement/Startup/Startup.cs:line 533
[2022/10/03 12:20:50][INFO] at ScenarioMeasurement.Startup.Main(String appExe, MetricType metricType, String scenarioName, String traceName, Boolean processWillExit, Int32 iterations, String iterationSetup, String setupArgs, String iterationCleanup, String cleanupArgs, Int32 timeout, Int32 measurementDelay, String appArgs, String logFileName, String workingDir, Boolean warmup, Boolean guiApp, Boolean skipProfileIteration, String reportJsonPath, String traceDirectory, String environmentVariables, String innerLoopCommand, String innerLoopCommandArgs, Boolean runWithoutExit, Int32 hotReloadIters, Boolean skipMeasurementIteration, Boolean parseOnly, Boolean runWithDotnet) in /__w/1/s/src/tools/ScenarioMeasurement/Startup/Startup.cs:line 337
[2022/10/03 12:20:50][INFO] at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
[2022/10/03 12:20:50][INFO] at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
[2022/10/03 12:20:50][INFO] --- End of inner exception stack trace ---
[2022/10/03 12:20:50][INFO] at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
[2022/10/03 12:20:50][INFO] at System.Reflection.RuntimeMethodInfo.InvokeWithManyArguments(RuntimeMethodInfo mi, Int32 argCount, Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
[2022/10/03 12:20:50][INFO] at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
[2022/10/03 12:20:50][INFO] at System.CommandLine.Invocation.ModelBindingCommandHandler.InvokeAsync(InvocationContext context)
[2022/10/03 12:20:50][INFO] at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<b__0>d.MoveNext()
[2022/10/03 12:20:50][INFO] --- End of stack trace from previous location ---
[2022/10/03 12:20:50][INFO] at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<b__19_0>d.MoveNext()
[2022/10/03 12:20:50][INFO] --- End of stack trace from previous location ---
[2022/10/03 12:20:50][INFO] at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass14_0.<b__0>d.MoveNext()
[2022/10/03 12:20:50][INFO] --- End of stack trace from previous location ---
[2022/10/03 12:20:50][INFO] at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass22_0.<b__0>d.MoveNext()
[2022/10/03 12:20:50][INFO] --- End of stack trace from previous location ---
[2022/10/03 12:20:50][INFO] at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass21_0.<b__0>d.MoveNext()
[2022/10/03 12:20:50][INFO] --- End of stack trace from previous location ---
[2022/10/03 12:20:50][INFO] at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<b__20_0>d.MoveNext()
[2022/10/03 12:20:50][INFO] --- End of stack trace from previous location ---
[2022/10/03 12:20:50][INFO] at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<b__18_0>d.MoveNext()
[2022/10/03 12:20:50][INFO] --- End of stack trace from previous location ---
[2022/10/03 12:20:50][INFO] at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<b__10_0>d.MoveNext()
[2022/10/03 12:20:50][INFO] --- End of stack trace from previous location ---
[2022/10/03 12:20:50][INFO] at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<b__9_0>d.MoveNext()
[2022/10/03 12:20:50][INFO] --- End of stack trace from previous location ---
[2022/10/03 12:20:50][INFO] at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass5_0.<b__0>d.MoveNext()
[2022/10/03 12:20:50][INFO] --- End of stack trace from previous location ---
[2022/10/03 12:20:50][INFO] at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass12_0.<b__0>d.MoveNext()
[2022/10/03 12:20:50][ERROR] Process exited with status 1
[2022/10/03 12:20:50][INFO] Run failure registered
Traceback (most recent call last):
File "test.py", line 14, in
Runner(traits).run()
File "/datadisks/disk1/work/9D9C08CB/p/shared/runner.py", line 244, in run
startup.runtests(self.traits)
File "/datadisks/disk1/work/9D9C08CB/p/shared/startup.py", line 152, in runtests
RunCommand(startup_args, verbose=True).run()
File "/datadisks/disk1/work/9D9C08CB/p/scripts/performance/common.py", line 248, in run
returncode, quoted_cmdline)
subprocess.CalledProcessError: Command '$ /datadisks/disk1/work/9D9C08CB/p/startup/Startup --app-exe dotnet --metric-type ProcessTime --trace-name "F# Compiler Service Clean Build_startup" --gui-app false --process-will-exit true --iterations 1 --timeout 600 --warmup true --working-dir app --report-json-path traces/perf-lab-report.json --trace-directory traces --scenario-name "F# Compiler Service Clean Build" --app-args build --environment-variables MSBUILDDISABLENODEREUSE=1;DOTNET_MULTILEVEL_LOOKUP=0 --iteration-setup python3 --setup-args " /datadisks/disk1/work/9D9C08CB/p/shared/sdk_iteration_setup.py setup_build" --iteration-cleanup python3 --cleanup-args " /datadisks/disk1/work/9D9C08CB/p/shared/sdk_iteration_setup.py cleanup"' returned non-zero exit status 1.
```

Is this something that can be fixed in the F# scenario? If so, how?

Otherwise here's an example fix that would ignore empty results:
- https://github.com/0101/performance/commit/6a10733d50da0e8d3a223b7924e98fd9d0d46709 + https://github.com/0101/performance/commit/1e8a12263b4302252004646e4d1bfffc614a6c36

If we don't want to ignore empty results it might be worth it to throw a custom exception that would explain what's happening and how to fix it.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.