dotnet / dotnet/arcade

RunTests target hides exception thrown during NUnit fixture setup

Open
#14,812 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
729
Forks
397
Avg merge
3d 15m
Merged PRs (30d)
149

Description

I hit a scenario where running tests via arcade hides some important messages indicating that test setup failed and some tests were skipped. Running tests normally using the dotnet sdk surfaces this information as expected. Repro steps:

- `git clone https://github.com/dotnet/cecil`
- `cd cecil`
- `git checkout ff87dfd51654949f6379059c258b7f8c13e10d29`
- `./build.sh -test` (succeeds without indicating there were any issues)
```
Downloading 'https://dotnet.microsoft.com/download/dotnet/scripts/v1/dotnet-install.sh'
Attempting to install 'sdk v9.0.100-preview.4.24267.66' from public_location.
dotnet-install: Attempting to download using primary link https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.4.24267.66/dotnet-sdk-9.0.100-preview.4.24267.66-linux-x64.tar.gz
dotnet-install: Remote file https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.4.24267.66/dotnet-sdk-9.0.100-preview.4.24267.66-linux-x64.tar.gz size is 240934655 bytes.
dotnet-install: Extracting archive from https://dotnetcli.azureedge.net/dotnet/Sdk/9.0.100-preview.4.24267.66/dotnet-sdk-9.0.100-preview.4.24267.66-linux-x64.tar.gz
dotnet-install: Downloaded file size is 240934655 bytes.
dotnet-install: The remote and local file sizes are equal.
dotnet-install: Installed version is 9.0.100-preview.4.24267.66
dotnet-install: Adding to current process PATH: `/home/svbomer/tmp/cecil/.dotnet`. Note: This change will be visible only when sourcing script.
dotnet-install: Note that the script does not resolve dependencies during installation.
dotnet-install: To check the list of dependencies, go to https://learn.microsoft.com/dotnet/core/install, select your operating system and check the "Dependencies" section.
dotnet-install: Installation finished successfully.

Build succeeded in 1.5s
Restore complete (1.1s)
Mono.Cecil succeeded (2.9s) → artifacts/bin/Mono.Cecil/Debug/netstandard2.0/Mono.Cecil.dll
Mono.Cecil.Rocks succeeded (0.8s) → artifacts/bin/Mono.Cecil.Rocks/Debug/netstandard2.0/Mono.Cecil.Rocks.dll
Mono.Cecil.Mdb succeeded (0.8s) → artifacts/bin/Mono.Cecil.Mdb/Debug/netstandard2.0/Mono.Cecil.Mdb.dll
Mono.Cecil.Pdb succeeded (1.0s) → artifacts/bin/Mono.Cecil.Pdb/Debug/netstandard2.0/Mono.Cecil.Pdb.dll
Mono.Cecil.Tests succeeded (2.4s) → artifacts/bin/Mono.Cecil.Tests/Debug/net9.0/Mono.Cecil.Tests.dll
Mono.Cecil.Mdb.Tests succeeded (0.6s) → artifacts/bin/Mono.Cecil.Mdb.Tests/Debug/net9.0/Mono.Cecil.Mdb.Tests.dll
Mono.Cecil.Pdb.Tests succeeded (0.7s) → artifacts/bin/Mono.Cecil.Pdb.Tests/Debug/net9.0/Mono.Cecil.Pdb.Tests.dll
Mono.Cecil.Rocks.Tests succeeded (0.8s) → artifacts/bin/Mono.Cecil.Rocks.Tests/Debug/net9.0/Mono.Cecil.Rocks.Tests.dll

Build succeeded in 34.5
```
- `.dotnet/dotnet test Mono.Cecil.sln` (logs a warning indicating some tests didn't run because setup failed):
```
Restore complete (1.3s)
Mono.Cecil succeeded (1.0s) → artifacts/bin/Mono.Cecil/Debug/netstandard2.0/Mono.Cecil.dll
Mono.Cecil.Mdb succeeded (0.2s) → artifacts/bin/Mono.Cecil.Mdb/Debug/netstandard2.0/Mono.Cecil.Mdb.dll
Mono.Cecil.Rocks succeeded (0.2s) → artifacts/bin/Mono.Cecil.Rocks/Debug/netstandard2.0/Mono.Cecil.Rocks.dll
Mono.Cecil.Pdb succeeded (0.2s) → artifacts/bin/Mono.Cecil.Pdb/Debug/netstandard2.0/Mono.Cecil.Pdb.dll
Mono.Cecil.Tests succeeded (0.7s) → artifacts/bin/Mono.Cecil.Tests/Debug/net9.0/Mono.Cecil.Tests.dll
Mono.Cecil.Tests test succeeded with 2 warning(s) (23.9s)
/home/svbomer/tmp/cecil/.dotnet/sdk/9.0.100-preview.4.24267.66/Microsoft.TestPlatform.targets(46,5): warning : SetUp failed for test fixture Mono.Cecil.Tests.ILProcessorTests
/home/svbomer/tmp/cecil/.dotnet/sdk/9.0.100-preview.4.24267.66/Microsoft.TestPlatform.targets(46,5): warning :
An exception was thrown while loading the test.
System.ArgumentException: Unknown framework version 9.5 (Parameter 'version')
at NUnit.Framework.Internal.RuntimeFramework.ThrowInvalidFrameworkVersion(Version version)
at NUnit.Framework.Internal.RuntimeFramework.InitFromFrameworkVersion(Version version)
at NUnit.Framework.Internal.RuntimeFramework..ctor(RuntimeType runtime, Version version)
at NUnit.Framework.Internal.RuntimeFramework.<>c.<.cctor>b__33_0()
at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
at System.Lazy`1.CreateValue()
at NUnit.Framework.Internal.RuntimeFramework.get_CurrentFramework()
at NUnit.Framework.Internal.PlatformHelper..ctor()
at NUnit.Framework.TestCaseAttribute.BuildFrom(IMethodInfo method, Test suite)+MoveNext()
at NUnit.Framework.Internal.Builders.DefaultTestCaseBuilder.BuildFrom(IMethodInfo method, Test parentSuite)
at NUnit.Framework.Internal.Builders.NUnitTestFixtureBuilder.BuildTestCase(IMethodInfo method, TestSuite suite)
at NUnit.Framework.Internal.Builders.NUnitTestFixtureBuilder.AddTestCasesToFixture(TestFixture fixture, IPreFilter filter)
at NUnit.Framework.Internal.Builders.NUnitTestFixtureBuilder.BuildFrom(ITypeInfo typeInfo, IPreFilter filter, ITestFixtureData testFixtureData)
at NUnit.Framework.TestFixtureAttribute.BuildFrom(ITypeInfo typeInfo, IPreFilter filter)+MoveNext()
at NUnit.Framework.Internal.Builders.DefaultSuiteBuilder.BuildFrom(ITypeInfo typeInfo, IPreFilter filter)
Mono.Cecil.Pdb.Tests succeeded (0.2s) → artifacts/bin/Mono.Cecil.Pdb.Tests/Debug/net9.0/Mono.Cecil.Pdb.Tests.dll
Mono.Cecil.Rocks.Tests succeeded (0.3s) → artifacts/bin/Mono.Cecil.Rocks.Tests/Debug/net9.0/Mono.Cecil.Rocks.Tests.dll
Mono.Cecil.Mdb.Tests succeeded (0.3s) → artifacts/bin/Mono.Cecil.Mdb.Tests/Debug/net9.0/Mono.Cecil.Mdb.Tests.dll
Mono.Cecil.Pdb.Tests test succeeded (1.3s)
Mono.Cecil.Rocks.Tests test succeeded with 2 warning(s) (1.7s)
/home/svbomer/tmp/cecil/.dotnet/sdk/9.0.100-preview.4.24267.66/Microsoft.TestPlatform.targets(46,5): warning : SetUp failed for test fixture Mono.Cecil.Tests.DocCommentIdTests
/home/svbomer/tmp/cecil/.dotnet/sdk/9.0.100-preview.4.24267.66/Microsoft.TestPlatform.targets(46,5): warning :
An exception was thrown while loading the test.
System.ArgumentException: Unknown framework version 9.5 (Parameter 'version')
at NUnit.Framework.Internal.RuntimeFramework.ThrowInvalidFrameworkVersion(Version version)
at NUnit.Framework.Internal.RuntimeFramework.InitFromFrameworkVersion(Version version)
at NUnit.Framework.Internal.RuntimeFramework..ctor(RuntimeType runtime, Version version)
at NUnit.Framework.Internal.RuntimeFramework.<>c.<.cctor>b__33_0()
at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
at System.Lazy`1.CreateValue()
at NUnit.Framework.Internal.RuntimeFramework.get_CurrentFramework()
at NUnit.Framework.Internal.PlatformHelper..ctor()
at NUnit.Framework.TestCaseAttribute.BuildFrom(IMethodInfo method, Test suite)+MoveNext()
at NUnit.Framework.Internal.Builders.DefaultTestCaseBuilder.BuildFrom(IMethodInfo method, Test parentSuite)
at NUnit.Framework.Internal.Builders.NUnitTestFixtureBuilder.BuildTestCase(IMethodInfo method, TestSuite suite)
at NUnit.Framework.Internal.Builders.NUnitTestFixtureBuilder.AddTestCasesToFixture(TestFixture fixture, IPreFilter filter)
at NUnit.Framework.Internal.Builders.NUnitTestFixtureBuilder.BuildFrom(ITypeInfo typeInfo, IPreFilter filter, ITestFixtureData testFixtureData)
at NUnit.Framework.TestFixtureAttribute.BuildFrom(ITypeInfo typeInfo, IPreFilter filter)+MoveNext()
at NUnit.Framework.Internal.Builders.DefaultSuiteBuilder.BuildFrom(ITypeInfo typeInfo, IPreFilter filter)
Mono.Cecil.Mdb.Tests test succeeded (5.2s)

Test summary: total: 342, failed: 0, succeeded: 335, skipped: 7, duration: 29.4s
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.