cake-build / cake-build/cake

Return number of failing NUnit tests

Open
#2,772 6 comments 0 reactions 0 assignees View on GitHub
Improvement
Dominant language
C#
Stars
4.2k
Forks
778
Avg merge
1h 15m
Merged PRs (30d)
19

Description

### What You Are Seeing?

When running NUnit Tests from Cake and any test fails Cake throws an exception. This is due to the way return codes are handled by the [NUnit3Runner](https://github.com/cake-build/cake/blob/develop/src/Cake.Common/Tools/NUnit/NUnit3Runner.cs):

```csharp
if (exitCode <= 0)
{
switch (exitCode)
{
// handle other stuff, not relevant
}
}
else
{
error = string.Format(CultureInfo.InvariantCulture, "{0} test(s) failed", exitCode);
}

const string message = "{0}: {1} (exit code {2}).";
throw new CakeException(string.Format(CultureInfo.InvariantCulture, message, GetToolName(), error, exitCode));
```

At first this makes sense, since the return code != 0 indicates no success. But when I use the [NUnit3Settings](https://cakebuild.net/api/Cake.Common.Tools.NUnit/NUnit3Settings/) with `StopOnError = false` I would expect not only the NUnit console runner to continue when a test failed but also the Cake [NUnit3Runner](https://github.com/cake-build/cake/blob/develop/src/Cake.Common/Tools/NUnit/NUnit3Runner.cs).

Why? With the NUnit console runner any return code above zero describes the number of failing tests, and is not an indication of something that "failed" in a technical sense, e.g. when I run tests and set `StopOnError = false` I expect to be the one responsible for handling failures (in my case, reporting them to TeamCity or writing a playlist file) - I don't expect the need of catching an exception, and then again deciding on my own if I can handle the exception (return code > 0) or I have to re-throw the exception.

### What is Expected?

I expect that the [NUnit3Runner](https://github.com/cake-build/cake/blob/develop/src/Cake.Common/Tools/NUnit/NUnit3Runner.cs) doesn't throw an exception for return codes above zero when I use `StopOnError = false` while running NUnit tests.
I'd also love to have access to the test results using the [NUnit3Result](https://cakebuild.net/api/Cake.Common.Tools.NUnit/NUnit3Result/) (I know it contains the path to the result XML and some way of being able to transform the XML, but I'm talking about a list of for example `NUnit3TestCaseResult` (I just totally made this class up on the fly) which contains the name of the test, failure message etc - A deserialized representation of the NUnit result XML file, basically).

Contributor guide

Open the contributing guide

Research direction

Start in src/Cake.Common/Tools/NUnit/NUnit3Runner.cs and review how NUnit3Settings.StopOnError and NUnit3Result are used. Trace the NUnit exit-code handling and result XML path, then define completion as handling positive failing-test counts without the unwanted exception when StopOnError is false and exposing the requested result data if that scope is retained.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
cli, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.