allure-framework / allure-framework/allure-csharp

Reqnroll: inconclusive tests are reported as broken

Open
#557 1 comment 1 reaction 0 assignees View on GitHub
bug theme:reqnroll
Dominant language
C#
Stars
125
Forks
76
Avg merge
1d 2h
Merged PRs (30d)
18

Description

When marking a test as inconclusive (e.g., with `Assert.Inconclusive`), Reqnroll reports it as skipped:

```
S /start-specflow-net8.0-dotnet-dotnet> dotnet test
Determining projects to restore...
All projects are up-to-date for restore.
Examples.Allure -> /start-specflow-net8.0-dotnet-dotnet/tests/Examples.Allure/bin/Debug/net8.0/Examples.Allure.dll
Test run for /start-specflow-net8.0-dotnet-dotnet/tests/Examples.Allure/bin/Debug/net8.0/Examples.Allure.dll (.NETCoreApp,Version=v8.0)
VSTest version 17.11.1 (arm64)

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Skipped TestScenario [50 ms]

None - Failed: 0, Passed: 0, Skipped: 0, Total: 0, Duration: 50 ms - Examples.Allure.dll (net8.0)
```

However, Allure Report shows such tests as broken:

Some frameworks (NUnit) allow using `TestIgnore`:

```csharp
using Reqnroll;
using System;
using Reqnroll.UnitTestProvider;

namespace Examples.Allure
{
[Binding]
public class Steps
{
private readonly IUnitTestRuntimeProvider runtimeProvider;

public Steps(IUnitTestRuntimeProvider runtimeProvider)
{
this.runtimeProvider = runtimeProvider;
}

[When("the second step passes")]
public void Step2()
{
this.runtimeProvider.TestIgnore("Can't run here");
}

/* the rest of the steps */
}
}
```

But in this case, the step is still reported as broken:

Other frameworks (MsTest) don't support even that.

#### Expected behavior

Both the step and the test should be reported as skipped to match what Reqnroll reports by itself.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue with `dotnet test` in the shown Examples.Allure project, using both `Assert.Inconclusive` and `IUnitTestRuntimeProvider.TestIgnore`. Trace how Reqnroll step and test outcomes are mapped into Allure results; done means both inconclusive tests and ignored steps are reported as skipped across the supported frameworks.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.