allure-framework / allure-framework/allure-csharp
Reqnroll: inconclusive tests are reported as broken
- Dominant language
- C#
- Stars
- 125
- Forks
- 76
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 18
Description
When marking a test as inconclusive (e.g., with `Assert.Inconclusive`), Reqnroll reports it as skipped:
```
S /start-specflow-net8.0-dotnet-dotnet> dotnet test
Determining projects to restore...
All projects are up-to-date for restore.
Examples.Allure -> /start-specflow-net8.0-dotnet-dotnet/tests/Examples.Allure/bin/Debug/net8.0/Examples.Allure.dll
Test run for /start-specflow-net8.0-dotnet-dotnet/tests/Examples.Allure/bin/Debug/net8.0/Examples.Allure.dll (.NETCoreApp,Version=v8.0)
VSTest version 17.11.1 (arm64)
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Skipped TestScenario [50 ms]
None - Failed: 0, Passed: 0, Skipped: 0, Total: 0, Duration: 50 ms - Examples.Allure.dll (net8.0)
```
However, Allure Report shows such tests as broken:
Some frameworks (NUnit) allow using `TestIgnore`:
```csharp
using Reqnroll;
using System;
using Reqnroll.UnitTestProvider;
namespace Examples.Allure
{
[Binding]
public class Steps
{
private readonly IUnitTestRuntimeProvider runtimeProvider;
public Steps(IUnitTestRuntimeProvider runtimeProvider)
{
this.runtimeProvider = runtimeProvider;
}
[When("the second step passes")]
public void Step2()
{
this.runtimeProvider.TestIgnore("Can't run here");
}
/* the rest of the steps */
}
}
```
But in this case, the step is still reported as broken:
Other frameworks (MsTest) don't support even that.
#### Expected behavior
Both the step and the test should be reported as skipped to match what Reqnroll reports by itself.
Contributor guide
Research direction
Start by reproducing the issue with `dotnet test` in the shown Examples.Allure project, using both `Assert.Inconclusive` and `IUnitTestRuntimeProvider.TestIgnore`. Trace how Reqnroll step and test outcomes are mapped into Allure results; done means both inconclusive tests and ignored steps are reported as skipped across the supported frameworks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100