allure-framework / allure-framework/allure-csharp

Distinguish between failed and broken tests when using a 3rd party assertion library with Allure.NUnit and Allure.Xunit

Open
#520 7 comments 4 reactions 0 assignees View on GitHub
good first issue task:improvement theme:nunit theme:xunit
Dominant language
C#
Stars
125
Forks
76
Avg merge
1d 2h
Merged PRs (30d)
18

Description

[//]: # (
. Note: for support questions, please create a discussion at https://github.com/orgs/allure-framework/discussions.
. This repository's issues are reserved for feature requests and bug reports.
.
. Make sure you have a clear name for your issue. The name should start with a capital
. letter and no dot is required in the end of the sentence. An example of good issue names:
.
. - The report is broken in IE11
. - Add an ability to disable default plugins
. - Support emoji in test descriptions
)

#### I'm submitting a ...
- [X] bug report
- [ ] feature request
- [ ] support request => Please do not submit support request here, see note at the top of this template.

#### What is the current behavior?
The test case failed using fluentassertions or Playwright are shown as broken instead of failed

#### If the current behavior is a bug, please provide the steps to reproduce and, if possible, a minimal demo of the problem
Create a simple test using FluentAssertions or Playwright dotnet, when you run the tests the resutl is shown as broken
![image](https://github.com/allure-framework/allure-csharp/assets/162322781/65d64dba-6041-4c8e-8c6f-8c808156ae2d)

#### What is the expected behavior?
The test should be failed if I have configured the list of exceptions (failExceptions).

#### What is the motivation / use case for changing the behavior?

#### Please tell us about your environment:

- OS Windows
- .NET framework .NET 8
- Test framework: nunit@4.0.0
- Allure adapter: allure-nunit@2.12.0

#### Other information

I was taking a look to the AllureNUnitHelper and I was able to see that the IsBroken function is waiting elements in result.Assertions list but when you use FluentAssertions or Playwright this list is not update so the tests are always mark as broken.
![image](https://github.com/allure-framework/allure-csharp/assets/162322781/502e443e-f0e5-4aed-9447-d1a649b62f74)

Contributor guide

Open the contributing guide

Research direction

Start with AllureNUnitHelper.IsBroken and inspect how result.Assertions is used when running a FluentAssertions or Playwright test. Reproduce the behavior with the stated .NET 8 and NUnit setup, then verify that exceptions listed in failExceptions are reported as failed rather than broken.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, playwright
Domain
testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.