allure-framework / allure-framework/allure-js

Newman: wrong allure-report after collection run

Open
#759 0 comments 0 reactions 0 assignees View on GitHub
theme:newman type:bug
Dominant language
TypeScript
Stars
281
Forks
137
Avg merge
1d 11h
Merged PRs (30d)
13

Description

**Describe the bug**
Test is passed in allure-report, but collection run result is failed

**To Reproduce**
Steps to reproduce the behavior:
1. Run any collection with expected failed result (e.g. endpoint is not accessible) without allure-report:
`newman run `
2. Take a look at result
3. Retry first step with allure-report:
`newman run -r allure`
4. See allure-report

**Expected behavior**
Allure-report has all tests failed

**Screenshots**
![image](https://github.com/allure-framework/allure-js/assets/97187806/59b2e46b-8070-48c8-8c7a-0b267967da12)
![image](https://github.com/allure-framework/allure-js/assets/97187806/33205420-a1e8-4542-8edd-269587869da4)
[allure-results.zip](https://github.com/allure-framework/allure-js/files/12411375/allure-results.zip)

**Desktop (please complete the following information):**
- OS: Windows 10 Pro
- Browser: Chrome
- Version: 116.0.5845.97
- Allure version: 2.23.1

**Smartphone (please complete the following information):**
Not applicable

**Additional context**
I didn't check mixed result with buth passed and failed tests

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the two Newman commands in the issue, first without the allure reporter and then with it, and compare the generated Allure report with the attached allure-results.zip. Trace how the failed collection result is represented in the Allure output. Done means failed collection tests are reported as failed in Allure.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.