codeceptjs / codeceptjs/CodeceptJS

Silent Failures of Assertions When Using --grep with Full Feature-Scenario Name

Đang mở
#5,528 1 bình luận 0 reaction 0 người được giao Xem trên GitHub
stale
Ngôn ngữ chính
JavaScript
Star
4.2k
Fork
757
Merge trung bình
2 ngày 9 giờ
Pull request đã merge (30 ngày)
16

Mô tả

## Summary

I am encountering an issue when running specific scenarios with the `--grep` flag. Specifically, when using `npx codeceptjs run --grep "Featurename: Scenarioname"`, node assertions like `assert.equal(1,2)` are not being processed correctly and test failures are silent. However, running the same test with `npx codeceptjs run --grep "Scenarioname"` works as expected. Since a couple of versions CodeceptJS gives the first command as a suggestion if no matching scenario was found for a test run:
```
No tests found by pattern: /Featurename: Scenarioname_misspelled/
Maybe you wanted to run one of these tests?
...
To run the first test use the following command:
npx codeceptjs run --debug --grep "Featurename: Scenarioname "
```

This issue leads to assertions seemingly passing, and the test run result shows a green `OK | 0 passed`, despite a failure condition and the test run itself being aborted correctly.
If a pattern is suggested for use, it should work as expected and handle failing assertions correctly, which in this case it does not.

### Steps to Reproduce

1. Define a Feature and Scenario Outline in a .feature file
2. Add a step definition for the Scenario that contains a false node assertion in a .steps.ts file
3. Run CodeceptJS with the full feature-scenario name as a filter:
`npx codeceptjs run --grep "Featurename: Scenarioname"`
If the assertion evaluates to a failure the test will silently pass, and the terminal will show `OK | 0 passed` even if the whole test should have failed.
4. If the same test is run with just the scenario name filter:
`npx codeceptjs run --grep "Scenarioname"`
The test executes correctly, fails and prints `FAIL | 0 passed, 1 failed`.

## Code Example
### Feature File:
```
Feature: Billing

As a user, I want to do billing calculations.

Scenario Outline: Monthly
Given I am on page billing
When I settle the bill
Then I see that the number of billed items is correct

Examples:
| usecase | dateFrom | dateTo | billedItems |
| Normal | 2023-01-01| 2023-01-31 | 10 |

```
Step Definition:
```
import assert from "node:assert"
....
Then("I see that the number of billed items {int} is correct", async (billedItems: number) => {
assert.equal(billedItems, 10); // Example check, should fail if numbers don't match
})
```

## Expected Behavior
When running `npx codeceptjs run --grep "Featurename: Scenarioname"`, the assertions should execute, and any failures should be properly tracked and printed by CodeceptJS.
The output should show either 1 failed or 1 passed, depending on the assertion and should not silently fail with 0 passed.

### Environment

CodeceptJS version: 3.7.8
Node.js version: v25.9.0

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.