codeceptjs / codeceptjs/CodeceptJS

Silent Failures of Assertions When Using --grep with Full Feature-Scenario Name

オープン
#5,528 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る
stale
主要言語
JavaScript
スター
4.2k
フォーク
757
平均マージ
2日 9時間
マージ済み PR(30日)
16

説明

## Summary

I am encountering an issue when running specific scenarios with the `--grep` flag. Specifically, when using `npx codeceptjs run --grep "Featurename: Scenarioname"`, node assertions like `assert.equal(1,2)` are not being processed correctly and test failures are silent. However, running the same test with `npx codeceptjs run --grep "Scenarioname"` works as expected. Since a couple of versions CodeceptJS gives the first command as a suggestion if no matching scenario was found for a test run:
```
No tests found by pattern: /Featurename: Scenarioname_misspelled/
Maybe you wanted to run one of these tests?
...
To run the first test use the following command:
npx codeceptjs run --debug --grep "Featurename: Scenarioname "
```

This issue leads to assertions seemingly passing, and the test run result shows a green `OK | 0 passed`, despite a failure condition and the test run itself being aborted correctly.
If a pattern is suggested for use, it should work as expected and handle failing assertions correctly, which in this case it does not.

### Steps to Reproduce

1. Define a Feature and Scenario Outline in a .feature file
2. Add a step definition for the Scenario that contains a false node assertion in a .steps.ts file
3. Run CodeceptJS with the full feature-scenario name as a filter:
`npx codeceptjs run --grep "Featurename: Scenarioname"`
If the assertion evaluates to a failure the test will silently pass, and the terminal will show `OK | 0 passed` even if the whole test should have failed.
4. If the same test is run with just the scenario name filter:
`npx codeceptjs run --grep "Scenarioname"`
The test executes correctly, fails and prints `FAIL | 0 passed, 1 failed`.

## Code Example
### Feature File:
```
Feature: Billing

As a user, I want to do billing calculations.

Scenario Outline: Monthly
Given I am on page billing
When I settle the bill
Then I see that the number of billed items is correct

Examples:
| usecase | dateFrom | dateTo | billedItems |
| Normal | 2023-01-01| 2023-01-31 | 10 |

```
Step Definition:
```
import assert from "node:assert"
....
Then("I see that the number of billed items {int} is correct", async (billedItems: number) => {
assert.equal(billedItems, 10); // Example check, should fail if numbers don't match
})
```

## Expected Behavior
When running `npx codeceptjs run --grep "Featurename: Scenarioname"`, the assertions should execute, and any failures should be properly tracked and printed by CodeceptJS.
The output should show either 1 failed or 1 passed, depending on the assertion and should not silently fail with 0 passed.

### Environment

CodeceptJS version: 3.7.8
Node.js version: v25.9.0

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

Feature/Scenario Outline と .steps.ts のステップ定義で問題を再現し、`npx codeceptjs run --grep "Featurename: Scenarioname"` とシナリオのみを実行するコマンドを比較します。CLI の `--grep` 処理から始め、完全な feature-scenario の一致がどのように assertion を実行し、結果を記録するかを追跡します。失敗した assertion が正しく報告され、実行結果に誤解を招く `OK | 0 passed` が表示されなくなれば完了です。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
javascript, node.js, typescript
領域
cli, testing
issue の種類
バグ
難易度
3/5
見積もり時間
1〜2日
活発さ
静か
明瞭さ
おおむね明確
初心者へのやさしさ
52/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。