codeceptjs / codeceptjs/CodeceptJS
🐛 Bug Report: Background failures generate event.test.failed instead of event.test.skipped for subsequent tests
- Dominant language
- JavaScript
- Stars
- 4.2k
- Forks
- 757
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 16
Description
### Description
Since CodeceptJS 3.7.5, when a Background hook fails in a Gherkin feature, **all** tests in the same Feature file receive `event.test.failed` events instead of `event.test.skipped`. This is a breaking change from version 3.6.7 that significantly complicates result reporting and event handling.
**Core Issue**: When Background fails, CodeceptJS triggers `event.test.failed` for:
1. Tests that failed on Background ✓ (expected)
2. Subsequent tests marked as failed without ability to trigger `event.test.skipped` or `event.test.after` properly
3. Tests in the same suite file that were **never meant to run** (not selected by `--grep`)
This forces custom reporting systems to manually filter and track which tests actually executed before sending results. Previously, simple event handlers worked fine - now complex workarounds are required.
### Expected Behavior (CodeceptJS 3.6.7)
When a Background fails:
- **First test**: receives `event.test.failed`
- **Subsequent tests**: receive `event.test.skipped`
```javascript
// Clean event handling in 3.6.7
event.dispatcher.on(event.test.skipped, (test) => {
// This worked for all tests after first Background failure ✅
sendSkippedReport(test)
})
```
### Actual Behavior (CodeceptJS 3.7.5+)
When a Background fails:
- **All tests**: receive `event.test.failed`
- `event.test.skipped` **never fires**
```javascript
// Complex workaround needed in 3.7.5+
let firstBackgroundFailure = true
event.dispatcher.on(event.test.failed, (test, error) => {
const isBackgroundFailure = test.ctx?.test?.originalTitle?.startsWith('"before each" hook:')
// Manually handle what should be skipped tests
if (isBackgroundFailure && !firstBackgroundFailure) {
// Have to manually convert failed → skipped ❌
sendSkippedReport(test)
return
}
if (isBackgroundFailure) {
firstBackgroundFailure = false
}
})
```
### Problems with Current Behavior
#### 1. **Semantic Incorrectness**
Tests that **never started executing** (only Background failed) are marked as `failed`. Semantically, a test that didn't run should be `skipped`, not `failed`.
#### 2. **Breaking Change Without Clear Benefits**
The change from 3.6.7 to 3.7.5 broke existing event handlers without providing clear advantages. All code relying on `event.test.skipped` for Background failures stopped working.
#### 3. **Event Generation Inconsistency**
**For regular Scenarios:**
When Background fails on first test, subsequent tests don't trigger `event.test.after`, making it impossible to send reports for them:
```gherkin
Feature: Test
Background:
Given failing step
Scenario: Test 1 # Background fails here
Then something
Scenario: Test 2 # event.test.after doesn't process this properly
Then something
```
**For Scenario Outline (running by tests, not by suites):**
When using `--grep` to run a single test, CodeceptJS generates `event.test.failed` for **all examples** in the suite, even those from different Scenario Outlines that weren't selected:
```gherkin
Scenario Outline: Test
# Background fails here
Examples:
| case |
| A | # Not selected by --grep, different Scenario Outline
| B | # Selected by --grep ← only this should run
| C | # Not selected by --grep, different Scenario Outline
```
**Result**: All three tests (A, B, C) receive `event.test.failed`, even though only B was selected to run.
**Key issue**: `event.test.before` fires only for filtered tests, but `event.test.failed` and `event.test.after` fire for ALL examples, making it impossible to distinguish which tests actually ran.
#### 4. **Complex Workarounds Required**
To maintain reasonable behavior, we now need:
- Track which tests actually started (`event.test.before`)
- Track first vs subsequent Background failures
- Manually convert `failed` → `skipped` in `event.test.failed` handler
- Filter out events for tests that never ran
- Initialize tracking in `event.test.failed` for tests where Background failed before `event.test.before` could fire
#### 5. **Report Analysis Difficulties**
- **Before**: "1 failed, 5 skipped" = clear that Background failed
- **After**: "6 failed" = looks like 6 different failures, unclear what happened
#### 6. **Multiple Events for Same Test**
When Background retry mechanism triggers (e.g., with `retryFailedStep` plugin), the same test receives `event.test.failed` **multiple times**, leading to duplicate reports if not handled carefully.
### Steps to Reproduce
**Case 1: Regular Scenarios**
1. Create a feature file with Background and multiple Scenarios
2. Make Background fail
3. Observe that second test doesn't properly trigger reporting in `event.test.after`
```gherkin
Feature: Test
Background:
Given failing step # ← Fails here
Scenario: Test 1
Then something
Scenario: Test 2 # This won't be properly reported
Then something
```
**Case 2: Scenario Outline with --grep**
1. Create a Scenario Outline with multiple examples
2. Run single test with `--grep @TAG-B`
3. Observe that ALL examples receive `event.test.failed`, not just the filtered one
```gherkin
Feature: Test
Background:
Given failing step # ← Fails here
Scenario Outline: Test
Then something
Examples:
| case | tag |
| A | @TAG-A |
| B | @TAG-B | # Only this is selected
| C | @TAG-C |
```
**Expected**: Only B receives events
**Actual**: A, B, and C all receive `event.test.failed`
### Environment
- CodeceptJS version: 3.7.5+
- Node version: 22.4.0
- Helpers: WebDriver, Gherkin
### Proposed Solution
**Option 1 (Preferred)**: Revert to 3.6.7 behavior
- First test with Background failure → `event.test.failed`
- Subsequent tests → `event.test.skipped`
**Option 2**: Add new event type
- `event.test.failedInBackground` or similar
- Allows distinguishing Background failures from test failures
- Maintains backward compatibility
**Option 3**: Configuration option
```javascript
// codecept.conf.js
gherkin: {
backgroundFailureBehavior: 'skip' // or 'fail'
}
```
### Impact
This breaking change affects:
- Custom reporting plugins
- CI/CD result analysis
- Test retry logic
- Statistics collection
- Any code relying on `event.test.skipped`
### References
Comment in our codebase acknowledging this breaking change:
```javascript
// CodeceptJS 3.6.7 → 3.7.5: tests failing in Background are now 'failed' instead of 'skipped'
// Handle Background failures as skipped to maintain compatibility
```
---
**Would it be possible to either revert this change or provide a configuration option to restore the 3.6.7 behavior?** The current implementation significantly complicates result handling and doesn't align with the semantic meaning of "skipped" vs "failed".
Contributor guide
Assessment
This issue has not been assessed yet.