codeceptjs / codeceptjs/CodeceptJS

🐛 Bug Report: Background failures generate event.test.failed instead of event.test.skipped for subsequent tests

Open
#5,402 1 comment 0 reactions 0 assignees View on GitHub
stale
Dominant language
JavaScript
Stars
4.2k
Forks
757
Avg merge
2d 9h
Merged PRs (30d)
16

Description

### Description

Since CodeceptJS 3.7.5, when a Background hook fails in a Gherkin feature, **all** tests in the same Feature file receive `event.test.failed` events instead of `event.test.skipped`. This is a breaking change from version 3.6.7 that significantly complicates result reporting and event handling.

**Core Issue**: When Background fails, CodeceptJS triggers `event.test.failed` for:
1. Tests that failed on Background ✓ (expected)
2. Subsequent tests marked as failed without ability to trigger `event.test.skipped` or `event.test.after` properly
3. Tests in the same suite file that were **never meant to run** (not selected by `--grep`)

This forces custom reporting systems to manually filter and track which tests actually executed before sending results. Previously, simple event handlers worked fine - now complex workarounds are required.

### Expected Behavior (CodeceptJS 3.6.7)

When a Background fails:
- **First test**: receives `event.test.failed`
- **Subsequent tests**: receive `event.test.skipped`

```javascript
// Clean event handling in 3.6.7
event.dispatcher.on(event.test.skipped, (test) => {
// This worked for all tests after first Background failure ✅
sendSkippedReport(test)
})
```

### Actual Behavior (CodeceptJS 3.7.5+)

When a Background fails:
- **All tests**: receive `event.test.failed`
- `event.test.skipped` **never fires**

```javascript
// Complex workaround needed in 3.7.5+
let firstBackgroundFailure = true

event.dispatcher.on(event.test.failed, (test, error) => {
const isBackgroundFailure = test.ctx?.test?.originalTitle?.startsWith('"before each" hook:')

// Manually handle what should be skipped tests
if (isBackgroundFailure && !firstBackgroundFailure) {
// Have to manually convert failed → skipped ❌
sendSkippedReport(test)
return
}

if (isBackgroundFailure) {
firstBackgroundFailure = false
}
})
```

### Problems with Current Behavior

#### 1. **Semantic Incorrectness**
Tests that **never started executing** (only Background failed) are marked as `failed`. Semantically, a test that didn't run should be `skipped`, not `failed`.

#### 2. **Breaking Change Without Clear Benefits**
The change from 3.6.7 to 3.7.5 broke existing event handlers without providing clear advantages. All code relying on `event.test.skipped` for Background failures stopped working.

#### 3. **Event Generation Inconsistency**

**For regular Scenarios:**
When Background fails on first test, subsequent tests don't trigger `event.test.after`, making it impossible to send reports for them:

```gherkin
Feature: Test
Background:
Given failing step

Scenario: Test 1 # Background fails here
Then something

Scenario: Test 2 # event.test.after doesn't process this properly
Then something
```

**For Scenario Outline (running by tests, not by suites):**
When using `--grep` to run a single test, CodeceptJS generates `event.test.failed` for **all examples** in the suite, even those from different Scenario Outlines that weren't selected:

```gherkin
Scenario Outline: Test
# Background fails here

Examples:
| case |
| A | # Not selected by --grep, different Scenario Outline
| B | # Selected by --grep ← only this should run
| C | # Not selected by --grep, different Scenario Outline
```

**Result**: All three tests (A, B, C) receive `event.test.failed`, even though only B was selected to run.

**Key issue**: `event.test.before` fires only for filtered tests, but `event.test.failed` and `event.test.after` fire for ALL examples, making it impossible to distinguish which tests actually ran.

#### 4. **Complex Workarounds Required**
To maintain reasonable behavior, we now need:
- Track which tests actually started (`event.test.before`)
- Track first vs subsequent Background failures
- Manually convert `failed` → `skipped` in `event.test.failed` handler
- Filter out events for tests that never ran
- Initialize tracking in `event.test.failed` for tests where Background failed before `event.test.before` could fire

#### 5. **Report Analysis Difficulties**
- **Before**: "1 failed, 5 skipped" = clear that Background failed
- **After**: "6 failed" = looks like 6 different failures, unclear what happened

#### 6. **Multiple Events for Same Test**
When Background retry mechanism triggers (e.g., with `retryFailedStep` plugin), the same test receives `event.test.failed` **multiple times**, leading to duplicate reports if not handled carefully.

### Steps to Reproduce

**Case 1: Regular Scenarios**
1. Create a feature file with Background and multiple Scenarios
2. Make Background fail
3. Observe that second test doesn't properly trigger reporting in `event.test.after`

```gherkin
Feature: Test
Background:
Given failing step # ← Fails here

Scenario: Test 1
Then something

Scenario: Test 2 # This won't be properly reported
Then something
```

**Case 2: Scenario Outline with --grep**
1. Create a Scenario Outline with multiple examples
2. Run single test with `--grep @TAG-B`
3. Observe that ALL examples receive `event.test.failed`, not just the filtered one

```gherkin
Feature: Test
Background:
Given failing step # ← Fails here

Scenario Outline: Test
Then something

Examples:
| case | tag |
| A | @TAG-A |
| B | @TAG-B | # Only this is selected
| C | @TAG-C |
```

**Expected**: Only B receives events
**Actual**: A, B, and C all receive `event.test.failed`

### Environment

- CodeceptJS version: 3.7.5+
- Node version: 22.4.0
- Helpers: WebDriver, Gherkin

### Proposed Solution

**Option 1 (Preferred)**: Revert to 3.6.7 behavior
- First test with Background failure → `event.test.failed`
- Subsequent tests → `event.test.skipped`

**Option 2**: Add new event type
- `event.test.failedInBackground` or similar
- Allows distinguishing Background failures from test failures
- Maintains backward compatibility

**Option 3**: Configuration option
```javascript
// codecept.conf.js
gherkin: {
backgroundFailureBehavior: 'skip' // or 'fail'
}
```

### Impact

This breaking change affects:
- Custom reporting plugins
- CI/CD result analysis
- Test retry logic
- Statistics collection
- Any code relying on `event.test.skipped`

### References

Comment in our codebase acknowledging this breaking change:
```javascript
// CodeceptJS 3.6.7 → 3.7.5: tests failing in Background are now 'failed' instead of 'skipped'
// Handle Background failures as skipped to maintain compatibility
```

---

**Would it be possible to either revert this change or provide a configuration option to restore the 3.6.7 behavior?** The current implementation significantly complicates result handling and doesn't align with the semantic meaning of "skipped" vs "failed".

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.