cucumber / cucumber/godog

[jUnit formatter] Skipped steps are considered as errors

Open
#375 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.7k
Forks
280
Avg merge
2d 20h
Merged PRs (30d)
2

Description

We noticed that when a step fails in an scenario, subsequent steps are marked as skipped, which is OK imho.

When processing a test case (a scenario), all the steps that are skipped are included as errors (although having the `skipped` type). And because they are marked as error XML elements, we are not able to identify the root cause of why the scenario failed.

But why? Because the CI (Jenkins) provides a higher ranking to errors than to failures, causing the UI to display the fist step error instead of the first failure, which is misleading. I.e.:

```xml







```
Screenshot 2021-02-10 at 13 22 37

Thankfully we store the XML file and then we were able to read the real error, hid by the first error.

With the above XML file, I'd expect the UI to display the failure step. In fact, removing all errors simply keeping failures makes the UI to display the proper error message.

Note: we are aware that, potentially, it could be an issue in the Jenkins side where errors are overriding the real failure, but wanted to share here our expectations first.

Contributor guide

Open the contributing guide

Research direction

Start by locating the jUnit formatter and the code that serializes scenario steps into the shown XML. Reproduce a scenario where a failed step causes later steps to be skipped, then inspect the generated testcase elements. Done means skipped steps are not emitted as error elements while the original failure remains visible.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.