pytest-dev / pytest-dev/pytest-bdd

Cucumber-Json flags failed tests as passed if an exception occurs

Open
#430 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.5k
Forks
251
Avg merge
43m
Merged PRs (30d)
2

Description

Version: bdd-4,0,2 pytest: 6.2.2

Summary
When a test fails with eg an keyerror exception due to an invalid test step, the cucumber-json will mark the test case as passed. When imported back into eg XRAY the test is marked as passed when in fact it has not.

This happens because the invalid step is not included in the test report scenario.steps and will therefore not appear in the cucumber-json causing test cases results to be inaccurate.

Steps to Repro:

  1. Execute a feature file containing a step not defined in the framework. This will trigger an KeyError exception and a FixtureLookupError. Ensure the --cucumber-json flag was passed into pytest-bdd :
Scenario:  repro
    Given this step should exist and be valid
    And this step should not exist

  1. Examine the resultant cucumber-json file. Notice the test case is marked as passed as the erroring test step was not included.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue by running the feature with an undefined step and the --cucumber-json flag, then inspect the generated cucumber-JSON and scenario.steps handling. The fix is complete when a KeyError or FixtureLookupError makes the test case appear as failed rather than passed in the report.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.