pytest-dev / pytest-dev/pytest-bdd
Cucumber-Json flags failed tests as passed if an exception occurs
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 251
- Avg merge
- 43m
- Merged PRs (30d)
- 2
Description
Version: bdd-4,0,2 pytest: 6.2.2
Summary
When a test fails with eg an keyerror exception due to an invalid test step, the cucumber-json will mark the test case as passed. When imported back into eg XRAY the test is marked as passed when in fact it has not.
This happens because the invalid step is not included in the test report scenario.steps and will therefore not appear in the cucumber-json causing test cases results to be inaccurate.
Steps to Repro:
- Execute a feature file containing a step not defined in the framework. This will trigger an KeyError exception and a FixtureLookupError. Ensure the --cucumber-json flag was passed into pytest-bdd :
Scenario: repro
Given this step should exist and be valid
And this step should not exist
- Examine the resultant cucumber-json file. Notice the test case is marked as passed as the erroring test step was not included.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the issue by running the feature with an undefined step and the --cucumber-json flag, then inspect the generated cucumber-JSON and scenario.steps handling. The fix is complete when a KeyError or FixtureLookupError makes the test case appear as failed rather than passed in the report.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100