Add ability to specify expected failures in blackbox negative testing
Open
Nobody has claimed this yet.
area/testing
- Dominant language
- Go
- Stars
- 974
- Forks
- 296
- Avg merge
- 6d 14h
- Merged PRs (30d)
- 9
Description
Feature Request
Desired Feature
Add functionality to be able to declare specific expected failure cases for negative testing rather than assuming that any failure is expected.
The current system allows potential regressions to be masked if the test run still fails for other reasons.
Other Information
Originally brought up in https://github.com/coreos/ignition/pull/746#discussion_r260448340
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the blackbox negative-testing implementation and reviewing the discussion in coreos/ignition#746. Define how individual expected failure cases are declared and matched; done means expected failures are accepted while unrelated failures still fail the test run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100