Redo the design of functional tests
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.3k
- Forks
- 835
- Avg merge
- 5d 3h
- Merged PRs (30d)
- 1
Description
Describe the bug
Currently the functional tests just measure a count of the number of high, medium, low confidence and severity for a file. This mostly works, but can hide false positives and isn't the best way to do an absolute test that a function gets a desired result. It relies on counts and not whether an individual line caused an bandit error or warning.
To Reproduce
Look at functional tests and how they work.
Expected behavior
Instead of the current method, I think we need to devise a way that each line is checked rather than comparing a total count for a file. Maybe we don't even need the files. Maybe these just become better unit tests of each plugin.
Bandit version
bandit 1.4.1
python version = 2.7.10 (default, Oct 6 2017, 22:29:07) [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.31)]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the functional tests and how their current file-level confidence and severity counts are calculated. Define a test approach that checks individual lines or plugin results rather than aggregate counts, and consider whether these should become unit tests; the work is done when false positives are exposed by precise expected results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100