pytest-dev / pytest-dev/pytest
--last-failed --last-failed-no-failures none is now collecting much more tests than it did in 5.4.3
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.5k
- Forks
- 3.4k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 35
Description
Hi, in work I have been recently upgrading from pytest 5.4.3 -> 6.1.2, I observe a behaviour discrepency in the:
pytest_collection_modifyitems hook, in 5.4.3 given our setup we get the following:
pytest --last-failed --last-failed-no-failures none
# internal plugin registered via pytest_plugins=[]
@pytest.hookimpl(trylast=True)
def pytest_collection_modifyitems(self, items: List[Item]) -> None:
try:
for item in items:
log.debug(f"collected item: {item.name}")
do_something()
except Exception as exc:
pytest.exit(str(exc))
In the above snippet, the import thing is len(items) is 2. inline with this:
{
"scrubbed/sensitive/info/test_two.py::TestTwo::test_two": true
"scrubbed/sensitive/info/test_one.py::TestOne::test_one": true
}
I upgraded to 6.1.2 and with the same last failed cache highlighted above, len of the items in my hook implementation above is now pretty much every single test we have, 2600+.
note: This is barebones at the moment, scrubbing sensitive info is going to take a while
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the command pytest --last-failed --last-failed-no-failures none with the shown pytest_collection_modifyitems hook under pytest 5.4.3 and 6.1.2, using the described last-failed cache. Compare the collected item count and confirm whether the newer behavior returns only the two cached failures rather than 2600+ tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100