pytest-dev / pytest-dev/pytest

--last-failed --last-failed-no-failures none is now collecting much more tests than it did in 5.4.3

Open
#8,078 12 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic: collection type: regression
Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

Hi, in work I have been recently upgrading from pytest 5.4.3 -> 6.1.2, I observe a behaviour discrepency in the:

pytest_collection_modifyitems hook, in 5.4.3 given our setup we get the following:

pytest --last-failed --last-failed-no-failures none 
    # internal plugin registered via pytest_plugins=[]

    @pytest.hookimpl(trylast=True)
    def pytest_collection_modifyitems(self, items: List[Item]) -> None:
        try:
            for item in items:
                log.debug(f"collected item: {item.name}")
            do_something()
        except Exception as exc:
            pytest.exit(str(exc))

In the above snippet, the import thing is len(items) is 2. inline with this:

{
  "scrubbed/sensitive/info/test_two.py::TestTwo::test_two": true
  "scrubbed/sensitive/info/test_one.py::TestOne::test_one": true
}

I upgraded to 6.1.2 and with the same last failed cache highlighted above, len of the items in my hook implementation above is now pretty much every single test we have, 2600+.

note: This is barebones at the moment, scrubbing sensitive info is going to take a while

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the command pytest --last-failed --last-failed-no-failures none with the shown pytest_collection_modifyitems hook under pytest 5.4.3 and 6.1.2, using the described last-failed cache. Compare the collected item count and confirm whether the newer behavior returns only the two cached failures rather than 2600+ tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.