pytest-dev / pytest-dev/pytest-random-order

Regression in `--failed-first`?

Open
#53 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
79
Forks
14
PR merge metrics
No merged PRs in 30d

Description

#28 appears to have fixed a problem with the --failed-first option in 0.8.0, but --failed-first breaks for me in the latest version with the following traceback:

$ pytest --failed-first
=============================================================================== test session starts ================================================================================
platform darwin -- Python 3.10.13, pytest-7.4.3, pluggy-1.3.0
Test order randomisation NOT enabled. Enable with --random-order or --random-order-bucket=<bucket_type>
rootdir: [...]
configfile: pytest.ini
plugins: random-order-1.1.0, anyio-4.0.0, cov-4.1.0, vcr-1.0.2, asyncio-0.21.1, hypothesis-6.88.3, Faker-19.13.0, profiling-1.7.0, xdist-3.3.1
asyncio: mode=auto
collected 355 items
run-last-failure: rerun previous 21 failures first
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "[...]/.pyenv/versions/3.10.13/envs/[...]/lib/python3.10/site-packages/_pytest/main.py", line 271, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "[...]/.pyenv/versions/3.10.13/envs/[...]/lib/python3.10/site-packages/_pytest/main.py", line 324, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "[...]/.pyenv/versions/3.10.13/envs/[...]/lib/python3.10/site-packages/pluggy/_hooks.py", line 493, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>   File "[...]/.pyenv/versions/3.10.13/envs/[...]/lib/python3.10/site-packages/pluggy/_manager.py", line 115, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "[...]/.pyenv/versions/3.10.13/envs/[...]/lib/python3.10/site-packages/pluggy/_callers.py", line 152, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "[...]/.pyenv/versions/3.10.13/envs/[...]/lib/python3.10/site-packages/pluggy/_result.py", line 114, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "[...]/.pyenv/versions/3.10.13/envs/[...]/lib/python3.10/site-packages/pluggy/_callers.py", line 77, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "[...]/.pyenv/versions/3.10.13/envs/[...]/lib/python3.10/site-packages/_pytest/main.py", line 335, in pytest_collection
INTERNALERROR>     session.perform_collect()
INTERNALERROR>   File "[...]/.pyenv/versions/3.10.13/envs/[...]/lib/python3.10/site-packages/_pytest/main.py", line 678, in perform_collect
INTERNALERROR>     hook.pytest_collection_modifyitems(
INTERNALERROR>   File "[...]/.pyenv/versions/3.10.13/envs/[...]/lib/python3.10/site-packages/pluggy/_hooks.py", line 493, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>   File "[...]/.pyenv/versions/3.10.13/envs/[...]/lib/python3.10/site-packages/pluggy/_manager.py", line 115, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "[...]/.pyenv/versions/3.10.13/envs/[...]/lib/python3.10/site-packages/pluggy/_callers.py", line 152, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "[...]/.pyenv/versions/3.10.13/envs/[...]/lib/python3.10/site-packages/pluggy/_result.py", line 114, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "[...]/.pyenv/versions/3.10.13/envs/[...]/lib/python3.10/site-packages/pluggy/_callers.py", line 77, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "[...]/.pyenv/versions/3.10.13/envs/[...]/lib/python3.10/site-packages/random_order/plugin.py", line 72, in pytest_collection_modifyitems
INTERNALERROR>     process_failed_first_last_failed(session, config, items)
INTERNALERROR>   File "[...]/.pyenv/versions/3.10.13/envs/[...]/lib/python3.10/site-packages/random_order/cache.py", line 28, in process_failed_first_last_failed
INTERNALERROR>     raise NotImplementedError()
INTERNALERROR> NotImplementedError

============================================================================== no tests ran in 1.18s ===============================================================================

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing pytest --failed-first with the configuration and plugins shown in the traceback, then read the related behavior from #28. Trace the interaction between random_order/plugin.py and random_order/cache.py; done means --failed-first no longer raises NotImplementedError and reruns previous failures first.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.