pytest-dev / pytest-dev/pytest
`--fixtures` and `--fixtures-per-test` exit with code 0 on collection errors, internal errors and `pytest.exit()`
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.5k
- Forks
- 3.4k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 35
Description
Description
pytest --fixtures and pytest --fixtures-per-test exit with status 0 when collection fails. A plain pytest run or pytest --collect-only on the same file exits with 2. Both options also exit with 0 when a hook raises an internal error (normally 3) and when a plugin calls pytest.exit() with a non-zero returncode.
The exit code should match what pytest returns for the same failure without these options: 2 (ExitCode.INTERRUPTED) for collection errors, or 1 (ExitCode.TESTS_FAILED) with --continue-on-collection-errors; 3 (ExitCode.INTERNAL_ERROR) for internal errors; and the returncode passed to pytest.exit(). Scripts and CI jobs that run these options currently see success after pytest has reported an error.
Exit codes observed:
| Scenario | no option | --collect-only |
--continue-on-collection-errors |
--fixtures |
--fixtures-per-test |
|---|---|---|---|---|---|
| Collection error (example 1) | 2 | 2 | 1 | 0 | 0 |
RuntimeError in hook, INTERNALERROR (example 2) |
3 | — | — | 0 | 0 |
pytest.exit("stop", returncode=7) in hook (example 3) |
7 | — | — | 0 | 0 |
Two separate problems produce this.
1. The session exit status is discarded. pytest_cmdline_main() in src/_pytest/fixtures.py (lines 1675–1682) ignores the return value of showfixtures() and show_fixtures_per_test():
def pytest_cmdline_main(config: Config) -> int | ExitCode | None:
if config.option.showfixtures:
showfixtures(config)
return 0
if config.option.show_fixtures_per_test:
show_fixtures_per_test(config)
return 0
return None
Both functions return the result of wrap_session() (src/_pytest/main.py, line 320), which sets session.exitstatus when an exception reaches it: INTERNAL_ERROR for unexpected exceptions, and the given returncode for pytest.exit(). This accounts for examples 2 and 3.
2. Collection errors never produce a non-zero status. A collection error does not raise. Session.perform_collect() only increments session.testsfailed. In a normal run, pytest_runtestloop() (src/_pytest/main.py, lines 400–404) turns that count into an interruption:
def pytest_runtestloop(session: Session) -> bool:
if session.testsfailed and not session.config.option.continue_on_collection_errors:
raise session.Interrupted(
f"{session.testsfailed} error{'s' if session.testsfailed != 1 else ''} during collection"
)
_showfixtures_main() (line 2505) and _show_fixtures_per_test() (line 2450) in src/_pytest/fixtures.py call session.perform_collect() directly, never reach pytest_runtestloop, and never check session.testsfailed. They return None, so wrap_session() stores ExitCode.OK via session.exitstatus = doit(config, session) or 0 (src/_pytest/main.py, line 333).
Fixing problem 1 alone is not enough. With only pytest_cmdline_main() changed to return the wrap_session() result, examples 2 and 3 exit with the correct codes, but example 1 still exits with 0 for both options.
pip list
Package Version
--------- -----------------------
iniconfig 2.3.0
packaging 26.3
pip 26.2.1
pluggy 1.6.0
Pygments 2.21.0
pytest 9.2.0.dev335+g99ab2accc
pytest and operating system versions
- pytest 9.2.0.dev335+g99ab2accc (installed from
mainat commit99ab2accc) - Python 3.12.3
- Ubuntu 24.04.4 LTS, Linux 6.8.0-124-generic x86_64
Minimal example
Example 1: collection error
# test_bad.py
def test_broken(:
pass
$ python -m pytest --fixtures test_bad.py; echo "exit=$?"
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-9.2.0.dev335+g99ab2accc, pluggy-1.6.0
rootdir: /tmp/repro/bad
collected 0 items / 1 error
cache -- .../_pytest/cacheprovider.py:581
Return a cache object that can persist state between testing sessions.
[... remaining fixture list omitted ...]
==================================== ERRORS ====================================
_________________________ ERROR collecting test_bad.py _________________________
[... traceback omitted ...]
E File "/tmp/repro/bad/test_bad.py", line 1
E def test_broken(:
E ^
E SyntaxError: invalid syntax
=========================== short test summary info ============================
ERROR test_bad.py
=============================== 1 error in 0.09s ===============================
exit=0
The same input without --fixtures:
$ python -m pytest test_bad.py; echo "exit=$?"
...
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.16s ===============================
exit=2
python -m pytest --fixtures-per-test test_bad.py also exits with 0.
Example 2: internal error (separate empty directory)
# conftest.py
def pytest_collection_modifyitems(items):
raise RuntimeError("boom")
# test_ok.py
def test_ok():
pass
$ python -m pytest --fixtures; echo "exit=$?"
...
INTERNALERROR> raise RuntimeError("boom")
INTERNALERROR> RuntimeError: boom
============================ no tests ran in 0.00s =============================
exit=0
Without --fixtures, the exit code is 3.
Example 3: pytest.exit() with a return code (separate empty directory, same test_ok.py)
# conftest.py
import pytest
def pytest_collection_modifyitems(items):
pytest.exit("stop", returncode=7)
$ python -m pytest --fixtures; echo "exit=$?"
...
collected 1 item
============================ no tests ran in 0.08s =============================
!!!!!!!!!!!!!!!!!!!!!!!!! _pytest.outcomes.Exit: stop !!!!!!!!!!!!!!!!!!!!!!!!!!
exit=0
Without --fixtures, the exit code is 7.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with pytest_cmdline_main() in src/_pytest/fixtures.py and follow showfixtures(), show_fixtures_per_test(), and wrap_session() in src/_pytest/main.py. Reproduce the collection, internal-error, and pytest.exit() examples, then verify both fixture options return the matching non-zero status for each case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100