pytest-dev / pytest-dev/pytest

`--fixtures` and `--fixtures-per-test` exit with code 0 on collection errors, internal errors and `pytest.exit()`

Open
#15,047 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

Description

pytest --fixtures and pytest --fixtures-per-test exit with status 0 when collection fails. A plain pytest run or pytest --collect-only on the same file exits with 2. Both options also exit with 0 when a hook raises an internal error (normally 3) and when a plugin calls pytest.exit() with a non-zero returncode.

The exit code should match what pytest returns for the same failure without these options: 2 (ExitCode.INTERRUPTED) for collection errors, or 1 (ExitCode.TESTS_FAILED) with --continue-on-collection-errors; 3 (ExitCode.INTERNAL_ERROR) for internal errors; and the returncode passed to pytest.exit(). Scripts and CI jobs that run these options currently see success after pytest has reported an error.

Exit codes observed:

Scenario no option --collect-only --continue-on-collection-errors --fixtures --fixtures-per-test
Collection error (example 1) 2 2 1 0 0
RuntimeError in hook, INTERNALERROR (example 2) 3 0 0
pytest.exit("stop", returncode=7) in hook (example 3) 7 0 0

Two separate problems produce this.

1. The session exit status is discarded. pytest_cmdline_main() in src/_pytest/fixtures.py (lines 1675–1682) ignores the return value of showfixtures() and show_fixtures_per_test():

def pytest_cmdline_main(config: Config) -> int | ExitCode | None:
    if config.option.showfixtures:
        showfixtures(config)
        return 0
    if config.option.show_fixtures_per_test:
        show_fixtures_per_test(config)
        return 0
    return None

Both functions return the result of wrap_session() (src/_pytest/main.py, line 320), which sets session.exitstatus when an exception reaches it: INTERNAL_ERROR for unexpected exceptions, and the given returncode for pytest.exit(). This accounts for examples 2 and 3.

2. Collection errors never produce a non-zero status. A collection error does not raise. Session.perform_collect() only increments session.testsfailed. In a normal run, pytest_runtestloop() (src/_pytest/main.py, lines 400–404) turns that count into an interruption:

def pytest_runtestloop(session: Session) -> bool:
    if session.testsfailed and not session.config.option.continue_on_collection_errors:
        raise session.Interrupted(
            f"{session.testsfailed} error{'s' if session.testsfailed != 1 else ''} during collection"
        )

_showfixtures_main() (line 2505) and _show_fixtures_per_test() (line 2450) in src/_pytest/fixtures.py call session.perform_collect() directly, never reach pytest_runtestloop, and never check session.testsfailed. They return None, so wrap_session() stores ExitCode.OK via session.exitstatus = doit(config, session) or 0 (src/_pytest/main.py, line 333).

Fixing problem 1 alone is not enough. With only pytest_cmdline_main() changed to return the wrap_session() result, examples 2 and 3 exit with the correct codes, but example 1 still exits with 0 for both options.

pip list
Package   Version
--------- -----------------------
iniconfig 2.3.0
packaging 26.3
pip       26.2.1
pluggy    1.6.0
Pygments  2.21.0
pytest    9.2.0.dev335+g99ab2accc
pytest and operating system versions
  • pytest 9.2.0.dev335+g99ab2accc (installed from main at commit 99ab2accc)
  • Python 3.12.3
  • Ubuntu 24.04.4 LTS, Linux 6.8.0-124-generic x86_64
Minimal example

Example 1: collection error

# test_bad.py
def test_broken(:
    pass
$ python -m pytest --fixtures test_bad.py; echo "exit=$?"
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-9.2.0.dev335+g99ab2accc, pluggy-1.6.0
rootdir: /tmp/repro/bad
collected 0 items / 1 error
cache -- .../_pytest/cacheprovider.py:581
    Return a cache object that can persist state between testing sessions.

[... remaining fixture list omitted ...]

==================================== ERRORS ====================================
_________________________ ERROR collecting test_bad.py _________________________
[... traceback omitted ...]
E     File "/tmp/repro/bad/test_bad.py", line 1
E       def test_broken(:
E                       ^
E   SyntaxError: invalid syntax
=========================== short test summary info ============================
ERROR test_bad.py
=============================== 1 error in 0.09s ===============================
exit=0

The same input without --fixtures:

$ python -m pytest test_bad.py; echo "exit=$?"
...
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.16s ===============================
exit=2

python -m pytest --fixtures-per-test test_bad.py also exits with 0.

Example 2: internal error (separate empty directory)

# conftest.py
def pytest_collection_modifyitems(items):
    raise RuntimeError("boom")
# test_ok.py
def test_ok():
    pass
$ python -m pytest --fixtures; echo "exit=$?"
...
INTERNALERROR>     raise RuntimeError("boom")
INTERNALERROR> RuntimeError: boom
============================ no tests ran in 0.00s =============================
exit=0

Without --fixtures, the exit code is 3.

Example 3: pytest.exit() with a return code (separate empty directory, same test_ok.py)

# conftest.py
import pytest


def pytest_collection_modifyitems(items):
    pytest.exit("stop", returncode=7)
$ python -m pytest --fixtures; echo "exit=$?"
...
collected 1 item
============================ no tests ran in 0.08s =============================
!!!!!!!!!!!!!!!!!!!!!!!!! _pytest.outcomes.Exit: stop !!!!!!!!!!!!!!!!!!!!!!!!!!
exit=0

Without --fixtures, the exit code is 7.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with pytest_cmdline_main() in src/_pytest/fixtures.py and follow showfixtures(), show_fixtures_per_test(), and wrap_session() in src/_pytest/main.py. Reproduce the collection, internal-error, and pytest.exit() examples, then verify both fixture options return the matching non-zero status for each case.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.