pytest-dev / pytest-dev/pytest

--last-failed can silently skip failing custom items with brackets in their names

Open
#15,045 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

When a custom collector creates an Item whose name contains brackets, --last-failed can omit that item even though it failed in the previous run. If another previously failing item has been fixed, pytest reports success while the bracket-named item still fails in a full run.

Reproduced on main at 99ab2acccff15f757260a1e1fbb0d2a47fc01041 (9.2.0.dev335+g99ab2accc), Linux x86_64, Python 3.12.3, pluggy 1.6.0. Plugin autoload is disabled. The reproducer only needs pytest and the standard library. The documented YAML collector exhibits the same behavior; pytest 9.1.1 handles that YAML scenario correctly.

Save this as reproduce.py and run it using the Python environment containing the current development version of pytest:

import json
import os
from pathlib import Path
import subprocess
import sys
import tempfile

conftest = '''import json
import pytest


def pytest_collect_file(parent, file_path):
    if file_path.name == "test_cases.json":
        return Cases.from_parent(parent, path=file_path)


class Cases(pytest.File):
    def collect(self):
        for name, passed in json.loads(self.path.read_text()).items():
            yield Case.from_parent(self, name=name, passed=passed)


class Case(pytest.Item):
    def __init__(self, *, passed, **kwargs):
        super().__init__(**kwargs)
        self.passed = passed

    def runtest(self):
        assert self.passed
'''

with tempfile.TemporaryDirectory() as directory:
    root = Path(directory)
    (root / 'conftest.py').write_text(conftest)
    (root / 'pytest.ini').write_text('[pytest]\n')
    cases = root / 'test_cases.json'
    env = dict(os.environ, PYTEST_DISABLE_PLUGIN_AUTOLOAD='1')
    results = []
    for label, data, args in [
        ('initial', {'a_bad[one]': False, 'b_fixed': False}, []),
        ('last-failed after fixing b_fixed', {'a_bad[one]': False, 'b_fixed': True}, ['--lf']),
        ('full run with same input', {'a_bad[one]': False, 'b_fixed': True}, []),
    ]:
        cases.write_text(json.dumps(data))
        run = subprocess.run([sys.executable, '-m', 'pytest', '-v', '--tb=no', *args], cwd=root, env=env)
        print(f'{label}: exit {run.returncode}', flush=True)
        results.append(run.returncode)
    print(f'Exit sequence: {results}', flush=True)

Expected exit sequence: [1, 1, 1] (initial failures, rerun after fixing only b_fixed, full run).
Actual exit sequence: [1, 0, 1]. The second invocation collects only one item:

collecting ... collected 1 item
run-last-failure: rerun previous 1 failure
test_cases.json::b_fixed PASSED
1 passed

The subsequent full run reports a_bad[one] FAILED and b_fixed PASSED.

The failure appears to be in NodeId identity across the string cache boundary. The live custom item has names=("a_bad[one]",), params=None; parsing its cached string gives names=("a_bad",), params="one". These have identical public nodeid strings but compare and hash differently. LFPluginCollWrapper filters items using these structured IDs. If there is only the bracket-named failure, the no-matching-failure fallback runs everything; the second, matching failure is what exposes the omission and false success.

I plan to follow up with a focused fix and regression tests. This report and reproducer were prepared with OpenAI Codex assistance; the exit sequence above was verified by running the script.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the supplied reproduce.py against the current pytest checkout and inspect LFPluginCollWrapper, especially its filtering of structured node IDs and cached strings. Add focused regression coverage for a custom item named a_bad[one] alongside a previously failing item that becomes passing. Done means --lf reruns the bracket-named failure and the regression tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.