pytest-dev / pytest-dev/pytest

Test discovery finds module that doesn't exist

Open
#9,586 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

plugin: doctests topic: collection type: bug
Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

In https://github.com/jaraco/inflect/pull/135, the contributor has accidentally misnamed the package. They attempted to rename inflect (module) to inflect (package). Instead, by accident, they renamed inflect to inflect.inflect and as a result, created an empty inflect module (as a namespace package). I would have expected the tests to fail, but they did not. Something about the test collection caused pytest to find inflect/inflect.py as the top-level inflect module, masking the mistake.

troubleshooting

I tried to create a minimal reproducer, but the reproducer fails as expected:

draft $ tree
.
├── inflect
│   └── inflect.py
└── test_something.py

1 directory, 2 files
draft $ cat inflect/*
value = 'exists'
draft $ cat test*
import inflect


def test_something():
    assert inflect.value == 'exists'
draft $ pip-run pytest -- -m pytest
Collecting pytest
  Using cached pytest-6.2.5-py3-none-any.whl (280 kB)
Collecting py>=1.8.2
  Using cached py-1.11.0-py2.py3-none-any.whl (98 kB)
Collecting packaging
  Using cached packaging-21.3-py3-none-any.whl (40 kB)
Collecting pluggy<2.0,>=0.12
  Using cached pluggy-1.0.0-py2.py3-none-any.whl (13 kB)
Collecting iniconfig
  Using cached iniconfig-1.1.1-py2.py3-none-any.whl (5.0 kB)
Collecting toml
  Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting attrs>=19.2.0
  Using cached attrs-21.4.0-py2.py3-none-any.whl (60 kB)
Collecting pyparsing!=3.0.5,>=2.0.2
  Using cached pyparsing-3.0.7-py3-none-any.whl (98 kB)
Installing collected packages: pyparsing, toml, py, pluggy, packaging, iniconfig, attrs, pytest
Successfully installed attrs-21.4.0 iniconfig-1.1.1 packaging-21.3 pluggy-1.0.0 py-1.11.0 pyparsing-3.0.7 pytest-6.2.5 toml-0.10.2
WARNING: You are using pip version 21.3.1; however, version 22.0.2 is available.
You should consider upgrading via the '/Library/Frameworks/Python.framework/Versions/3.10/bin/python3.10 -m pip install --upgrade pip' command.
====================================================================== test session starts =======================================================================
platform darwin -- Python 3.10.1, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /Users/jaraco/draft
collected 1 item                                                                                                                                                 

test_something.py F                                                                                                                                        [100%]

============================================================================ FAILURES ============================================================================
_________________________________________________________________________ test_something _________________________________________________________________________

    def test_something():
>       assert inflect.value == 'exists'
E       AttributeError: module 'inflect' has no attribute 'value'

test_something.py:5: AttributeError
==================================================================== short test summary info =====================================================================
FAILED test_something.py::test_something - AttributeError: module 'inflect' has no attribute 'value'
======================================================================= 1 failed in 0.03s ========================================================================

I'm confident that one of the plugins (doctests, mypy, black, flake8, ...) is probably implicated, but when I try to enable doctests, I get another error:

draft $ pip-run -q pytest -- -m pytest --doctest-modules
====================================================================== test session starts =======================================================================
platform darwin -- Python 3.10.1, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /Users/jaraco/draft
collected 1 item / 1 error                                                                                                                                       

============================================================================= ERRORS =============================================================================
______________________________________________________________ ERROR collecting inflect/inflect.py _______________________________________________________________
/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-q5qn9lt4/_pytest/runner.py:311: in from_call
    result: Optional[TResult] = func()
/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-q5qn9lt4/_pytest/runner.py:341: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-q5qn9lt4/_pytest/doctest.py:532: in collect
    module = import_path(self.fspath)
/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-q5qn9lt4/_pytest/pathlib.py:533: in import_path
    if module_file.endswith((".pyc", ".pyo")):
E   AttributeError: 'NoneType' object has no attribute 'endswith'
==================================================================== short test summary info =====================================================================
ERROR inflect/inflect.py - AttributeError: 'NoneType' object has no attribute 'endswith'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================================================================== 1 error in 0.10s ========================================================================

It seems that error is occurring because inflect is discovered as a namespace package, but doesn't have a __file__ attribute.

I found that I could minimally reproduce the non-failure by (a) putting the test in a subdirectory and (b) invoking doctests.

draft $ tree
.
├── inflect
│   └── inflect.py
└── tests
    └── test_something.py

2 directories, 2 files
draft $ cat inflect/*
value = 'exists'
draft $ cat tests/*
import inflect


def test_something():
    assert inflect.value == 'exists'
draft $ pip-run -q pytest -- -m pytest --doctest-modules
====================================================================== test session starts =======================================================================
platform darwin -- Python 3.10.1, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /Users/jaraco/draft
collected 1 item                                                                                                                                                 

tests/test_something.py .                                                                                                                                  [100%]

======================================================================= 1 passed in 0.02s ========================================================================

As you can see, the test passes, even though the 'inflect' module should have only been found at inflect.inflect. Remove the --doctest-modules and the expected failure occurs:

draft $ pip-run -q pytest -- -m pytest
====================================================================== test session starts =======================================================================
platform darwin -- Python 3.10.1, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /Users/jaraco/draft
collected 1 item                                                                                                                                                 

tests/test_something.py F                                                                                                                                  [100%]

============================================================================ FAILURES ============================================================================
_________________________________________________________________________ test_something _________________________________________________________________________

    def test_something():
>       assert inflect.value == 'exists'
E       AttributeError: module 'inflect' has no attribute 'value'

tests/test_something.py:5: AttributeError
==================================================================== short test summary info =====================================================================
FAILED tests/test_something.py::test_something - AttributeError: module 'inflect' has no attribute 'value'
======================================================================= 1 failed in 0.03s ========================================================================

I'm fairly certain this missed failure is just another manifestation of #3396 and #8332. In this case, the namespace package was an accident.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reproducer using a namespace package and run pytest --doctest-modules. Read _pytest/doctest.py and _pytest/pathlib.py around doctest collection and import_path, then compare the behavior with issues #3396 and #8332. Done means the accidental namespace package no longer causes pytest to discover the wrong module or silently pass the test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.