pytest-dev / pytest-dev/pytest-xdist

ImportError with --looponfail and "python -m pytest" if project is not installed

Open
#335 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
1.9k
Forks
287
Avg merge
9h 30m
Merged PRs (30d)
2

Description

For some reason I am getting an ImportError for the project itself when using --looponfail with python -m pytest, and the project not being installed via pip install -e ..

The workaround is to either pip install -e . the project, or use -o django_find_project=1, which will massage sys.path based on the manage.py it finds.

This does not happen when omitting --looponfail, so there appears to be something different in this regard in how sys.path is handled.

% python -m pytest -o rsyncdirs=project --tx ssh=gw0.XXX.eu-west-1.compute.internal.xdistnode//chdir=/srv/project/src --dist=load '-o rsyncdirs=project' --looponfail --cov=project --cov-report=term-missing:skip-covered --cov-report=xml --run-slow-tests --ff
Traceback (most recent call last):                                                                                                                                                            
  File "…/pyenv/3.6.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main                                                                                          
    "__main__", mod_spec)                                                                                                                                                                     
  File "…/pyenv/3.6.6/lib/python3.6/runpy.py", line 85, in _run_code                                                                                                     
    exec(code, run_globals)                                                                                                                                                                   
  File "…/project/.venv/lib/python3.6/site-packages/pytest.py", line 68, in <module>                                                                              
    raise SystemExit(pytest.main())                                                                                                                                                           
  File "…/project/.venv/lib/python3.6/site-packages/_pytest/config/__init__.py", line 65, in main                                                                 
    return config.hook.pytest_cmdline_main(config=config)                                                                                                                                     
  File "…/project/.venv/lib/python3.6/site-packages/pluggy/hooks.py", line 258, in __call__                                                                       
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)                                                                                                                   
  File "…/project/.venv/lib/python3.6/site-packages/pluggy/manager.py", line 67, in _hookexec                                                                     
    return self._inner_hookexec(hook, methods, kwargs)                                                                                                                                        
  File "…/project/.venv/lib/python3.6/site-packages/pluggy/manager.py", line 61, in <lambda>                                                                      
    firstresult=hook.spec_opts.get('firstresult'),                                                                                                                                            
  File "…/project/.venv/lib/python3.6/site-packages/pluggy/callers.py", line 201, in _multicall                                                                   
    return outcome.get_result()                                                                                                                                                               
  File "…/project/.venv/lib/python3.6/site-packages/pluggy/callers.py", line 76, in get_result                                                                    
    raise ex[1].with_traceback(ex[2])                                                                                                                                                         
  File "…/project/.venv/lib/python3.6/site-packages/pluggy/callers.py", line 180, in _multicall                                                                   
    res = hook_impl.function(*args)                                                                                                                                                           
  File "…/Vcs/pytest-xdist/xdist/looponfail.py", line 36, in pytest_cmdline_main                                                                                                   
    looponfail_main(config)                                                                                                                                                                   
  File "…/Vcs/pytest-xdist/xdist/looponfail.py", line 45, in looponfail_main
    try:
  File "…/Vcs/pytest-xdist/xdist/looponfail.py", line 119, in loop_once
    result = self.runsession()
  File "…/Vcs/pytest-xdist/xdist/looponfail.py", line 108, in runsession
    return self.channel.receive()
  File "…/project/.venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 737, in receive
    raise self._getremoteerror() or EOFError()
execnet.gateway_base.RemoteError: Traceback (most recent call last):
  File "…/Vcs/pytest-django/pytest_django/plugin.py", line 114, in _handle_import_error
    yield
  File "…/Vcs/pytest-django/pytest_django/plugin.py", line 259, in pytest_load_initial_conftests
    dj_settings.DATABASES
  File "…/project/.venv/lib/python3.6/site-packages/django/conf/__init__.py", line 56, in __getattr__
    self._setup(name)
  File "…/project/.venv/lib/python3.6/site-packages/django/conf/__init__.py", line 43, in _setup
    self._wrapped = Settings(settings_module)
  File "…/project/.venv/lib/python3.6/site-packages/django/conf/__init__.py", line 106, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "…/pyenv/3.6.6/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'project'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "…/project/.venv/lib/python3.6/site-packages/execnet/gateway_base.py", line 1076, in executetask
    function(channel, **kwargs)
  File "<remote exec>", line 20, in init_worker_session
  File "…/project/.venv/lib/python3.6/site-packages/_pytest/config/__init__.py", line 646, in fromdictargs
    config.parse(args, addopts=False)
  File "…/project/.venv/lib/python3.6/site-packages/_pytest/config/__init__.py", line 782, in parse
    self._preparse(args, addopts=addopts)
  File "…/project/.venv/lib/python3.6/site-packages/_pytest/config/__init__.py", line 744, in _preparse
    early_config=self, args=args, parser=self._parser
  File "…/project/.venv/lib/python3.6/site-packages/pluggy/hooks.py", line 258, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "…/project/.venv/lib/python3.6/site-packages/pluggy/manager.py", line 67, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "…/project/.venv/lib/python3.6/site-packages/pluggy/manager.py", line 61, in <lambda>
    firstresult=hook.spec_opts.get('firstresult'),
  File "…/project/.venv/lib/python3.6/site-packages/pluggy/callers.py", line 201, in _multicall
    return outcome.get_result()
  File "…/project/.venv/lib/python3.6/site-packages/pluggy/callers.py", line 76, in get_result
    raise ex[1].with_traceback(ex[2])
  File "…/project/.venv/lib/python3.6/site-packages/pluggy/callers.py", line 180, in _multicall
    res = hook_impl.function(*args)
  File "…/Vcs/pytest-django/pytest_django/plugin.py", line 259, in pytest_load_initial_conftests
    dj_settings.DATABASES
  File "…/pyenv/3.6.6/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "…/Vcs/pytest-django/pytest_django/plugin.py", line 118, in _handle_import_error
    raise ImportError(msg)
ImportError: No module named 'project'

pytest-django did not search for Django projects since it is disabled in the configuration ("django_find_project = false")

Version v1.22.5-1-g96af694

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with xdist/looponfail.py, especially looponfail_main, loop_once, and runsession, and compare worker startup with normal python -m pytest behavior. Read the referenced pytest_django/plugin.py path and reproduce the reported command without installing the project. Done means loop-on-fail handles the project import consistently without breaking the documented django_find_project workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
django, python
Domain
distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.