pytest-dev / pytest-dev/pytest-xdist

-n auto and tox on windows does not work

Open
#52 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
1.9k
Forks
287
Avg merge
9h 30m
Merged PRs (30d)
2

Description

multiprocessing.cpu_count() uses int(os.environ['NUMBER_OF_PROCESSORS']) to get the number of CPUs. I guess this does not work as tox cleans all the environment variables. Thus I suggest using another approach. There are several listed in http://stackoverflow.com/questions/1006289/how-to-find-out-the-number-of-cpus-using-python.

File "C:\Python27\Lib\runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "C:\Python27\Lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "c:\windows\temp\devpi-test829\ngc-tsg-1.17.0.3687097\.tox\python\Scripts\py.test.EXE\__main__.py", line 9, in <module>
  File "c:\windows\temp\devpi-test829\ngc-tsg-1.17.0.3687097\.tox\python\lib\site-packages\_pytest\config.py", line 39, in main
    config = _prepareconfig(args, plugins)
  File "c:\windows\temp\devpi-test829\ngc-tsg-1.17.0.3687097\.tox\python\lib\site-packages\_pytest\config.py", line 118, in _prepareconfig
    pluginmanager=pluginmanager, args=args)
  File "c:\windows\temp\devpi-test829\ngc-tsg-1.17.0.3687097\.tox\python\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 724, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "c:\windows\temp\devpi-test829\ngc-tsg-1.17.0.3687097\.tox\python\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 338, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "c:\windows\temp\devpi-test829\ngc-tsg-1.17.0.3687097\.tox\python\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 333, in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
  File "c:\windows\temp\devpi-test829\ngc-tsg-1.17.0.3687097\.tox\python\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 595, in execute
    return _wrapped_call(hook_impl.function(*args), self.execute)
  File "c:\windows\temp\devpi-test829\ngc-tsg-1.17.0.3687097\.tox\python\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 249, in _wrapped_call
    wrap_controller.send(call_outcome)
  File "c:\windows\temp\devpi-test829\ngc-tsg-1.17.0.3687097\.tox\python\lib\site-packages\_pytest\helpconfig.py", line 28, in pytest_cmdline_parse
    config = outcome.get_result()
  File "c:\windows\temp\devpi-test829\ngc-tsg-1.17.0.3687097\.tox\python\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 279, in get_result
    _reraise(*ex)  # noqa
  File "c:\windows\temp\devpi-test829\ngc-tsg-1.17.0.3687097\.tox\python\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 264, in __init__
    self.result = func()
  File "c:\windows\temp\devpi-test829\ngc-tsg-1.17.0.3687097\.tox\python\lib\site-packages\_pytest\vendored_packages\pluggy.py", line 596, in execute
    res = hook_impl.function(*args)
  File "c:\windows\temp\devpi-test829\ngc-tsg-1.17.0.3687097\.tox\python\lib\site-packages\_pytest\config.py", line 856, in pytest_cmdline_parse
    self.parse(args)
  File "c:\windows\temp\devpi-test829\ngc-tsg-1.17.0.3687097\.tox\python\lib\site-packages\_pytest\config.py", line 961, in parse
    self._preparse(args, addopts=addopts)
  File "c:\windows\temp\devpi-test829\ngc-tsg-1.17.0.3687097\.tox\python\lib\site-packages\_pytest\config.py", line 926, in _preparse
    self.known_args_namespace = ns = self._parser.parse_known_args(args, namespace=self.option.copy())
  File "c:\windows\temp\devpi-test829\ngc-tsg-1.17.0.3687097\.tox\python\lib\site-packages\_pytest\config.py", line 491, in parse_known_args
    return self.parse_known_and_unknown_args(args, namespace=namespace)[0]
  File "c:\windows\temp\devpi-test829\ngc-tsg-1.17.0.3687097\.tox\python\lib\site-packages\_pytest\config.py", line 499, in parse_known_and_unknown_args
    return optparser.parse_known_args(args, namespace=namespace)
  File "C:\Python27\Lib\argparse.py", line 1720, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "C:\Python27\Lib\argparse.py", line 1926, in _parse_known_args
    start_index = consume_optional(start_index)
  File "C:\Python27\Lib\argparse.py", line 1866, in consume_optional
    take_action(action, args, option_string)
  File "C:\Python27\Lib\argparse.py", line 1778, in take_action
    argument_values = self._get_values(action, argument_strings)
  File "C:\Python27\Lib\argparse.py", line 2204, in _get_values
    value = self._get_value(action, arg_string)
  File "C:\Python27\Lib\argparse.py", line 2233, in _get_value
    result = type_func(arg_string)
  File "c:\windows\temp\devpi-test829\ngc-tsg-1.17.0.3687097\.tox\python\lib\site-packages\xdist\plugin.py", line 8, in parse_numprocesses
    return multiprocessing.cpu_count()
  File "C:\Python27\Lib\multiprocessing\__init__.py", line 136, in cpu_count
    raise NotImplementedError('cannot determine number of cpus')
NotImplementedError: cannot determine number of cpus

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in xdist/plugin.py at parse_numprocesses and compare its multiprocessing.cpu_count() call with the Windows traceback showing failure under tox. Reproduce the -n auto case on Windows with tox, then verify that automatic process selection no longer raises NotImplementedError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.