pytest-dev / pytest-dev/pytest-xdist

Options with nonserializable types break under xdist

Open
#384 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
1.9k
Forks
287
Avg merge
9h 30m
Merged PRs (30d)
2

Description

I wanted a command-line option to take a filename as its value. It works fine with vanilla py.test, but not with xdist.

$ cat test_stuff.py 
def test_stuff():
    pass

$ cat conftest.py 
import pathlib
import pytest
@pytest.hookimpl
def pytest_addoption(parser):
    parser.addoption("--my-option", type=pathlib.Path)

$ py.test --my-option=/
============================= test session starts ==============================
platform linux -- Python 3.6.5, pytest-4.0.1, py-1.7.0, pluggy-0.8.0
rootdir: /tmp/test, inifile:
plugins: xdist-1.24.1, forked-0.2
collected 1 item                                                               

test_stuff.py .                                                          [100%]

=========================== 1 passed in 0.01 seconds ===========================

$ py.test --my-option=/ -n2
================================================== test session starts ==================================================
platform linux -- Python 3.6.5, pytest-4.0.1, py-1.7.0, pluggy-0.8.0
rootdir: /tmp/test, inifile:
plugins: xdist-1.24.1, forked-0.2
gw0 C / gw1 IINTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/tmp/venv/lib64/python3.6/site-packages/execnet/gateway_base.py", line 1383, in _save
INTERNALERROR>     dispatch = self._dispatch[tp]
INTERNALERROR> KeyError: <class 'pathlib.PosixPath'>
INTERNALERROR> 
INTERNALERROR> During handling of the above exception, another exception occurred:
INTERNALERROR> 
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/tmp/venv/lib64/python3.6/site-packages/_pytest/main.py", line 183, in wrap_session
INTERNALERROR>     config.hook.pytest_sessionstart(session=session)
INTERNALERROR>   File "/tmp/venv/lib64/python3.6/site-packages/pluggy/hooks.py", line 284, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/tmp/venv/lib64/python3.6/site-packages/pluggy/manager.py", line 67, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/tmp/venv/lib64/python3.6/site-packages/pluggy/manager.py", line 61, in <lambda>
INTERNALERROR>     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR>   File "/tmp/venv/lib64/python3.6/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/tmp/venv/lib64/python3.6/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/tmp/venv/lib64/python3.6/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/tmp/venv/lib64/python3.6/site-packages/xdist/dsession.py", line 81, in pytest_sessionstart
INTERNALERROR>     nodes = self.nodemanager.setup_nodes(putevent=self.queue.put)
INTERNALERROR>   File "/tmp/venv/lib64/python3.6/site-packages/xdist/workermanage.py", line 67, in setup_nodes
INTERNALERROR>     nodes.append(self.setup_node(spec, putevent))
INTERNALERROR>   File "/tmp/venv/lib64/python3.6/site-packages/xdist/workermanage.py", line 76, in setup_node
INTERNALERROR>     node.setup()
INTERNALERROR>   File "/tmp/venv/lib64/python3.6/site-packages/xdist/workermanage.py", line 246, in setup
INTERNALERROR>     self.channel.send((self.workerinput, args, option_dict))
INTERNALERROR>   File "/tmp/venv/lib64/python3.6/site-packages/execnet/gateway_base.py", line 717, in send
INTERNALERROR>     self.gateway._send(Message.CHANNEL_DATA, self.id, dumps_internal(item))
INTERNALERROR>   File "/tmp/venv/lib64/python3.6/site-packages/execnet/gateway_base.py", line 1354, in dumps_internal
INTERNALERROR>     return _Serializer().save(obj)
INTERNALERROR>   File "/tmp/venv/lib64/python3.6/site-packages/execnet/gateway_base.py", line 1372, in save
INTERNALERROR>     self._save(obj)
INTERNALERROR>   File "/tmp/venv/lib64/python3.6/site-packages/execnet/gateway_base.py", line 1390, in _save
INTERNALERROR>     dispatch(self, obj)
INTERNALERROR>   File "/tmp/venv/lib64/python3.6/site-packages/execnet/gateway_base.py", line 1475, in save_tuple
INTERNALERROR>     self._save(item)
INTERNALERROR>   File "/tmp/venv/lib64/python3.6/site-packages/execnet/gateway_base.py", line 1390, in _save
INTERNALERROR>     dispatch(self, obj)
INTERNALERROR>   File "/tmp/venv/lib64/python3.6/site-packages/execnet/gateway_base.py", line 1471, in save_dict
INTERNALERROR>     self._write_setitem(key, value)
INTERNALERROR>   File "/tmp/venv/lib64/python3.6/site-packages/execnet/gateway_base.py", line 1465, in _write_setitem
INTERNALERROR>     self._save(value)
INTERNALERROR>   File "/tmp/venv/lib64/python3.6/site-packages/execnet/gateway_base.py", line 1388, in _save
INTERNALERROR>     raise DumpError("can't serialize %s" % (tp,))
INTERNALERROR> execnet.gateway_base.DumpError: can't serialize <class 'pathlib.PosixPath'>

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Run the supplied minimal reproduction with pytest and xdist, then trace option_dict through xdist/workermanage.py and the execnet serialization failure shown in the traceback. Done means a pathlib.Path option value works with -n2 without an internal error, while the existing vanilla pytest behavior remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.