pytest-dev / pytest-dev/pytest-xdist
Options with nonserializable types break under xdist
Open
Nobody has claimed this yet.
bug
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 287
- Avg merge
- 9h 30m
- Merged PRs (30d)
- 2
Description
I wanted a command-line option to take a filename as its value. It works fine with vanilla py.test, but not with xdist.
$ cat test_stuff.py
def test_stuff():
pass
$ cat conftest.py
import pathlib
import pytest
@pytest.hookimpl
def pytest_addoption(parser):
parser.addoption("--my-option", type=pathlib.Path)
$ py.test --my-option=/
============================= test session starts ==============================
platform linux -- Python 3.6.5, pytest-4.0.1, py-1.7.0, pluggy-0.8.0
rootdir: /tmp/test, inifile:
plugins: xdist-1.24.1, forked-0.2
collected 1 item
test_stuff.py . [100%]
=========================== 1 passed in 0.01 seconds ===========================
$ py.test --my-option=/ -n2
================================================== test session starts ==================================================
platform linux -- Python 3.6.5, pytest-4.0.1, py-1.7.0, pluggy-0.8.0
rootdir: /tmp/test, inifile:
plugins: xdist-1.24.1, forked-0.2
gw0 C / gw1 IINTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/tmp/venv/lib64/python3.6/site-packages/execnet/gateway_base.py", line 1383, in _save
INTERNALERROR> dispatch = self._dispatch[tp]
INTERNALERROR> KeyError: <class 'pathlib.PosixPath'>
INTERNALERROR>
INTERNALERROR> During handling of the above exception, another exception occurred:
INTERNALERROR>
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/tmp/venv/lib64/python3.6/site-packages/_pytest/main.py", line 183, in wrap_session
INTERNALERROR> config.hook.pytest_sessionstart(session=session)
INTERNALERROR> File "/tmp/venv/lib64/python3.6/site-packages/pluggy/hooks.py", line 284, in __call__
INTERNALERROR> return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR> File "/tmp/venv/lib64/python3.6/site-packages/pluggy/manager.py", line 67, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> File "/tmp/venv/lib64/python3.6/site-packages/pluggy/manager.py", line 61, in <lambda>
INTERNALERROR> firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR> File "/tmp/venv/lib64/python3.6/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR> return outcome.get_result()
INTERNALERROR> File "/tmp/venv/lib64/python3.6/site-packages/pluggy/callers.py", line 80, in get_result
INTERNALERROR> raise ex[1].with_traceback(ex[2])
INTERNALERROR> File "/tmp/venv/lib64/python3.6/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> File "/tmp/venv/lib64/python3.6/site-packages/xdist/dsession.py", line 81, in pytest_sessionstart
INTERNALERROR> nodes = self.nodemanager.setup_nodes(putevent=self.queue.put)
INTERNALERROR> File "/tmp/venv/lib64/python3.6/site-packages/xdist/workermanage.py", line 67, in setup_nodes
INTERNALERROR> nodes.append(self.setup_node(spec, putevent))
INTERNALERROR> File "/tmp/venv/lib64/python3.6/site-packages/xdist/workermanage.py", line 76, in setup_node
INTERNALERROR> node.setup()
INTERNALERROR> File "/tmp/venv/lib64/python3.6/site-packages/xdist/workermanage.py", line 246, in setup
INTERNALERROR> self.channel.send((self.workerinput, args, option_dict))
INTERNALERROR> File "/tmp/venv/lib64/python3.6/site-packages/execnet/gateway_base.py", line 717, in send
INTERNALERROR> self.gateway._send(Message.CHANNEL_DATA, self.id, dumps_internal(item))
INTERNALERROR> File "/tmp/venv/lib64/python3.6/site-packages/execnet/gateway_base.py", line 1354, in dumps_internal
INTERNALERROR> return _Serializer().save(obj)
INTERNALERROR> File "/tmp/venv/lib64/python3.6/site-packages/execnet/gateway_base.py", line 1372, in save
INTERNALERROR> self._save(obj)
INTERNALERROR> File "/tmp/venv/lib64/python3.6/site-packages/execnet/gateway_base.py", line 1390, in _save
INTERNALERROR> dispatch(self, obj)
INTERNALERROR> File "/tmp/venv/lib64/python3.6/site-packages/execnet/gateway_base.py", line 1475, in save_tuple
INTERNALERROR> self._save(item)
INTERNALERROR> File "/tmp/venv/lib64/python3.6/site-packages/execnet/gateway_base.py", line 1390, in _save
INTERNALERROR> dispatch(self, obj)
INTERNALERROR> File "/tmp/venv/lib64/python3.6/site-packages/execnet/gateway_base.py", line 1471, in save_dict
INTERNALERROR> self._write_setitem(key, value)
INTERNALERROR> File "/tmp/venv/lib64/python3.6/site-packages/execnet/gateway_base.py", line 1465, in _write_setitem
INTERNALERROR> self._save(value)
INTERNALERROR> File "/tmp/venv/lib64/python3.6/site-packages/execnet/gateway_base.py", line 1388, in _save
INTERNALERROR> raise DumpError("can't serialize %s" % (tp,))
INTERNALERROR> execnet.gateway_base.DumpError: can't serialize <class 'pathlib.PosixPath'>
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Run the supplied minimal reproduction with pytest and xdist, then trace option_dict through xdist/workermanage.py and the execnet serialization failure shown in the traceback. Done means a pathlib.Path option value works with -n2 without an internal error, while the existing vanilla pytest behavior remains intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100