pytest-dev / pytest-dev/pytest-xdist

`gevent` execmode hangs in python 3.7.5

Open
#653 0 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.9k
Forks
288
Avg merge
9h 30m
Merged PRs (30d)
2

Description

The problem only occurs at python3.

Environment:

  • OS: Linux faa410b1f09b 4.19.121-linuxkit #1 SMP Thu Jan 21 15:36:34 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
  • Python version: Python 3.7.5

Dependencies:

apipkg         1.5
attrs          20.3.0
execnet        1.8.0
gevent         21.1.2
greenlet       1.0.0
iniconfig      1.1.1
packaging      20.9
pip            19.2.3
pluggy         0.13.1
py             1.10.0
pyparsing      2.4.7
pytest         6.2.3
pytest-forked  1.3.0
pytest-xdist   2.2.1
setuptools     41.2.0
toml           0.10.2
zope.event     4.5.0
zope.interface 5.4.0

Command: pytest -d --tx popen//execmodel=gevent test.py

Code:

from gevent import monkey
monkey.patch_all()


def test_assert_0():
    assert 0


def test_assert_1():
    assert 1

In Python 2 environment command runs all tests, reports failure on test_assert_0 and ends. In Python 3 environment test hangs before scanning for tests.

Execnet debug logs:

====================================== test session starts =======================================
platform darwin -- Python 3.8.2, pytest-6.2.3, py-1.10.0, pluggy-0.13.1
rootdir: /Users/changwan.park/Documents/py3_venv
plugins: xdist-2.2.1, forked-1.3.0
gw0 C[13946] gw0 [receiver-thread] RECEIVERTHREAD: starting to run
[13946] gw0 sent <Message CHANNEL_EXEC channel=1 lendata=8631>
[13946] gw0 sent <Message CHANNEL_DATA channel=1 lendata=470>
[13948] creating workergateway on <execnet.gateway_base.Popen2IO object at 0x10dc89bb0>
[13948] gw0-worker [serve] spawning receiver thread
[13948] gw0-worker [serve] joining receiver thread
[13948] gw0-worker waiting for receiver thread to finish
[13948] gw0-worker [receiver-thread] RECEIVERTHREAD: starting to run
[13948] gw0-worker [receiver-thread] received <Message CHANNEL_EXEC channel=1 lendata=8631>
[13948] gw0-worker [receiver-thread] received <Message CHANNEL_DATA channel=1 lendata=470>

Different issue compared with https://github.com/pytest-dev/execnet/issues/75. By comparing execnet debug trace, above example hangs after gw0-worker only receives gw0but does not execute the received command.

After some digging, I found out the hanging occurres since remote_exec hangs. See https://github.com/pytest-dev/execnet/issues/123.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue with the shown pytest command and test.py under Python 3.7.5, then compare the execnet debug trace with the Python 2 behavior. Start by reading the referenced execnet remote_exec issue and the gevent execmode path; done means the command no longer hangs before test collection and reports the expected failing assertion.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.