pytest-dev / pytest-dev/pytest-xdist

Network connectitivity issues with xdist

Open
#940 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.9k
Forks
287
Avg merge
9h 30m
Merged PRs (30d)
2

Description

I am trying to run a subset of tests with pytest of which many require an internet connection, to install packages from pip.

I have verified the test suite passes, and enabled test parallelization using pytest-xdist. I am using auto-detection for the number of processes. Now I experience that various tests are failing because of broken internet connection. An extract from a log file containing the pip error:

PIP STDOUT
----------
Looking in indexes: http://username:****@127.0.0.1:8080/simple
Collecting zest-releaser==7.1.0
Downloading http://127.0.0.1:8080/packages/zest.releaser-7.1.0-py3-none-any.whl (93 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 93.7/93.7 kB 250.7 MB/s eta 0:00:00
Requirement already satisfied: setuptools in /tmp/pytest-of-user/pytest-0/popen-gw3/session_shareddir0/lib/python3.10/site-packages (from zest-releaser==7.1.0) (65.6.3)
Collecting requests
Downloading http://127.0.0.1:8080/packages/requests-2.28.1-py3-none-any.whl (62 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.8/62.8 kB 207.4 MB/s eta 0:00:00
Collecting colorama
Downloading http://127.0.0.1:8080/packages/colorama-0.4.6-py2.py3-none-any.whl (25 kB)
Collecting twine>=1.6.0
Downloading http://127.0.0.1:8080/packages/twine-4.0.2-py3-none-any.whl (36 kB)
Collecting urllib3>=1.26.0
Downloading http://127.0.0.1:8080/packages/urllib3-1.26.13-py2.py3-none-any.whl (140 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 140.6/140.6 kB 229.1 MB/s eta 0:00:00
Collecting importlib-metadata>=3.6
Downloading http://127.0.0.1:8080/packages/importlib_metadata-5.1.0-py3-none-any.whl (21 kB)
Collecting keyring>=15.1
Downloading http://127.0.0.1:8080/packages/keyring-23.11.0-py3-none-any.whl (36 kB)
Collecting rich>=12.0.0
Downloading http://127.0.0.1:8080/packages/rich-12.6.0-py3-none-any.whl (237 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 237.5/237.5 kB 277.7 MB/s eta 0:00:00
Collecting requests-toolbelt!=0.9.0,>=0.8.0
Downloading http://127.0.0.1:8080/packages/requests_toolbelt-0.10.1-py2.py3-none-any.whl (54 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.5/54.5 kB 132.3 MB/s eta 0:00:00
Collecting readme-renderer>=35.0
Downloading http://127.0.0.1:8080/packages/readme_renderer-37.3-py3-none-any.whl (14 kB)
Collecting rfc3986>=1.4.0
Downloading http://127.0.0.1:8080/packages/rfc3986-2.0.0-py2.py3-none-any.whl (31 kB)
Collecting pkginfo>=1.8.1
Downloading http://127.0.0.1:8080/packages/pkginfo-1.9.2-py3-none-any.whl (26 kB)
Collecting idna<4,>=2.5
Downloading http://127.0.0.1:8080/packages/idna-3.4-py3-none-any.whl (61 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.5/61.5 kB 107.2 MB/s eta 0:00:00
Collecting charset-normalizer<3,>=2
Downloading http://127.0.0.1:8080/packages/charset_normalizer-2.1.1-py3-none-any.whl (39 kB)
Collecting certifi>=2017.4.17
Downloading http://127.0.0.1:8080/packages/certifi-2022.9.24-py3-none-any.whl (161 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 161.1/161.1 kB 265.6 MB/s eta 0:00:00
Collecting zipp>=0.5
Downloading http://127.0.0.1:8080/packages/zipp-3.11.0-py3-none-any.whl (6.6 kB)
Collecting jeepney>=0.4.2
Downloading http://127.0.0.1:8080/packages/jeepney-0.8.0-py3-none-any.whl (48 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.4/48.4 kB 108.2 MB/s eta 0:00:00
Collecting jaraco.classes
Downloading http://127.0.0.1:8080/packages/jaraco.classes-3.2.3-py3-none-any.whl (6.0 kB)
Collecting SecretStorage>=3.2
Downloading http://127.0.0.1:8080/packages/SecretStorage-3.3.3-py3-none-any.whl (15 kB)
Collecting bleach>=2.1.0
Downloading http://127.0.0.1:8080/packages/bleach-5.0.1-py3-none-any.whl (160 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 160.9/160.9 kB 169.0 MB/s eta 0:00:00
Collecting Pygments>=2.5.1
Downloading http://127.0.0.1:8080/packages/Pygments-2.13.0-py3-none-any.whl (1.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 170.9 MB/s eta 0:00:00
Collecting docutils>=0.13.1
Downloading http://127.0.0.1:8080/packages/docutils-0.19-py3-none-any.whl (570 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 570.5/570.5 kB 124.7 MB/s eta 0:00:00
INFO: pip is looking at multiple versions of rfc3986 to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of requests-toolbelt to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of readme-renderer to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of pkginfo to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of keyring to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of importlib-metadata to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of idna to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of charset-normalizer to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of certifi to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of colorama to determine which version is compatible with other requirements. This could take a while.
Collecting colorama

PIP STDERR
----------
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))': /simple/commonmark/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPConnection object at 0x7fb91b878b80>: Failed to establish a new connection: [Errno 111] Connection refused')': /simple/commonmark/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPConnection object at 0x7fb91b878d90>: Failed to establish a new connection: [Errno 111] Connection refused')': /simple/commonmark/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPConnection object at 0x7fb91b878fa0>: Failed to establish a new connection: [Errno 111] Connection refused')': /simple/commonmark/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPConnection object at 0x7fb91b840c70>: Failed to establish a new connection: [Errno 111] Connection refused')': /simple/commonmark/
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPConnection object at 0x7fb91b7f7fa0>: Failed to establish a new connection: [Errno 111] Connection refused')': /packages/colorama-0.4.3-py2.py3-none-any.whl
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPConnection object at 0x7fb91b854970>: Failed to establish a new connection: [Errno 111] Connection refused')': /packages/colorama-0.4.3-py2.py3-none-any.whl
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPConnection object at 0x7fb91b856740>: Failed to establish a new connection: [Errno 111] Connection refused')': /packages/colorama-0.4.3-py2.py3-none-any.whl
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPConnection object at 0x7fb91b855cc0>: Failed to establish a new connection: [Errno 111] Connection refused')': /packages/colorama-0.4.3-py2.py3-none-any.whl
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPConnection object at 0x7fb91b8560e0>: Failed to establish a new connection: [Errno 111] Connection refused')': /packages/colorama-0.4.3-py2.py3-none-any.whl
ERROR: Could not install packages due to an OSError: HTTPConnectionPool(host='127.0.0.1', port=8080): Max retries exceeded with url: /packages/colorama-0.4.3-py2.py3-none-any.whl (Caused by NewConnectionError('<pip._vendor.urllib3.connection.HTTPConnection object at 0x7fb91b857fd0>: Failed to establish a new connection: [Errno 111] Connection refused'))

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names pytest, pytest-xdist, and pip but no repository files or tests. Start by reproducing the reported subset with pytest-xdist's automatically detected process count and inspect the pip connection failures; done means the affected tests can install their packages reliably while running in parallel.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devtools, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.