dask / dask/distributed

pytest collection errors after following setup guidelines

Open
#5,212 5 comments 0 reactions 0 assignees View on GitHub
documentation
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

**What happened**:
After following the [new contributor guidelines for distributed](https://distributed.dask.org/en/latest/develop.html), pytest finds both errors in test collection and failing tests.

**What you expected to happen**:
I had expected the pytest suite to complete without failures.

Presumably there is something wonky with my sockets, but I haven't done anything strange recently and should have the defaults for everything in Ubuntu.

**Minimal Complete Verifiable Example**:

```
git clone git@github.com:dask/distributed.git
cd distributed
conda env create --file continuous_integration/environment-3.8.yaml
conda activate dask-distributed
python -m pip install -e .
py.test distributed --verbose
```

pytest collection errors:

```
$ py.test distributed --verbose
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /home/genevieve/anaconda3/envs/dask-distributed/bin/python
cachedir: .pytest_cache
rootdir: /home/genevieve/GitHub/temp/distributed, configfile: setup.cfg
plugins: asyncio-0.12.0, rerunfailures-10.1, repeat-0.8.0, timeout-1.4.2
timeout: 300.0s
timeout method: thread
timeout func_only: False
collected 2180 items / 2 errors / 10 skipped / 2168 selected                 

==================================== ERRORS ====================================
____________ ERROR collecting distributed/comm/tests/test_comms.py _____________
distributed/utils.py:133: in _get_ip
    sock.connect((host, port))
E  OSError: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:
distributed/comm/tests/test_comms.py:47: in
    EXTERNAL_IP6 = get_ipv6()
distributed/utils.py:164: in get_ipv6
    return _get_ip(host, port, family=socket.AF_INET6)
cytoolz/functoolz.pyx:476: in cytoolz.functoolz._memoize.__call__
    ???
distributed/utils.py:142: in _get_ip
    addr_info = socket.getaddrinfo(
../../../anaconda3/envs/dask-distributed/lib/python3.8/socket.py:918: in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
E  socket.gaierror: [Errno -5] No address associated with hostname
______________ ERROR collecting distributed/comm/tests/test_ws.py ______________
distributed/utils.py:133: in _get_ip
    sock.connect((host, port))
E  OSError: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:
distributed/comm/tests/test_ws.py:23: in
    from .test_comms import check_tls_extra
:991: in _find_and_load
    ???
:975: in _find_and_load_unlocked
    ???
:671: in _load_unlocked
    ???
../../../anaconda3/envs/dask-distributed/lib/python3.8/site-packages/_pytest/assertion/rewrite.py:170: in exec_module
    exec(co, module.__dict__)
distributed/comm/tests/test_comms.py:47: in
    EXTERNAL_IP6 = get_ipv6()
distributed/utils.py:164: in get_ipv6
    return _get_ip(host, port, family=socket.AF_INET6)
cytoolz/functoolz.pyx:476: in cytoolz.functoolz._memoize.__call__
    ???
distributed/utils.py:142: in _get_ip
    addr_info = socket.getaddrinfo(
../../../anaconda3/envs/dask-distributed/lib/python3.8/socket.py:918: in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
E  socket.gaierror: [Errno -5] No address associated with hostname
=========================== short test summary info ============================
SKIPPED [1] distributed/comm/tests/test_ucx.py:7: could not import 'ucp': No module named 'ucp'
SKIPPED [1] distributed/comm/tests/test_ucx_config.py:19: could not import 'ucp': No module named 'ucp'
SKIPPED [1] distributed/diagnostics/tests/test_nvml.py:7: could not import 'pynvml': No module named 'pynvml'
SKIPPED [1] distributed/protocol/tests/test_arrow.py:3: could not import 'pyarrow': No module named 'pyarrow'
SKIPPED [1] distributed/protocol/tests/test_cupy.py:9: could not import 'cupy': No module named 'cupy'
SKIPPED [1] distributed/protocol/tests/test_keras.py:3: could not import 'keras': No module named 'tensorflow'
SKIPPED [1] distributed/protocol/tests/test_numba.py:9: could not import 'numba.cuda': No module named 'numba'
SKIPPED [1] distributed/protocol/tests/test_rmm.py:8: could not import 'numba.cuda': No module named 'numba'
SKIPPED [1] distributed/protocol/tests/test_sparse.py:4: could not import 'sparse': No module named 'sparse'
SKIPPED [1] distributed/protocol/tests/test_torch.py:6: could not import 'torch': No module named 'torch'
ERROR distributed/comm/tests/test_comms.py - socket.gaierror: [Errno -5] No a...
ERROR distributed/comm/tests/test_ws.py - socket.gaierror: [Errno -5] No addr...
!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!

```

**Anything else we need to know?**:

The test collection errors occur in `pytest distributed/comm/tests`

Additionally, several cli tests fail on my machine as well: `pytest distributed/cli/tests`
```
FAILED distributed/cli/tests/test_dask_scheduler.py::test_defaults - socket.g...
FAILED distributed/cli/tests/test_dask_scheduler.py::test_hostport - socket.g...
FAILED distributed/cli/tests/test_dask_scheduler.py::test_no_dashboard - asse...
FAILED distributed/cli/tests/test_dask_scheduler.py::test_dashboard_whitelist
```

Details:

```
=================================== FAILURES ===================================
________________________________ test_defaults _________________________________

host = '2001:4860:4860::8888', port = 80, family =

    @toolz.memoize
    def _get_ip(host, port, family):
        # By using a UDP socket, we don't actually try to connect but
        # simply select the local address through which *host* is reachable.
        sock = socket.socket(family, socket.SOCK_DGRAM)
        try:
>          sock.connect((host, port))
E          OSError: [Errno 101] Network is unreachable

distributed/utils.py:133: OSError

During handling of the above exception, another exception occurred:

loop =

    def test_defaults(loop):
        with popen(["dask-scheduler"]):
   
            async def f():
                # Default behaviour is to listen on all addresses
                await assert_can_connect_from_everywhere_4_6(8786, timeout=5.0)
   
            with Client(f"127.0.0.1:{Scheduler.default_port}", loop=loop) as c:
>              c.sync(f)

distributed/cli/tests/test_dask_scheduler.py:36:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
distributed/client.py:845: in sync
    return sync(
distributed/utils.py:325: in sync
    raise exc.with_traceback(tb)
distributed/utils.py:308: in f
    result[0] = yield future
../../../anaconda3/envs/dask-distributed/lib/python3.8/site-packages/tornado/gen.py:762: in run
    value = future.result()
distributed/cli/tests/test_dask_scheduler.py:33: in f
    await assert_can_connect_from_everywhere_4_6(8786, timeout=5.0)
distributed/utils_test.py:1193: in assert_can_connect_from_everywhere_4_6
    assert_can_connect("%s://[%s]:%d" % (protocol, get_ipv6(), port), **kwargs),
distributed/utils.py:164: in get_ipv6
    return _get_ip(host, port, family=socket.AF_INET6)
cytoolz/functoolz.pyx:476: in cytoolz.functoolz._memoize.__call__
    ???
distributed/utils.py:142: in _get_ip
    addr_info = socket.getaddrinfo(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

host = 'genevieve-G5-5500', port = 80, family =
type = , proto = 17, flags = 0

    def getaddrinfo(host, port, family=0, type=0, proto=0, flags=0):
        """Resolve host and port into list of address info entries.
   
        Translate the host/port argument into a sequence of 5-tuples that contain
        all the necessary arguments for creating a socket connected to that service.
        host is a domain name, a string representation of an IPv4/v6 address or
        None. port is a string service name such as 'http', a numeric port number or
        None. By passing None as the value of host and port, you can pass NULL to
        the underlying C API.
   
        The family, type and proto arguments can be optionally specified in order to
        narrow the list of addresses returned. Passing zero as a value for each of
        these arguments selects the full range of results.
        """
        # We override this function since we want to translate the numeric family
        # and socket type values to enum constants.
        addrlist = []
>      for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
E      socket.gaierror: [Errno -5] No address associated with hostname

../../../anaconda3/envs/dask-distributed/lib/python3.8/socket.py:918: gaierror
----------------------------- Captured stdout call -----------------------------

Print from stderr
  /
=================

distributed.scheduler - INFO - -----------------------------------------------
distributed.http.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy
/home/genevieve/GitHub/temp/distributed/distributed/node.py:160: UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 33299 instead
  warnings.warn(
distributed.scheduler - INFO - -----------------------------------------------
distributed.scheduler - INFO - Clear task state
distributed.scheduler - INFO -  Scheduler at:  tcp://192.168.1.102:8786
distributed.scheduler - INFO -  dashboard at:                    :33299
distributed.scheduler - INFO - Receive client connection: Client-c457fb8c-fe6f-11eb-b6c8-e19f018f439a
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Remove client Client-c457fb8c-fe6f-11eb-b6c8-e19f018f439a
distributed.scheduler - INFO - Remove client Client-c457fb8c-fe6f-11eb-b6c8-e19f018f439a
distributed.scheduler - INFO - Close client connection: Client-c457fb8c-fe6f-11eb-b6c8-e19f018f439a
distributed.scheduler - INFO - End scheduler at 'tcp://192.168.1.102:8786'
Traceback (most recent call last):
  File "/home/genevieve/anaconda3/envs/dask-distributed/bin/dask-scheduler", line 33, in
    sys.exit(load_entry_point('distributed', 'console_scripts', 'dask-scheduler')())
  File "/home/genevieve/GitHub/temp/distributed/distributed/cli/dask_scheduler.py", line 217, in go
    main()
  File "/home/genevieve/anaconda3/envs/dask-distributed/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/home/genevieve/anaconda3/envs/dask-distributed/lib/python3.8/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/home/genevieve/anaconda3/envs/dask-distributed/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/genevieve/anaconda3/envs/dask-distributed/lib/python3.8/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/home/genevieve/GitHub/temp/distributed/distributed/cli/dask_scheduler.py", line 208, in main
    loop.run_sync(run)
  File "/home/genevieve/anaconda3/envs/dask-distributed/lib/python3.8/site-packages/tornado/ioloop.py", line 529, in run_sync
    raise TimeoutError("Operation timed out after %s seconds" % timeout)
tornado.util.TimeoutError: Operation timed out after None seconds

Print from stdout
=================

________________________________ test_hostport _________________________________

host = '2001:4860:4860::8888', port = 80, family =

    @toolz.memoize
    def _get_ip(host, port, family):
        # By using a UDP socket, we don't actually try to connect but
        # simply select the local address through which *host* is reachable.
        sock = socket.socket(family, socket.SOCK_DGRAM)
        try:
>          sock.connect((host, port))
E          OSError: [Errno 101] Network is unreachable

distributed/utils.py:133: OSError

During handling of the above exception, another exception occurred:

loop =

    def test_hostport(loop):
        with popen(["dask-scheduler", "--no-dashboard", "--host", "127.0.0.1:8978"]):
   
            async def f():
                # The scheduler's main port can't be contacted from the outside
                await assert_can_connect_locally_4(8978, timeout=5.0)
   
            with Client("127.0.0.1:8978", loop=loop) as c:
                assert len(c.nthreads()) == 0
>              c.sync(f)

distributed/cli/tests/test_dask_scheduler.py:51:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
distributed/client.py:845: in sync
    return sync(
distributed/utils.py:325: in sync
    raise exc.with_traceback(tb)
distributed/utils.py:308: in f
    result[0] = yield future
../../../anaconda3/envs/dask-distributed/lib/python3.8/site-packages/tornado/gen.py:762: in run
    value = future.result()
distributed/cli/tests/test_dask_scheduler.py:47: in f
    await assert_can_connect_locally_4(8978, timeout=5.0)
distributed/utils_test.py:1226: in assert_can_connect_locally_4
    assert_cannot_connect("tcp://[%s]:%d" % (get_ipv6(), port), **kwargs),
distributed/utils.py:164: in get_ipv6
    return _get_ip(host, port, family=socket.AF_INET6)
cytoolz/functoolz.pyx:476: in cytoolz.functoolz._memoize.__call__
    ???
distributed/utils.py:142: in _get_ip
    addr_info = socket.getaddrinfo(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

host = 'genevieve-G5-5500', port = 80, family =
type = , proto = 17, flags = 0

    def getaddrinfo(host, port, family=0, type=0, proto=0, flags=0):
        """Resolve host and port into list of address info entries.
   
        Translate the host/port argument into a sequence of 5-tuples that contain
        all the necessary arguments for creating a socket connected to that service.
        host is a domain name, a string representation of an IPv4/v6 address or
        None. port is a string service name such as 'http', a numeric port number or
        None. By passing None as the value of host and port, you can pass NULL to
        the underlying C API.
   
        The family, type and proto arguments can be optionally specified in order to
        narrow the list of addresses returned. Passing zero as a value for each of
        these arguments selects the full range of results.
        """
        # We override this function since we want to translate the numeric family
        # and socket type values to enum constants.
        addrlist = []
>      for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
E      socket.gaierror: [Errno -5] No address associated with hostname

../../../anaconda3/envs/dask-distributed/lib/python3.8/socket.py:918: gaierror
----------------------------- Captured stdout call -----------------------------

Print from stderr
  /
=================

distributed.scheduler - INFO - -----------------------------------------------
distributed.http.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy
/home/genevieve/GitHub/temp/distributed/distributed/node.py:160: UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 45903 instead
  warnings.warn(
distributed.scheduler - INFO - -----------------------------------------------
distributed.scheduler - INFO - Clear task state
distributed.scheduler - INFO -  Scheduler at:      tcp://127.0.0.1:8978
distributed.scheduler - INFO -  dashboard at:          127.0.0.1:45903
distributed.scheduler - INFO - Receive client connection: Client-c5649c23-fe6f-11eb-b6c8-e19f018f439a
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Remove client Client-c5649c23-fe6f-11eb-b6c8-e19f018f439a
distributed.scheduler - INFO - Remove client Client-c5649c23-fe6f-11eb-b6c8-e19f018f439a
distributed.scheduler - INFO - Close client connection: Client-c5649c23-fe6f-11eb-b6c8-e19f018f439a
distributed.scheduler - INFO - End scheduler at 'tcp://127.0.0.1:8978'
Traceback (most recent call last):
  File "/home/genevieve/anaconda3/envs/dask-distributed/bin/dask-scheduler", line 33, in
    sys.exit(load_entry_point('distributed', 'console_scripts', 'dask-scheduler')())
  File "/home/genevieve/GitHub/temp/distributed/distributed/cli/dask_scheduler.py", line 217, in go
    main()
  File "/home/genevieve/anaconda3/envs/dask-distributed/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/home/genevieve/anaconda3/envs/dask-distributed/lib/python3.8/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/home/genevieve/anaconda3/envs/dask-distributed/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/genevieve/anaconda3/envs/dask-distributed/lib/python3.8/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/home/genevieve/GitHub/temp/distributed/distributed/cli/dask_scheduler.py", line 208, in main
    loop.run_sync(run)
  File "/home/genevieve/anaconda3/envs/dask-distributed/lib/python3.8/site-packages/tornado/ioloop.py", line 529, in run_sync
    raise TimeoutError("Operation timed out after %s seconds" % timeout)
tornado.util.TimeoutError: Operation timed out after None seconds

Print from stdout
=================

______________________________ test_no_dashboard _______________________________

loop =

    def test_no_dashboard(loop):
        with popen(["dask-scheduler", "--no-dashboard"]):
            with Client(f"127.0.0.1:{Scheduler.default_port}", loop=loop):
                response = requests.get("http://127.0.0.1:8787/status/")
>              assert response.status_code == 404
E              assert 200 == 404
E                +200
E                -404

distributed/cli/tests/test_dask_scheduler.py:58: AssertionError
----------------------------- Captured stdout call -----------------------------

Print from stderr
  /
=================

distributed.scheduler - INFO - -----------------------------------------------
distributed.http.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy
/home/genevieve/GitHub/temp/distributed/distributed/node.py:160: UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 43953 instead
  warnings.warn(
distributed.scheduler - INFO - -----------------------------------------------
distributed.scheduler - INFO - Clear task state
distributed.scheduler - INFO -  Scheduler at:  tcp://192.168.1.102:8786
distributed.scheduler - INFO -  dashboard at:                    :43953
distributed.scheduler - INFO - Receive client connection: Client-c67c7632-fe6f-11eb-b6c8-e19f018f439a
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Remove client Client-c67c7632-fe6f-11eb-b6c8-e19f018f439a
distributed.scheduler - INFO - Remove client Client-c67c7632-fe6f-11eb-b6c8-e19f018f439a
distributed.scheduler - INFO - Close client connection: Client-c67c7632-fe6f-11eb-b6c8-e19f018f439a
distributed.scheduler - INFO - End scheduler at 'tcp://192.168.1.102:8786'
Traceback (most recent call last):
  File "/home/genevieve/anaconda3/envs/dask-distributed/bin/dask-scheduler", line 33, in
    sys.exit(load_entry_point('distributed', 'console_scripts', 'dask-scheduler')())
  File "/home/genevieve/GitHub/temp/distributed/distributed/cli/dask_scheduler.py", line 217, in go
    main()
  File "/home/genevieve/anaconda3/envs/dask-distributed/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/home/genevieve/anaconda3/envs/dask-distributed/lib/python3.8/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/home/genevieve/anaconda3/envs/dask-distributed/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/genevieve/anaconda3/envs/dask-distributed/lib/python3.8/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/home/genevieve/GitHub/temp/distributed/distributed/cli/dask_scheduler.py", line 208, in main
    loop.run_sync(run)
  File "/home/genevieve/anaconda3/envs/dask-distributed/lib/python3.8/site-packages/tornado/ioloop.py", line 529, in run_sync
    raise TimeoutError("Operation timed out after %s seconds" % timeout)
tornado.util.TimeoutError: Operation timed out after None seconds

Print from stdout
=================

___________________________ test_dashboard_whitelist ___________________________

loop =

    @pytest.mark.skipif(not LINUX, reason="Need 127.0.0.2 to mean localhost")
    def test_dashboard_whitelist(loop):
        pytest.importorskip("bokeh")
        with pytest.raises(Exception):
>          requests.get("http://localhost:8787/status/").ok
E          Failed: DID NOT RAISE

distributed/cli/tests/test_dask_scheduler.py:123: Failed

```

**Environment**:

- Dask version: latest `2021.08.0`
- Python version: 3.8
- Operating System: Ubuntu 20.04
- Install method (conda, pip, source): conda environment creation, pip editable installation of distributed

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.