Assuming `127.0.0.1` or `127.0.1.1` on OSError is incorrect, better throw the error.
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
Looking through the whole codebase, this is the only place (not in tests) that a host of `127.0.0.1` is assigned:
https://github.com/dask/distributed/blob/3551d1574c9cd72d60197cc84dd75702ebcfec54/distributed/comm/inproc.py#L43
Under certain circumstances, this OSError will throw and the actual IP address of the host won't be available.
```python
➜ docker exec -it dask-worker bash
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
root@lympha:/opt/app# python
Python 3.10.6 (main, Aug 3 2022, 10:24:07) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> family=socket.AF_INET
>>> sock = socket.socket(family, socket.SOCK_DGRAM)
>>> host="8.8.8.8"
>>> port=80
>>> sock.connect((host, port))
>>> ip = sock.getsockname()[0]
>>> ip
'192.168.1.128'
>>> addr_info = socket.getaddrinfo(
... socket.gethostname(), port, family, socket.SOCK_DGRAM, socket.IPPROTO_UDP
... )[0]
>>> ip_other_path = addr_info[4][0]
>>> ip_other_path
'127.0.1.1'
>>> sock.close()
>>>
root@lympha:/opt/app# exit
```
The root of this problem is here: https://github.com/dask/distributed/blob/3551d1574c9cd72d60197cc84dd75702ebcfec54/distributed/utils.py#L170.
This manifests itself in my setup, where I have a scheduler on one machine (`192.168.1.173`), a worker on the same machine (`192.168.1.173`), and a worker on another machine (`192.168.1.128`). The scheduler and both workers are run as Docker containers. In all cases, the `network_mode` of these containers is set to `host`. Here is a stack trace from the scheduler:
```bash
2022-08-07 01:48:03,181 - distributed.batched - INFO - Batched Comm Closed Client local=tcp://127.0.1.1:8786 remote=tcp://127.0.0.1:37186>
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 315, in write
raise StreamClosedError()
tornado.iostream.StreamClosedError: Stream is closed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/distributed/batched.py", line 97, in _background_send
nbytes = yield self.comm.write(
File "/usr/local/lib/python3.10/site-packages/tornado/gen.py", line 762, in run
value = future.result()
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 326, in write
convert_stream_closed_error(self, e)
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 144, in convert_stream_closed_error
raise CommClosedError(f"in {obj}: {exc}") from exc
distributed.comm.core.CommClosedError: in Client local=tcp://127.0.1.1:8786 remote=tcp://127.0.0.1:37186>: Stream is closed
2022-08-07 01:48:03,386 - distributed.scheduler - INFO - Close client connection: Client-df1885a7-15f1-11ed-8238-a85e45e84cc4
```
This should be extremely rare. Quite frankly, I'm a bit surprised as to why the OSError path is hit at all. So, I am investigating why this is the case (or at least why I suspect this is the case).
Here's a snippet from my `docker-compose.yml`
```yaml
dask-scheduler:
image: private/dask
container_name: dask-scheduler
network_mode: host
command: ["dask-scheduler"]
labels:
org.label-schema.group: "dask"
restart: always
dask-worker:
image: private/dask
container_name: dask-worker
network_mode: host
# IP_ADDR is 192.168.1.173
command: ["dask-worker", "tcp://${IP_ADDR}:8786", "--host", "${IP_ADDR}"]
labels:
org.label-schema.group: "dask"
restart: always
```
Script used to start the dask-worker on `192.168.1.128`:
```bash
IP_ADDR=$(hostname -I | cut -f1 -d' ')
export IP_ADDR
# IP_ADDR is 192.168.1.128
docker pull private/dask
docker image prune -f
docker run -d --name dask-worker --network="host" --rm private/dask dask-worker tcp://192.168.1.173:8786 --host ${IP_ADDR}
```
Contributor guide
Assessment
This issue has not been assessed yet.