dask / dask/distributed

Scheduler and Client thow error on connect

Open
#3,412 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

I am trying to connect to a cluster and I didn't get a chance to change any configurations yet so I think everything is set to default. To turn on the scheduler I use the CLI interface without any arguments `dask-scheduler` and to create client I use Jupyter Notebook that just runs:
```
import distributed
client = distributed.Client("scheduler.local:8786")
```

I'm pretty confident that error I am seeing has nothing to do with URL to scheduler that I provided because my client manages to crash the scheduler :laughing: .
In both of my containers I specify the version of dask to use 2.9.2 and they are built at the same time so I am also pretty sure that this issue has little to do with client and scheduler having different versions.

One thing worth to mention is that I have both client and scheduler running in two docker containers, and those container run fine when using `docker-compose` problem starts when I run these container inside Amazon's ECS.

At this point all I can think of is that tornado need to have set timeout setting?
Blank assertion error is not really helping pinpoint the problem.
Looking at the trace looks like no time out has been provided as default.

Client error:
```
AssertionError Traceback (most recent call last)
in
----> 2 client = distributed.Client("scheduler.local:8786")

/usr/local/lib/python3.8/site-packages/distributed/client.py in __init__(self, address, loop, timeout, set_as_default, scheduler_file, security, asynchronous, name, heartbeat_interval, serializers, deserializers, extensions, direct_to_workers, **kwargs)
726 ext(self)
727
--> 728 self.start(timeout=timeout)
729 Client._instances.add(self)
730

/usr/local/lib/python3.8/site-packages/distributed/client.py in start(self, **kwargs)
891 self._started = asyncio.ensure_future(self._start(**kwargs))
892 else:
--> 893 sync(self.loop, self._start, **kwargs)
894
895 def __await__(self):

/usr/local/lib/python3.8/site-packages/distributed/utils.py in sync(loop, func, callback_timeout, *args, **kwargs)
333 if error[0]:
334 typ, exc, tb = error[0]
--> 335 raise exc.with_traceback(tb)
336 else:
337 return result[0]

/usr/local/lib/python3.8/site-packages/distributed/utils.py in f()
317 if callback_timeout is not None:
318 future = gen.with_timeout(timedelta(seconds=callback_timeout), future)
--> 319 result[0] = yield future
320 except Exception as exc:
321 error[0] = sys.exc_info()

/usr/local/lib/python3.8/site-packages/tornado/gen.py in run(self)
733
734 try:
--> 735 value = future.result()
736 except Exception:
737 exc_info = sys.exc_info()

/usr/local/lib/python3.8/site-packages/distributed/client.py in _start(self, timeout, **kwargs)
984
985 try:
--> 986 await self._ensure_connected(timeout=timeout)
987 except OSError:
988 await self._close()

/usr/local/lib/python3.8/site-packages/distributed/client.py in _ensure_connected(self, timeout)
1065 msg = await comm.read()
1066 assert len(msg) == 1
-> 1067 assert msg[0]["op"] == "stream-start"
1068
1069 bcomm = BatchedSend(interval="10ms", loop=self.loop)

AssertionError:
```
Scheduler error:
```
Traceback (most recent call last):
File "/usr/local/bin/dask-scheduler", line 8, in
sys.exit(go())
File "/usr/local/lib/python3.8/site-packages/distributed/cli/dask_scheduler.py", line 248, in go
main()
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/distributed/cli/dask_scheduler.py", line 237, in main
loop.run_sync(run)
File "/usr/local/lib/python3.8/site-packages/tornado/ioloop.py", line 531, in run_sync
raise TimeoutError("Operation timed out after %s seconds" % timeout)
tornado.util.TimeoutError: Operation timed out after None seconds
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.