dask / dask/distributed

Jupyter server won't start from SpecCluster

Open
#6,886 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

**What happened**:
Testing out the new `jupyter` flag on distributed 2022.8, I ran into the following error:

**Minimal Complete Verifiable Example**:

```python
from distributed import SpecCluster, Scheduler
SpecCluster(
scheduler={"cls": Scheduler, "options": {'protocol': 'tcp://', 'interface': None, 'host': '0.0.0.0', 'port': 0, 'dashboard_address': ':8787', 'jupyter': True}}
)
```
Output:
```
[I 2022-08-15 15:22:14.752 ServerApp] jupyterlab | extension was successfully linked.
[I 2022-08-15 15:22:15.019 ServerApp] notebook_shim | extension was successfully linked.
[W 2022-08-15 15:22:15.036 ServerApp] All authentication is disabled. Anyone who can connect to this server will be able to run code.
2022-08-15 15:22:15,038 - distributed.deploy.spec - WARNING - Cluster closed without starting up
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/distributed/deploy/spec.py", line 308, in _start
self.scheduler = cls(**self.scheduler_spec.get("options", {}))
File "/usr/local/lib/python3.10/site-packages/distributed/scheduler.py", line 3034, in __init__
j.initialize(
File "/usr/local/lib/python3.10/site-packages/traitlets/config/application.py", line 110, in inner
return method(app, *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/jupyter_server/serverapp.py", line 2466, in initialize
self.init_signal()
File "/usr/local/lib/python3.10/site-packages/jupyter_server/serverapp.py", line 2103, in init_signal
signal.signal(signal.SIGINT, self._handle_sigint)
File "/usr/local/lib/python3.10/signal.py", line 56, in signal
handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
ValueError: signal only works in main thread of the main interpreter
```

**Anything else we need to know?**:
I am guessing this is because `SpecCluster` is calling `Scheduler()` from an async block?

**Environment**:

- Dask version: 2022.8
- Python version: 3.10
- jupyterlab version: 3.4.3
- jupyter-serverversion: 1.18.1

Contributor guide

Open the contributing guide

Research direction

Reproduce the copy-pastable example with distributed 2022.8, Python 3.10, and Jupyter Server 1.18.1. Read distributed/deploy/spec.py around _start and distributed/scheduler.py where Scheduler initializes Jupyter, then trace the signal setup shown in the traceback. Done means SpecCluster starts successfully with jupyter=True without the main-thread signal error, with regression coverage for this example.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter, python
Domain
distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.