Jupyter server won't start from SpecCluster
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
**What happened**:
Testing out the new `jupyter` flag on distributed 2022.8, I ran into the following error:
**Minimal Complete Verifiable Example**:
```python
from distributed import SpecCluster, Scheduler
SpecCluster(
scheduler={"cls": Scheduler, "options": {'protocol': 'tcp://', 'interface': None, 'host': '0.0.0.0', 'port': 0, 'dashboard_address': ':8787', 'jupyter': True}}
)
```
Output:
```
[I 2022-08-15 15:22:14.752 ServerApp] jupyterlab | extension was successfully linked.
[I 2022-08-15 15:22:15.019 ServerApp] notebook_shim | extension was successfully linked.
[W 2022-08-15 15:22:15.036 ServerApp] All authentication is disabled. Anyone who can connect to this server will be able to run code.
2022-08-15 15:22:15,038 - distributed.deploy.spec - WARNING - Cluster closed without starting up
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/distributed/deploy/spec.py", line 308, in _start
self.scheduler = cls(**self.scheduler_spec.get("options", {}))
File "/usr/local/lib/python3.10/site-packages/distributed/scheduler.py", line 3034, in __init__
j.initialize(
File "/usr/local/lib/python3.10/site-packages/traitlets/config/application.py", line 110, in inner
return method(app, *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/jupyter_server/serverapp.py", line 2466, in initialize
self.init_signal()
File "/usr/local/lib/python3.10/site-packages/jupyter_server/serverapp.py", line 2103, in init_signal
signal.signal(signal.SIGINT, self._handle_sigint)
File "/usr/local/lib/python3.10/signal.py", line 56, in signal
handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
ValueError: signal only works in main thread of the main interpreter
```
**Anything else we need to know?**:
I am guessing this is because `SpecCluster` is calling `Scheduler()` from an async block?
**Environment**:
- Dask version: 2022.8
- Python version: 3.10
- jupyterlab version: 3.4.3
- jupyter-serverversion: 1.18.1
Contributor guide
Research direction
Reproduce the copy-pastable example with distributed 2022.8, Python 3.10, and Jupyter Server 1.18.1. Read distributed/deploy/spec.py around _start and distributed/scheduler.py where Scheduler initializes Jupyter, then trace the signal setup shown in the traceback. Done means SpecCluster starts successfully with jupyter=True without the main-thread signal error, with regression coverage for this example.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter, python
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100