Make job submission asynchronous
- Dominant language
- Python
- Stars
- 256
- Forks
- 150
- PR merge metrics
- No merged PRs in 30d
Description
I have noticed that execution of commands (e.g., `condor_submit` for the condor backend) appear to be synchronous. In fact, there's a small note about this in the code itself:
https://github.com/dask/dask-jobqueue/blob/master/dask_jobqueue/core.py#L305
We've started to notice this particularly at very busy batch schedulers. For example, when dask labextension (https://github.com/dask/dask-labextension) is used in a Jupyter notebook, it will spawn the Dask scheduler inside the jupyter hub process (I think I got this terminology right?) and not the notebook. Because it's in the hub itself, if dask jobqueue is non-responsive then the entire UI freezes (as no I/O is done in the event loop). This triggers user complaints of "Jupyter stops working when we use Dask".
The impact of the blocking behavior can be easily seen by replacing the submit executable with a shell script that does a `sleep 20` before invoking the real submit executable.
Contributor guide
Assessment
This issue has not been assessed yet.