dask / dask/dask-jobqueue

Make job submission asynchronous

Open
#470 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
256
Forks
150
PR merge metrics
No merged PRs in 30d

Description

I have noticed that execution of commands (e.g., `condor_submit` for the condor backend) appear to be synchronous. In fact, there's a small note about this in the code itself:

https://github.com/dask/dask-jobqueue/blob/master/dask_jobqueue/core.py#L305

We've started to notice this particularly at very busy batch schedulers. For example, when dask labextension (https://github.com/dask/dask-labextension) is used in a Jupyter notebook, it will spawn the Dask scheduler inside the jupyter hub process (I think I got this terminology right?) and not the notebook. Because it's in the hub itself, if dask jobqueue is non-responsive then the entire UI freezes (as no I/O is done in the event loop). This triggers user complaints of "Jupyter stops working when we use Dask".

The impact of the blocking behavior can be easily seen by replacing the submit executable with a shell script that does a `sleep 20` before invoking the real submit executable.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.