dask / dask/dask-jobqueue

Allow to start a Scheduler in a batch job

Open
#186 13 comments 0 reactions 0 assignees View on GitHub
all job schedulers
Dominant language
Python
Stars
256
Forks
150
PR merge metrics
No merged PRs in 30d

Description

One of the goal of `ClusterManager` object is to be able to launch a remote scheduler. In dask-jobqueue scope, this probably means submitting a job which will start a Scheduler, and then connect to it.

We probably still lacks some remote interface between `ClusterManager` and `scheduler` object for this to work, so it will probably mean to extend APIs upstream.

Identified Scheduler method to provide:

- retire_workers(n, minimum, key)
- scheduler_info(), already existing, see if sufficient,
- add_diagnostic_plugin(), and mostly retrive plugin information remotely

I suspect that adaptive will need to change significantly too, this will maybe lead to having a transitional adaptive logic in dask-jobqueue, and other remote function to add in scheduler.

This is in the scope of #170.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.