Allow to start a Scheduler in a batch job
- Dominant language
- Python
- Stars
- 256
- Forks
- 150
- PR merge metrics
- No merged PRs in 30d
Description
One of the goal of `ClusterManager` object is to be able to launch a remote scheduler. In dask-jobqueue scope, this probably means submitting a job which will start a Scheduler, and then connect to it.
We probably still lacks some remote interface between `ClusterManager` and `scheduler` object for this to work, so it will probably mean to extend APIs upstream.
Identified Scheduler method to provide:
- retire_workers(n, minimum, key)
- scheduler_info(), already existing, see if sufficient,
- add_diagnostic_plugin(), and mostly retrive plugin information remotely
I suspect that adaptive will need to change significantly too, this will maybe lead to having a transitional adaptive logic in dask-jobqueue, and other remote function to add in scheduler.
This is in the scope of #170.
Contributor guide
Assessment
This issue has not been assessed yet.