dask / dask/dask-jobqueue

Instantiate a JobQueueCluster with an already started Scheduler

Open
#419 6 comments 3 reactions 0 assignees View on GitHub
all job schedulers
Dominant language
Python
Stars
256
Forks
150
PR merge metrics
No merged PRs in 30d

Description

While looking in several issues here, I feel that adding the ability to instanciate a JobQueueCluster with an already started Scheduler could help some use cases, and be a first step towards starting the Scheduler remotly on a job (#186).

The idea would be to add the possibility to do:
```python
cluster = FooCluster(scheduler=my_scheduler)
#or
cluster = FooCluster(scheduler_info=scheduler_file)
```
Then the JobQueueCluster won't instantiate a Scheduler, but just connect to an existing one.

I think this woul provide solutions to some issues:
* #407, we could reuse a LocalCluster Scheduler :
```python
loc_cluster = LocalCluster(...)
job_cluster = FooCluster(scheduler=loc_cluster.scheduler)
```
* #378, #381 :
```python
lowmem_cluster = FooCluster(cores=2,memory='4GB',resources=...)
highmem_cluster = FooCluster(scheduler=lowmeme_cluster.scheduler,cores=2,memory='16GB',resources=...)
```

Any thought?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.