dask / dask/distributed

Make SpecCluster declarative

Open
#4,606 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

`SpecCluster` and other components within distributed like `distributed.cli.dask_spec` aim to create clusters and workers according to a specification.

Currently these operations are performed imperatively. The `SpecCluster` class is given a spec and a resource is created to match the spec.

I would like to move things towards a declarative approach instead. This would mean the class is given a spec and it ensures that a resource exists to match that spec. If one doesn't exist it will create it, same as today. But if it already exists it will do nothing.

This slight difference of **ensuring** a resource instead of **creating** one will enable much more flexibility when it comes to recreating objects to represent existing resources. This ties into the lifecycle work I've been doing as part of #4263.

Opening this issue to signal my intent to work on this.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.