dask / dask/distributed

[Feature] Add worker capabilities/labels

Open
#2,616 7 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

It would be really nice if you could add something like capabilities/labels (however you want to call them) to `dask`

It would be something like:
```
dask-worker scheduler:8786 --capabilities "vaapi,nodejs"
dask-worker scheduler:8786
```

Where you could schedule a task like:
```python
add_t = client.submit(add, 1, 4)
video_t = client.submit(process, "/path/to/file", capabilities=["vaapi"])
```

So that certain tasks only get scheduled on certain workers, which support that kind of task

Additionally another feature to consider would be enum-like capabilities:
```
dask-worker scheduler:8786 --capabilities "some_feature,arch=amd64"
dask-worker scheduler:8786 --capabilities "arch=armv8"
```

With a python equivalent of:
```python
result_t = client.submit(process, some_data, capabilities={"some_feature": True, "arch": "amd64"})
other_result_t = client.submit(process, other_data, capabilities=["some_feature"])
```

The current _equivalent_ of capabilities would be something like:
```
dask-worker scheduler:8786 --resources "vaapi=9999"
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.