dask / dask/distributed

Extensions vs Plugins

Open
#6,011 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Currently we have two mechanisms to extend the scheduler and workers:

1. **Extensions:** primarily used by Dask developers to modularize the scheduler and worker systems.
2. **Plugins:** primarily used by Dask users to add new features to Dask from the user-side

### Plugins contract

Plugins have an explicit contract and methods like `transition`, `remove_worker`, and so on. They are well defined.

### Extension conventions

Extensions don't have such a contract but often do a few things by convention:

1. Define some new handlers and inject those into the server's handlers
2. Add themselves to server.extensions (although this is moved to a contract in #5957)

Well, that's pretty much it actually. In #5957 it's proposed to add a heartbeat mechanism .

Nothing here is set or required. Extensions are just classes that hang around. We don't really have any rules around them.

### Open questions

1. Should we try to merge these two so that there is only one extension mechanism?
2. If not, should we try to formalize Extensions a little more?

This is low priority (I think). I'm raising this mostly to capture a conversation that was in a PR but didn't get resolved.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.