JuliaParallel / JuliaParallel/Dagger.jl

Remove scheduler plugin machinery, make Sch programmable

Open
#166 1 comment 0 reactions 0 assignees View on GitHub
help wanted processors scheduler
Dominant language
Julia
Stars
723
Forks
90
Avg merge
1d 37m
Merged PRs (30d)
9

Description

The scheduler plugin system is (to my knowledge) unused by any modern users of Dagger. However, the presence of the potential for multiple external schedulers makes changing the scheduler API technically a breaking change, which would slow Dagger's development were that actually a concern. Of course, we still want to support different kinds of scheduling algorithms and optimizations that best suit a user's use case and DAG structure, so we should make the Sch scheduler user-programmable by making the current worker pressure algorithm optional, and having `schedule!` call a user-defined callable.

The option to change the internal scheduler algorithm is intended to only be used by adventurous users who understand that deadlocks/livelocks/hangs/etc. are all possible when changing the default scheduler, although we should expose "safe" semi-internal APIs that can perform common tasks correctly (which the default scheduler should also use whenever possible). Hopefully this change can spur developers and adventurous users to experiment with (and contribute) new scheduling algorithms which are better than the default scheduler for certain classes of workloads, with Dagger itself becoming the foundation for user-defined scheduling of distributed Julia code.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.