JuliaParallel / JuliaParallel/Dagger.jl

Add Services/Actors API

Open
#224 0 comments 4 reactions 0 assignees View on GitHub
enhancement scheduler
Dominant language
Julia
Stars
723
Forks
90
Avg merge
1d 37m
Merged PRs (30d)
9

Description

Stateless tasks are nice from a scheduling point-of-view, since we can re-schedule and rearrange tasks in any valid order. However, stateful computation allows for memory reuse, data locality, gated resource access, and often is just a better model for a given computation or operation.

We can add a "Services" API which is focused around the creation, management, and deletion of persistent tasks executing on a fixed processor:

- A Dagger service, like a task, would execute in a task context.
- Services can be created by any task or service (and services can create tasks and services), and are accessed (for invoking or deleting) by a handle returned by the creation function. The handle will be serializable.
- A service can be created on a specific processor if needed, or otherwise left to the scheduler to decide.
- State would be stored within the function itself (which would be a callable struct), arguments would be passed as normal, and the result would be returned to the caller (which can be a task or service).
- The scheduler may choose to track the execution and resource usage of a service externally, and use that to control its execution priority, as well as control scheduling of other tasks/services onto the same processor.
- A service can be called from anywhere, so if the service wants to receive/send a reference to a non-serializable object, it must be wrapped in a Chunk.
- Some helpful services might be provided by Dagger, such as a service-name registry (allowing services to be associated with and located by a unique String).
- The service handle will be cached in the scheduler so that, in the event that a worker dies, its services will be automatically restarted and the cached handle will be reused.
- Advanced: Raw access to any service's function object will be available for any task scheduled on the same processor, to allow for building functionality on top of existing services.

Note: This is a first pass at specifying the requirements; I'm open to any changes to the above!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.