TimelyDataflow / TimelyDataflow/timely-dataflow
Consider worker virtualization layer
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 293
- Avg merge
- 14h 46m
- Merged PRs (30d)
- 4
Description
Timely has workers exactly mapped to threads, but we could have a virtualization layer so that threads can host multiple (or fewer) logical workers.
There are a few reasons to do this, none of which are especially new:
- By over-sharding, migration and parallel recovery become easier.
- By always using powers of two workers, we can always use masks in data exchange.
There may be some other reasons, who knows, but at least we should talk through what is required for this to happen.
@utaal @antiguru
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no files, tests, or entry points. Start by clarifying the requirements for virtualizing logical workers over threads, including over-sharding, migration, parallel recovery, and power-of-two worker counts; done requires an agreed design and implementation scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100