TimelyDataflow / TimelyDataflow/timely-dataflow
Dynamically adding/removing workers
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 293
- Avg merge
- 14h 46m
- Merged PRs (30d)
- 4
Description
Hi, folks!
I apologize if this has been asked before, but I searched through the issues and only found two tangential ones (https://github.com/TimelyDataflow/timely-dataflow/issues/182 and https://github.com/TimelyDataflow/timely-dataflow/issues/319).
I'm curious about the possibility of adding and removing workers from a data flow cluster. More specifically, thinking about leveraging cloud where we might lose workers because nodes are replaced or we can add more to scale up. I've checked the documentation and it seems like the list of workers is static.
Is there any documentation about this? How would we manage to modify the cluster configuration?
Thanks!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the existing worker and cluster configuration documentation, then read the related issues 182 and 319. Determine whether adding or removing workers is supported and what configuration changes would be required for cloud node replacement or scaling. Done should clearly document the supported behavior and limitations, or define the required design if it is not supported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100