microsoft / microsoft/durabletask-netherite
Implement smarter partition balancer
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 239
- Forks
- 37
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 2
Description
Currently we use EventHubsProcessor (which is part of the EH client library) to balance partitions across nodes. However, this mechanism is unaware of load inside the partitions, and simply balances the number of partitions. For example, on three nodes and 12 partitions, each node will have four random partitions. This is an issue; for example, if only three partitions are busy, we can be unlucky and all three busy partitions are placed on the same node, wasting the other two nodes.
At some point we should perhaps implement a better partition balancer. This is a non-issue for planned K8s implementations since it does not use EventHubs at all.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading how EventHubsProcessor currently balances partitions across nodes and how partition load could be observed. Define the balancing behavior for uneven workloads, then verify that busy partitions are distributed rather than concentrated on one node; the issue provides no file or test entry point.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100