Ensure cluster manifest is complete before transitioning from `Joining` to `Active`
- Dominant language
- C#
- Stars
- 10.9k
- Forks
- 2.1k
- Avg merge
- 13h 56m
- Merged PRs (30d)
- 351
Description
In heterogenous clusters, where different silos have different sets of grains, etc, it's currently frequently the case that one silo will join an existing cluster, and it will start activating grains without yet having a complete view of the capabilities of the other silos in the cluster. This will cause the silo to place all grains locally and to fail placement of grains which are only supported by other hosts in the cluster.
If we wait for synchronization to complete during startup, before transitioning to `Active`, then we can ensure that if there are active hosts in the cluster which support a given grain type, then placement will consider those silos.
This will not solve the issue where (in a heterogeous cluster) one silo starts and is the first to join a cluster, but it tries to call a grain which is only supported by not-yet-available silos.
A similar issue is that we should synchronize deployment load statistics from other silos before transitioning to `Active`.
Contributor guide
Research direction
Start by tracing silo startup through the transition from `Joining` to `Active`, then inspect how cluster manifest and deployment load statistics synchronization are performed. Done means activation waits for the relevant synchronization so grain placement sees active hosts' capabilities and load data, while preserving the first-silo limitation described in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100