dotnet / dotnet/orleans

Ensure cluster manifest is complete before transitioning from `Joining` to `Active`

Open
#9,772 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
10.9k
Forks
2.1k
Avg merge
13h 56m
Merged PRs (30d)
351

Description

In heterogenous clusters, where different silos have different sets of grains, etc, it's currently frequently the case that one silo will join an existing cluster, and it will start activating grains without yet having a complete view of the capabilities of the other silos in the cluster. This will cause the silo to place all grains locally and to fail placement of grains which are only supported by other hosts in the cluster.

If we wait for synchronization to complete during startup, before transitioning to `Active`, then we can ensure that if there are active hosts in the cluster which support a given grain type, then placement will consider those silos.

This will not solve the issue where (in a heterogeous cluster) one silo starts and is the first to join a cluster, but it tries to call a grain which is only supported by not-yet-available silos.

A similar issue is that we should synchronize deployment load statistics from other silos before transitioning to `Active`.

Contributor guide

Open the contributing guide

Research direction

Start by tracing silo startup through the transition from `Joining` to `Active`, then inspect how cluster manifest and deployment load statistics synchronization are performed. Done means activation waits for the relevant synchronization so grain placement sees active hosts' capabilities and load data, while preserving the first-silo limitation described in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.