dotnet / dotnet/orleans

Seamless deployment with zero-down-time on kubernetes

Open
#8,738 3 comments 0 reactions 0 assignees View on GitHub
Needs: triage :mag:
Dominant language
C#
Stars
10.9k
Forks
2.1k
Avg merge
14h 42m
Merged PRs (30d)
354

Description

I run multiple pods of my service named XApp, each of them will run a silo and grain activations can happen on any of those silos based on the number of active grains in order to distribute the load among silos, Upon application shutdown(CurrentDomain.ProcessExit) I call silo shutdown using this:
```
var silo = _host.Services.GetRequiredService();
silo.Shutdown();
```
Upon deployment of any new version of XApp which might include any changes in my app logic or even in grain methods I encounter an issue, Deployment uses the rollingUpdate strategy, one new pod prepares to get in the ready state which will run a new silo for itself and in this between it might receive a message to deliver it to a silo! once the new pod is in ready state an old pod will get terminated, and alongside the bound silo will be terminated, this process continues on till all-new version pods get stable and all old versions get terminated naturally.
Since the grain could be already active on any other silo(which probably is about to be killed) it won't get activated on my new pod's silo. All messages to grains could get enqueued as well and termination will make those to not get processed, This causes downtime and also messages lost upon deployments which is not safe, Also I get the known error "Response did not arrive on time"

BTW I use ClusterClient or GrainFactory only inside the same pods and no other Apps can make direct grain calls/Stream subscriptions to those silos.
What can be the best practice to have a seamless deployment without missing any message in grains?
Or better to say, I want my all grains to get activated on new pods(including pending messages).

Contributor guide

Open the contributing guide

Research direction

Start with the CurrentDomain.ProcessExit shutdown handler, Silo.Shutdown(), and the Kubernetes rollingUpdate behavior described in the issue; review how ClusterClient and GrainFactory are used within each pod. Done would require an agreed, tested deployment approach that avoids lost or unprocessed grain messages during silo termination.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, kubernetes
Domain
devops, distributed-systems, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.