Seamless deployment with zero-down-time on kubernetes
- Dominant language
- C#
- Stars
- 10.9k
- Forks
- 2.1k
- Avg merge
- 14h 42m
- Merged PRs (30d)
- 354
Description
I run multiple pods of my service named XApp, each of them will run a silo and grain activations can happen on any of those silos based on the number of active grains in order to distribute the load among silos, Upon application shutdown(CurrentDomain.ProcessExit) I call silo shutdown using this:
```
var silo = _host.Services.GetRequiredService();
silo.Shutdown();
```
Upon deployment of any new version of XApp which might include any changes in my app logic or even in grain methods I encounter an issue, Deployment uses the rollingUpdate strategy, one new pod prepares to get in the ready state which will run a new silo for itself and in this between it might receive a message to deliver it to a silo! once the new pod is in ready state an old pod will get terminated, and alongside the bound silo will be terminated, this process continues on till all-new version pods get stable and all old versions get terminated naturally.
Since the grain could be already active on any other silo(which probably is about to be killed) it won't get activated on my new pod's silo. All messages to grains could get enqueued as well and termination will make those to not get processed, This causes downtime and also messages lost upon deployments which is not safe, Also I get the known error "Response did not arrive on time"
BTW I use ClusterClient or GrainFactory only inside the same pods and no other Apps can make direct grain calls/Stream subscriptions to those silos.
What can be the best practice to have a seamless deployment without missing any message in grains?
Or better to say, I want my all grains to get activated on new pods(including pending messages).
Contributor guide
Research direction
Start with the CurrentDomain.ProcessExit shutdown handler, Silo.Shutdown(), and the Kubernetes rollingUpdate behavior described in the issue; review how ClusterClient and GrainFactory are used within each pod. Done would require an agreed, tested deployment approach that avoids lost or unprocessed grain messages during silo termination.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, kubernetes
- Domain
- devops, distributed-systems, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100