Request for steps to upgrade Pulsar from Single AZ to Multi-AZ deployment on AWS
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
**Describe the bug**
We are looking for steps to upgrade from Single AZ to Multi-AZ deployment on AWS. The options considered and the challenges are listed below
**Option 1: Deploy a new statefulset which spans across different AZ and decommission Bookie one by one from single AZ statefulset.**
**Assumption** : Autorecovery replicates data so that when Bookies are decommissioned one by one there is no data loss.
**Questions :**
1. Number of EKS worker nodes in the cluster is 3. Considering anti affinity rule is set in Bookie ( which means only one Bookie can run on one node), we assume that the cluster needs to scales out to include 3 more nodes.
2. How to ensure the worker nodes on which the older Bookie statefulset runs is not moved to a different zone. Wouldn’t the autoscaling groups created on EKS distribute nodes equally across zones ? If the number of worker nodes in the auto scale group is set as 3 * total AZ (for example say in 3 AZ region set the count as 9) then probably the initial nodes of the cluster may stay in the same AZ.
3. How to ensure the replication is complete before shutting down the second/third bookie?
**Option 2: Migrate from single AZ to Multi AZ with downtime using EBS snapshots**
Can Pulsar deployment/statefulset be scaled down and EBS snapshotting be used to backup and restore data to EBS volumes in different AZ ?
Contributor guide
Research direction
The issue names Pulsar Bookie StatefulSets, autorecovery, EKS worker nodes, AZ placement, and EBS snapshots, but no repository files or tests. Start by reviewing those deployment and storage entry points, then verify the migration assumptions for both proposed options. Done means documenting validated single-AZ to multi-AZ steps that address replication, node placement, downtime, and snapshot constraints.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, distributed-systems, infrastructure
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100