apache / apache/pulsar

Request for steps to upgrade Pulsar from Single AZ to Multi-AZ deployment on AWS

Open
#11,717 1 comment 0 reactions 0 assignees View on GitHub
doc-required lifecycle/stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**Describe the bug**

We are looking for steps to upgrade from Single AZ to Multi-AZ deployment on AWS. The options considered and the challenges are listed below

**Option 1: Deploy a new statefulset which spans across different AZ and decommission Bookie one by one from single AZ statefulset.**

**Assumption** : Autorecovery replicates data so that when Bookies are decommissioned one by one there is no data loss.

**Questions :**
1. Number of EKS worker nodes in the cluster is 3. Considering anti affinity rule is set in Bookie ( which means only one Bookie can run on one node), we assume that the cluster needs to scales out to include 3 more nodes.
2. How to ensure the worker nodes on which the older Bookie statefulset runs is not moved to a different zone. Wouldn’t the autoscaling groups created on EKS distribute nodes equally across zones ? If the number of worker nodes in the auto scale group is set as 3 * total AZ (for example say in 3 AZ region set the count as 9) then probably the initial nodes of the cluster may stay in the same AZ.
3. How to ensure the replication is complete before shutting down the second/third bookie?

**Option 2: Migrate from single AZ to Multi AZ with downtime using EBS snapshots**

Can Pulsar deployment/statefulset be scaled down and EBS snapshotting be used to backup and restore data to EBS volumes in different AZ ?

Contributor guide

Open the contributing guide

Research direction

The issue names Pulsar Bookie StatefulSets, autorecovery, EKS worker nodes, AZ placement, and EBS snapshots, but no repository files or tests. Start by reviewing those deployment and storage entry points, then verify the migration assumptions for both proposed options. Done means documenting validated single-AZ to multi-AZ steps that address replication, node placement, downtime, and snapshot constraints.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, distributed-systems, infrastructure
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.