fly-apps / fly-apps/postgres-ha

PRIMARY_REGION not considered on cluster initialization

Open
#11 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
332
Forks
123
PR merge metrics
No merged PRs in 30d

Description

Stolon has a keeper flag named `can-be-master` that was introduced by the following PR:
https://github.com/sorintlab/stolon/pull/697

We use this flag in combination with the `PRIMARY_REGION` env var to restrict primary eligibility to a specific region.

**Problem**
The problem is that Stolon's Sentinel does not consider this flag when assigning the primary on cluster initialization. During the cluster initialization process, the primary keeper is chosen at random.

More information can be found here: https://github.com/sorintlab/stolon/issues/840

**Workaround**
The recommended approach as of right now, is to provision your cluster within the primary region BEFORE spinning up cross-region read-replicas. If you spin everything up at once, you will be at risk of an out-of-region member being assigned primary.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at Stolon’s Sentinel cluster-initialization path and trace how the primary keeper is selected, then inspect how the keeper’s can-be-master flag and PRIMARY_REGION affect eligibility. Reproduce initialization with cross-region keepers; done means an out-of-region or ineligible keeper cannot be selected as primary, while an eligible one can.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, postgres
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.