pingcap / pingcap/tidb

support deterministic failover schedule for placement rules

Open
#37,251 18 comments 3 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement

My deployment scenario involves two "primary" regions in AWS:
- us-east-1
- us-west-2

I have been experimenting with placement rules with a third region: `us-east-2`. This region should only be used for quorum, as there are no application servers hosted in it. So I define a placement policy as follows:

```sql
CREATE PLACEMENT POLICY `defaultpolicy` PRIMARY_REGION="us-east-1" REGIONS="us-east-1,us-west-2,us-east-2";
```

Because the pd-server supports a `weight` concept, when `us-east-1` fails I can have the pd-leader be deterministic and `us-west-2` will become the leader. However, there is no deterministic behavior of where the leader of regions for `defaultpolicy` will go. They will likely balance across `us-west-2` and `us-east-2`, which is not the desired behavior.

Ideally I want the priority for the leader to be in-order of the region-list. This means that `us-west-2` will become the new leader for all regions. Perhaps this could be conveyed with syntax like:

```sql
CREATE PLACEMENT POLICY `defaultpolicy` PRIMARY_REGION="us-east-1" REGIONS="us-east-1,us-west-2,us-east-2" SCHEDULE="DETERMINISTIC";
```

In fact, if this worked deterministically for leader-scheduling and follower-scheduling, and extension of this is I could create the following:

```sql
CREATE PLACEMENT POLICY `defaultpolicy` PRIMARY_REGION="us-east-1" REGIONS="us-east-1,us-west-2,us-east-2,us-west-1" SCHEDULE="DETERMINISTIC";
```

Since the default followers is 2, it would mean that `us-west-1` won't get regions scheduled _unless_ one of the other regions fails, which suits me perfectly. It will also mean that commit latency is only initially bad when failover to `us-west-2` first occurs. Over time as regions as migrated to `us-west-1`, the performance should be ~restored as quorum can be achieved on the west coast.

This is a really common deployment pattern in the continental USA, so I'm hoping it can be implemented :-)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.