citusdata / citusdata/citus

Add ability to upgrade workers nodes through logical replication

Open
#1,709 0 comments 0 reactions 0 assignees View on GitHub
feature
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

With logical replication becoming generally available, we could use it to upgrade worker nodes without downtime, this includes both Citus upgrades and postgres upgrades.

For example, the following command could create a publication for all shards on the primary of group 1 and corresponding subscription on `pg11node`.
```
SELECT add_logical_replica_node('pg11node', 5432, groupid := 1);
```

Once logical replication is reasonably caught up, another function could call the logic to block writes, wait for the logical replica to fully catch up, and make the node the new primary in the group, e.g.:
```
SELECT replace_primary_node('pg11node', 5432, groupid := 1);
```

We would probably also want to implement #1026, such that have a way to ensure that the new worker node has all the relevant functions, types, etc.

Logical replication does have various caveats compared to physical replication. We need to implement logic to handle DDL commands, shard moves, 2PCs, and other Citus features during upgrades and may have to block some of them.

Using a logical replica for failover is probably not feasible at this time. For instance, prepared transactions do not get written to the replica, which means the replica may not apply certain commands that other participants in the 2PC applied even if synchronous replication is enabled, violating atomicity. There is also no rewind mechanism for bringing back a failed primary after failing over to a logical replica.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.