Add ability to upgrade workers nodes through logical replication
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
With logical replication becoming generally available, we could use it to upgrade worker nodes without downtime, this includes both Citus upgrades and postgres upgrades.
For example, the following command could create a publication for all shards on the primary of group 1 and corresponding subscription on `pg11node`.
```
SELECT add_logical_replica_node('pg11node', 5432, groupid := 1);
```
Once logical replication is reasonably caught up, another function could call the logic to block writes, wait for the logical replica to fully catch up, and make the node the new primary in the group, e.g.:
```
SELECT replace_primary_node('pg11node', 5432, groupid := 1);
```
We would probably also want to implement #1026, such that have a way to ensure that the new worker node has all the relevant functions, types, etc.
Logical replication does have various caveats compared to physical replication. We need to implement logic to handle DDL commands, shard moves, 2PCs, and other Citus features during upgrades and may have to block some of them.
Using a logical replica for failover is probably not feasible at this time. For instance, prepared transactions do not get written to the replica, which means the replica may not apply certain commands that other participants in the 2PC applied even if synchronous replication is enabled, violating atomicity. There is also no rewind mechanism for bringing back a failed primary after failing over to a logical replica.
Contributor guide
Assessment
This issue has not been assessed yet.