citusdata / citusdata/citus

Sync pg_dist_node on node activation

Open
#2,570 1 comment 0 reactions 0 assignees View on GitHub
feature mx technical debt
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

### Problems with configuring node metadata

We currently only sync `pg_dist_node` to workers in MX, but populating `pg_dist_node` is necessary to use `pg_dist_poolinfo` and `pg_dist_authinfo` for inter-node communication. Right now, the infrastructure management layer would need to populate `pg_dist_node` manually on non-MX workers to properly set up poolers and authentication for inter-worker communication (e.g. for rebalancing, repartitioning, copying reference tables). However, we should make sure that node IDs and group IDs in `pg_dist_node` are consistent across nodes.

In regular Citus clusters, nodes are only ever addressed by their hostname and port, which means that it is not a strict requirement that node IDs are consistent across nodes. However, in MX we overwrite `pg_dist_node` on the worker in `start_metadata_sync_to_node` with the node IDs in the coordinator. That means that the node IDs in `pg_dist_poolinfo` and `pg_dist_authinfo` in MX workers need to (already) match the node IDs in `pg_dist_node` on the coordinator. This creates a disparity between the way MX and non-MX clusters are configured, and in either case it is not entirely clear what the right procedure for adding a node is.

### Syncing pg_dist_node on node activation

Instead of only syncing `pg_dist_node` only in MX, we should always sync it upon node activation (`master_add_node`/`master_activate_node`) prior to copying reference tables. This would remove the disparity between MX and non-MX clusters and take away the need for the infrastructure layer to populate `pg_dist_node` on workers. In addition, we would make sure that the node is actually up-and-running even when there are no reference tables.

### Rethinking pg_dist_poolinfo and pg_dist_authinfo configuration

Even after the above change we still have the problem of when and how to configure `pg_dist_poolinfo` and `pg_dist_authinfo` on the workers, since these tables reference nodes by their node ID. The procedure for adding a node would be: add the node on the coordinator using `master_add_inactive_node`, obtain its node ID, add entries to `pg_dist_poolinfo` and `pg_dist_authinfo` on all nodes using the node ID (*need to remove the foreign key!*), and then activate the node to sync `pg_dist_node`. However, this procedure is complex to implement correctly, especially in the presence of failures and concurrency (e.g. when first starting the cluster).

A possible alternative would be to replace the `nodeid` column in `pg_dist_poolinfo` and `pg_dist_authinfo` with hostname and port, or a node identifier assigned by the infrastructure management layer. That would allow you to configure `pg_dist_poolinfo` and `pg_dist_authinfo` before `pg_dist_node` is populated and node IDs are known on the worker.

Another alternative is to synchronize `pg_dist_authinfo` and `pg_dist_poolinfo` along with `pg_dist_node`. However, this has some downsides since we may not want to use the same settings on all nodes. `pg_dist_authinfo` may contain local file paths and the pooler configuration may differ across nodes. However, the former may be acceptable, and the latter could be solved if Citus itself implements or at least manages the outbound connection pooler(s).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.