citusdata / citusdata/citus

`stop_metadata_sync_to_node()` requires setting `hasmetadata` flag to `false` for other out-of-sync nodes

Open
#5,870 0 comments 0 reactions 0 assignees View on GitHub
mx usability
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

Suppose that we got into a situation where more than one MX nodes got out-of-sync for some reason (`metadatasynced = false`):

```sql
TABLE pg_dist_node;
┌────────┬─────────┬───────────┬──────────┬──────────┬─────────────┬──────────┬──────────┬─────────────┬────────────────┬──────────────────┐
│ nodeid │ groupid │ nodename │ nodeport │ noderack │ hasmetadata │ isactive │ noderole │ nodecluster │ metadatasynced │ shouldhaveshards │
├────────┼─────────┼───────────┼──────────┼──────────┼─────────────┼──────────┼──────────┼─────────────┼────────────────┼──────────────────┤
│ 2 │ 2 │ localhost │ 10701 │ default │ t │ t │ primary │ default │ f │ t │
│ 1 │ 1 │ localhost │ 10702 │ default │ t │ t │ primary │ default │ f │ t │
└────────┴─────────┴───────────┴──────────┴──────────┴─────────────┴──────────┴──────────┴─────────────┴────────────────┴──────────────────┘
(2 rows)
```

If the reason behind this situation is a bug in metadata syncing code, then nodes would never get synced again.

----------

In that case, the user might want to clear metadata snapshots on those nodes first (by using `stop_metadata_sync_to_node()`) and re-start metadata syncing again.

However, `stop_metadata_sync_to_node()` requires all other metadata nodes be synced already in order to update `pg_dist_node` entry for the node that we want to stop metadata syncing to, as in:

```sql
select stop_metadata_sync_to_node('localhost', 10701, clear_metadata=>true);
NOTICE: dropping metadata on the node (localhost,10701)
ERROR: localhost:10702 is a metadata node, but is out of sync
HINT: If the node is up, wait until metadata gets synced to it and try again.
Time: 46.204 ms

select stop_metadata_sync_to_node('localhost', 10702, clear_metadata=>true);
NOTICE: dropping metadata on the node (localhost,10702)
ERROR: localhost:10701 is a metadata node, but is out of sync
HINT: If the node is up, wait until metadata gets synced to it and try again.
```

----------

In that case, workaround is to set `hasmetadata` to `false` for one of those nodes and executing `stop_metadata_sync_to_node()` for the other one, as in:

```sql
update pg_dist_node SET hasmetadata = false where nodeport = 10701;
UPDATE 1

select stop_metadata_sync_to_node('localhost', 10702, clear_metadata=>true);
NOTICE: dropping metadata on the node (localhost,10702)
┌────────────────────────────┐
│ stop_metadata_sync_to_node │
├────────────────────────────┤
│ │
└────────────────────────────┘
(1 row)

select stop_metadata_sync_to_node('localhost', 10701, clear_metadata=>true);
NOTICE: dropping metadata on the node (localhost,10701)
┌────────────────────────────┐
│ stop_metadata_sync_to_node │
├────────────────────────────┤
│ │
└────────────────────────────┘
(1 row)
```

----------

We might want to come up with a better way of stopping metadata sync to multiple nodes instead of expecting users to set `hasmetadata` to `false` for other nodes.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.