citusdata / citusdata/citus

Rebalance bug with local tables on citus 10.2

Open
#7,395 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

I can't seem to reproduce the issue, but these are my observations:

Local table A with a shard A_1234 on the coordinator.
Reference table B has a fk to the local table.
Distributed table C has a fk to the reference table.

When a new node is added and rebalance triggered, the operation fails with "Table A_1234 not found" the error is received from a connection on the newly added node.

Looking on the logs of the new node these things stand out:
LOG: logical replication table synchronization worker for subscription "...", table "A_3456" has started.
LOG: logical replication table synchronization worker for subscription "...", table "A_3456" has finished.
SELECT worker_apply_inter_shard_ddl_command(7890, 'public', 1234, 'public', 'ALTER TABLE public.B ADD CONSTRAINT B_to_A_fk FOREIGN KEY (A_id) REFERENCES public.A(id) NOT VALID')

The fk between the reference table shard on the new node and the local table shard on the coordinator (A_1234) is being attempted to be created. This naturally fails.
Also, a logical replication subscription is created for some A_2345 shard. This shard does not seem to exist anywhere else in the cluster.

The A table and its shard are not involved in the rebalance plan generated by `get_rebalance_table_shards_plan`, and yet it seems like the rebalance is trying to copy some shard A_3456 (??and succeeds??), then tries to create the constraint between the B_7890 shard and A_1234.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.