citusdata / citusdata/citus

Consider adding a history of topology changes

Open
#6,054 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

To implement CDC for Citus, we would need to implement a logical decoder that filters out inserts that happen as a result of internal data moves (e.g. shard move, shard split, create_distributed_table, alter_distributed_table, undistribute_table).

The logical decoder may be arbitrarily far behind, and the state of the cluster at the time of the insert it is currently decoding may have been very different. Therefore, to know whether an insert needs to be decoded, it would be useful if Citus kept a history of topology changes. That way, the decoder can reconstruct the relevant metadata at a given LSN from the history. In particular, the history would tell us at which LSN a particular shard became active, and which distributed table (name) it was part of.

Keeping a history would also be useful for users to understand how their cluster evolved over time, and be able to debug issues that happened in the past.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.