HarperFast / HarperFast/harper-pro

Copy newly replicating tables

Open
#81 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

Currently we "copy" tables when first connecting a node with database that doesn't have a full transaction log of all writes. And once we do that, we simply track updates in the transaction log. However, if you have previously defined a table as not replicated and then redefine the table as replicated, that means there is no catch-up mechanism for that table if you later define it as a replicated table in a database that has been replicating for a while. We continue to replicate _new_ writes, and existing data in the source table is not replicated.
We should create some mechanism for tracking which tables we have previously subscribed to, and if there is a table that is added to the list, we should copy the table as appropriate.
However, it is not clear that we always want to do this. In some situations the tables were not replicated because the previous data shouldn't be replicated, and the tables may have significant divergence, and retroactively replicating data may not be right choice.
This issue was manifested as https://github.com/HarperFast/harper/issues/332 because if the `hdb_analytics_hostname` was not replicated and then switched to replicated, the previous node names don't get replicated.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.