HarperFast / HarperFast/harper-pro
Copy newly replicating tables
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
Currently we "copy" tables when first connecting a node with database that doesn't have a full transaction log of all writes. And once we do that, we simply track updates in the transaction log. However, if you have previously defined a table as not replicated and then redefine the table as replicated, that means there is no catch-up mechanism for that table if you later define it as a replicated table in a database that has been replicating for a while. We continue to replicate _new_ writes, and existing data in the source table is not replicated.
We should create some mechanism for tracking which tables we have previously subscribed to, and if there is a table that is added to the list, we should copy the table as appropriate.
However, it is not clear that we always want to do this. In some situations the tables were not replicated because the previous data shouldn't be replicated, and the tables may have significant divergence, and retroactively replicating data may not be right choice.
This issue was manifested as https://github.com/HarperFast/harper/issues/332 because if the `hdb_analytics_hostname` was not replicated and then switched to replicated, the previous node names don't get replicated.
Contributor guide
Assessment
This issue has not been assessed yet.