hemilabs / hemilabs/larry

Add merklized snapshotting

Open
#9 0 comments 0 reactions 0 assignees View on GitHub
enhancement good first issue
Dominant language
Go
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Currently when making a snapshot we use a hash accumulator to determine if the contents are identical on the source and target. This is great but this does not help with a very large database to determine where the failure exists.

We therefore should create a merklized snapshot that compresses N records per leaf. Let's say every 10K records we add a leaf that is the hash accumulator valut to a merkle tree, grow this tree to size M; once full store the merkle root in a new merkle tree that records snpshots of snapshots.

With this we can create a skip list to determine much quicker where failures are at.

The merkle is just an idea, we can also keep a simple list of hashes per N records, point is, we need to be able to find data coherency issues and reconcile them without having to traverse all records.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.