frappe / frappe/agent

Experimental: Add Probabilistic Hashing in Physical Backup

Open
#304 0 comments 0 reactions 1 assignee Claimed by @tanmoysrt View on GitHub
Dominant language
Python
Stars
125
Forks
147
Avg merge
17h 20m
Merged PRs (30d)
17

Description

In Physical Backup, currently we use only file metadata for validating snapshot.

We need more validations.
But, sha256 like checksums can't be used

- We can't read the whole file, because that will do high IO usage
- cryptographic checksumming is slow on large files (5~10GB)

**Solution :**

- Randomly pick segment of files and checksum those, so that we can validate later. 1MB read for 500MB file can be the target. (Need to do some math on how much reliable it can be)
- Use non-cryptographic algo instead : https://xxhash.com/

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.