paritytech / paritytech/web3-storage
Make client-side encryption per-chunk and deterministic to make the most of CDC
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 12
- Forks
- 3
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 33
Description
https://github.com/paritytech/web3-storage/issues/156 adds CDC to the codebase. It's a chunking methodology that separates chunks based on the content itself. This allows deduplicating chunks and reducing the number of changed chunks when editing blobs. There's a caveat though: it only works on plaintext. If you encrypt and then chunk, the ciphertext on an edited blob will not share the same byte patterns present in the ciphertext of the original blob. The way to make the most of CDC is to encrypt after chunking, and to do it in a deterministic way to make sure the same chunks get the same ciphertext.
So we need two things:
- Chunking should be done in the client, not on the storage provider
- Client-side encryption should be done after chunking, and it should be deterministic per user
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the CDC implementation introduced by issue #156 and the existing client-side encryption path; the issue names no files or tests. Confirm where chunking currently occurs, then define deterministic per-user encryption after client chunking, with tests showing identical chunks produce identical ciphertext and edited blobs preserve matching chunks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100