paritytech / paritytech/web3-storage

Make client-side encryption per-chunk and deterministic to make the most of CDC

Open
#262 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
12
Forks
3
Avg merge
2d 2h
Merged PRs (30d)
33

Description

https://github.com/paritytech/web3-storage/issues/156 adds CDC to the codebase. It's a chunking methodology that separates chunks based on the content itself. This allows deduplicating chunks and reducing the number of changed chunks when editing blobs. There's a caveat though: it only works on plaintext. If you encrypt and then chunk, the ciphertext on an edited blob will not share the same byte patterns present in the ciphertext of the original blob. The way to make the most of CDC is to encrypt after chunking, and to do it in a deterministic way to make sure the same chunks get the same ciphertext.

So we need two things:

  1. Chunking should be done in the client, not on the storage provider
  2. Client-side encryption should be done after chunking, and it should be deterministic per user

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the CDC implementation introduced by issue #156 and the existing client-side encryption path; the issue names no files or tests. Confirm where chunking currently occurs, then define deterministic per-user encryption after client chunking, with tests showing identical chunks produce identical ciphertext and edited blobs preserve matching chunks.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.