Support buffer de-duplication
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
It's possible that an object may have multiple numpy arrays with the same contents (this will arise in some LensKit use cases). We can support de-duplication by recording more robust checksums (MD5 or SHA) of buffers, and making the buffer store effectively content-addressed.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the buffer store and serialization entry points to understand how buffers are currently identified and reused. Compare MD5 and SHA options for identifying identical NumPy array contents, then define how content-addressed storage should behave and verify that objects containing duplicate arrays are de-duplicated without changing their restored contents.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100